huggingface/candle
1
0
Fork 0
mirror of https://github.com/huggingface/candle.git synced 2025-06-15 18:28:24 +00:00
Code Issues Packages Projects Releases Wiki Activity

Support for quantized tensors in the python api. (#706)

Browse Source 
* Add more pyo3 support.

* Add some support for quantized tensors in pyo3.

* Add an arc layer on qmatmul.

* Add the quantized matmul.

* Quantization support.

* More quantization support.

* Test the python quantization.
This commit is contained in:
Laurent Mazare
2023-09-01 16:53:42 +02:00
committed by GitHub
parent 237323c2bc
commit 2ed78ab336
3 changed files with 172 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
candle-pyo3/test.py
View File

@ -33,3 +33,9 @@ print(t.to_dtype("u8"))
t = candle.randn((5, 3))
print(t)
print(t.dtype)
t = candle.randn((16, 256))
quant_t = t.quantize("q6k")
dequant_t = quant_t.dequantize()
diff2 = (t - dequant_t).sqr()
print(diff2.mean_all())