mirror of
https://github.com/huggingface/candle.git
synced 2025-06-16 02:38:10 +00:00
403680f17d
* Metal quantized modifications proposal. - Add a device param, wherever needed. - Create new QMetal storage thing that implements QuantizedType. - Update everywhere needed. Fix Python. Fixing examples. Fix: fmt + clippy + stub. Moving everything around. Only missing the actual implems. Fixing everything + adding dequantized kernels. More work. Fixing matmul. Fmt + Clippy Some clippy fixes. Working state. Q2K Metal -> Bugged (also present in GGML). Q4K CPU -> Bugged (present previously, new test catch it). Q5K CPU -> Bugged (present previously). Q8_1 Both -> Never really implemented it seems Q8K metal -> Never implemented in metal Fixing Q2K bug (present in ggml). * Cleanup. * Fix the rebase. * Removing the fences speeds everything up and *is* correct this time... * Cleanup the fence. * After rebase. * Bad code removal. * Rebase after phi2 merge + fix replit default to CPU. * Making the CI happy. * More happy tests. --------- Co-authored-by: Nicolas Patry <nicolas@Nicolass-MacBook-Pro.local>
Installation
From the candle-pyo3 directory, enable a virtual env where you will want the
candle package to be installed then run.
maturin develop -r
python test.py
Generating Stub Files for Type Hinting
For type hinting support, the candle-pyo3 package requires *.pyi files. You can automatically generate these files using the stub.py script.
Steps:
- Install the package using
maturin. - Generate the stub files by running:
python stub.py
Validation:
To ensure that the stub files match the current implementation, execute:
python stub.py --check