mirror of
https://github.com/huggingface/candle.git
synced 2025-06-18 19:47:12 +00:00
9c4b4f0da0
- Add a device param, wherever needed. - Create new QMetal storage thing that implements QuantizedType. - Update everywhere needed. Fix Python. Fixing examples. Fix: fmt + clippy + stub. Moving everything around. Only missing the actual implems. Fixing everything + adding dequantized kernels. More work. Fixing matmul. Fmt + Clippy Some clippy fixes. Working state. Q2K Metal -> Bugged (also present in GGML). Q4K CPU -> Bugged (present previously, new test catch it). Q5K CPU -> Bugged (present previously). Q8_1 Both -> Never really implemented it seems Q8K metal -> Never implemented in metal Fixing Q2K bug (present in ggml).
Running T5 with Candle and WASM
Here, we provide two examples of how to run Bert using a Candle-compiled WASM binary and runtime.
Vanilla JS and WebWorkers
To build and test the UI made in Vanilla JS and WebWorkers, first we need to build the WASM library:
sh build-lib.sh
This will bundle the library under ./build and we can import it inside our WebWorker like a normal JS module:
import init, { ModelConditionalGeneration, ModelEncoder } from "./build/m.js";
For the quantized version, we need to import the quantized module:
import init, { ModelConditionalGeneration, ModelEncoder } from "./build/m-quantized.js";
The full example can be found under ./index.html. All needed assets are fetched from the web, so no need to download anything.
Finally, you can preview the example by running a local HTTP server. For example:
python -m http.server
Then open http://localhost:8000/index.html in your browser.