mirror of https://github.com/huggingface/candle.git synced 2025-06-16 18:48:51 +00:00

Files

Laurent Mazare b43ca493f6 Add more quantized flan t5 variants (#923 )

* Add the quantized flan-t5-large variant.

* Add more sizes.

2023-09-21 13:23:30 +01:00

main.rs

Add more quantized flan t5 variants (#923 )

2023-09-21 13:23:30 +01:00

README.md

T5 quantized example (#922 )

2023-09-21 12:33:15 +01:00

README.md

candle-quantized-t5

This example uses a quantized version of the t5 model.

$ cargo run --example quantized-t5 --release -- --prompt "translate to German: A beautiful candle."
...
 Eine schöne Kerze.

The weight file is automatically retrieved from the hub. It is also possible to generate quantized weight files from the original safetensors file by using the tensor-tools command line utility via:

cargo run --example tensor-tools --release -- quantize --quantization q6k PATH/TO/T5/model.safetensors /tmp/model.gguf