huggingface/candle
1
0
Fork 0
mirror of https://github.com/huggingface/candle.git synced 2025-06-15 18:28:24 +00:00
Code Issues Packages Projects Releases Wiki Activity

Q6K quantization (#495)

Browse Source 
* Print the detected arch options.

* Add the q6k quantization.

* Add a currently broken test.

* Bugfix.

* Bugfix.

* Another bugfix.

* Another bugfix + get the test to work.
This commit is contained in:
Laurent Mazare
2023-08-17 22:22:57 +01:00
committed by GitHub
parent fc81af1712
commit 557b2c28dd
4 changed files with 257 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
candle-examples/examples/quantized/main.rs
View File

@ -348,6 +348,14 @@ fn main() -> anyhow::Result<()> {
None
};
println!(
"avx: {}, neon: {}, simd128: {}, f16c: {}",
candle::utils::with_avx(),
candle::utils::with_neon(),
candle::utils::with_simd128(),
candle::utils::with_f16c()
);
let mut file = std::fs::File::open(&args.model()?)?;
let start = std::time::Instant::now();
let model = Content::read(&mut file)?;