6 Commits

Author	SHA1	Message	Date
Nicolas Patry	403680f17d	Quantized GGUF style (#1523) ... * Metal quantized modifications proposal. - Add a device param, wherever needed. - Create new QMetal storage thing that implements QuantizedType. - Update everywhere needed. Fix Python. Fixing examples. Fix: fmt + clippy + stub. Moving everything around. Only missing the actual implems. Fixing everything + adding dequantized kernels. More work. Fixing matmul. Fmt + Clippy Some clippy fixes. Working state. Q2K Metal -> Bugged (also present in GGML). Q4K CPU -> Bugged (present previously, new test catch it). Q5K CPU -> Bugged (present previously). Q8_1 Both -> Never really implemented it seems Q8K metal -> Never implemented in metal Fixing Q2K bug (present in ggml). * Cleanup. * Fix the rebase. * Removing the fences speeds everything up and *is* correct this time... * Cleanup the fence. * After rebase. * Bad code removal. * Rebase after phi2 merge + fix replit default to CPU. * Making the CI happy. * More happy tests. --------- Co-authored-by: Nicolas Patry <nicolas@Nicolass-MacBook-Pro.local>	2024-01-17 10:27:58 +01:00
Laurent Mazare	c12ad45562	Add a KV cache to marian decoding. (#1226)	2023-10-31 08:47:44 +00:00
Laurent Mazare	a11af79e23	Add a quantized blip model. (#1155) ... * Add a quantized blip model. * Integrate the quantized blip model to the actual example.	2023-10-22 20:33:25 +01:00
Laurent Mazare	df2f89b6cf	Add some KV cache to blip. (#1150) ... * Add some KV cache to blip. * Mention BLIP in the readme.	2023-10-22 09:44:48 +01:00
Laurent Mazare	2531b13bf8	Blip fixes (#1145) ... * Some fixes for the blip example. * Stop generating on sep tokens. * Clippy fixes. * rustfmt.	2023-10-21 21:34:48 +01:00
Laurent Mazare	0d9bb4eb18	Add the blip example. (#1144) ... * Add the blip example. * Tweak the example. * Implement the cross-attn logic. * Fix some shape mismatches. * Get some logits out. * Get some caption to be generated.	2023-10-21 20:05:02 +01:00