6 Commits

Author	SHA1	Message	Date
Laurent Mazare	7ebc3548e1	Use flash-attn in gemma. (#2195) ... * Use flash-attn in gemma. * Fix flash-attn for head dim 256.	2024-05-18 19:18:59 +02:00
Laurent Mazare	a0460cd2b1	Add the code-gemma models. (#2038) ... * Add the code-gemma models. * Tweak to the gemma config.	2024-04-10 21:19:21 +02:00
Laurent Mazare	33c9b66554	Add the new gemma models. (#2023) ... * Add the new gemma models. * Revert the lightning changes. * Support for the 1.1 models.	2024-04-06 21:25:38 +02:00
Laurent Mazare	21f1d04976	Add the instruction finetuned gemma variants. (#1790)	2024-03-02 18:56:59 +01:00
Laurent Mazare	8d04f70f4d	Fix the eos token for gemma. (#1753)	2024-02-24 11:07:02 +01:00
Laurent Mazare	45d5322d62	Add the Gemma models. (#1741) ... * Add the Gemma models. * Add the gemma example. * Adapt the RmsNorm. * Get the 2b model to work. * 7b support. * Use the config head dim. * Yet another fix. * Make the matrixes contiguous. * Also get the 7b model to work. * And add to the readme.	2024-02-21 22:02:50 +01:00