3 Commits

Author	SHA1	Message	Date
Laurent Mazare	3ad4770eb6	Use cat for faster MQA computation. (#2043) ... * Use cat for faster MQA computation. * Move the function to utils + use it in mistral. * Use the shared repeat-kv in a few more models. * Fix.	2024-04-12 09:15:10 +02:00
Jani Monoses	90fc82211f	Use a common with_tracing::RmsNorm in a few models. (#1871) ... * Add RmsNorm with tracing. * Use with_tracing::RmsNorm in some models.	2024-03-18 21:40:06 +01:00
Laurent Mazare	614842b311	Add the Mixtral model. (#1437) ... * Add the Mixtral model. * Add more of the mixtral layers. * Add the final layers for mixtral. * Sketch the expert selection. * Add some expert routing logic. * Hopefully finish the routing logic for mixtral. * Add the mixtral example. * Fix the weight filenames. * Bugfix. * Another fix. * Yet another fix + remove the unused pragma. * Shape fix. * Add a readme.	2023-12-15 14:19:56 -06:00