2 Commits

Author	SHA1	Message	Date
Jani Monoses	90fc82211f	Use a common with_tracing::RmsNorm in a few models. (#1871) ... * Add RmsNorm with tracing. * Use with_tracing::RmsNorm in some models.	2024-03-18 21:40:06 +01:00
Laurent Mazare	614842b311	Add the Mixtral model. (#1437) ... * Add the Mixtral model. * Add more of the mixtral layers. * Add the final layers for mixtral. * Sketch the expert selection. * Add some expert routing logic. * Hopefully finish the routing logic for mixtral. * Add the mixtral example. * Fix the weight filenames. * Bugfix. * Another fix. * Yet another fix + remove the unused pragma. * Shape fix. * Add a readme.	2023-12-15 14:19:56 -06:00