236b820e28
Another prelu bugfix. ( #1407 )
2023-12-06 09:54:41 +01:00
2648e797c2
Use the proper broadcasting for prelu. ( #1406 )
2023-12-05 07:09:31 +01:00
b5c283e86f
Add the prelu layer. ( #1402 )
2023-12-03 16:06:09 +00:00
18d30005c5
Add support to UL2 model family ( #1300 )
...
* Add support to UL2 model family
* Update docs with UL2
* Create ActivationWithOptionalGating to avoid polluting activations
* Also refactor quantized t5
* Remove useless conversion
* Revert Activation::NewGelu name change
* Remove useless return
* Apply rustfmt and clippy recommendations
* Reuse t5::ActivationWithOptionalGating in quantized version
* (cosmetic change) use a match rather than ifs + avoid early returns.
---------
Co-authored-by: Laurent <laurent.mazare@gmail.com >
2023-11-09 18:55:09 +01:00
a2a20aeecc
Add the swiglu activation from the chatglm PR. ( #1246 )
2023-11-02 20:01:34 +01:00
d39d0c40fd
Add hard-sigmoid and hard-swish activations ( #1244 )
...
* Add hard-sigmoid and hard-swish activations
* Update ops.rs
* Use / rather than div.
---------
Co-authored-by: Laurent <laurent.mazare@gmail.com >
2023-11-02 18:20:27 +01:00
392a00a147
Add support for the marian base model. ( #1221 )
2023-10-30 19:20:36 +00:00
c8face3f95
Add the relu2 and relu6 activations. ( #1201 )
2023-10-27 20:51:16 +01:00
53510ce427
Use a silu activation in mistral. ( #991 )
2023-09-29 07:06:54 +01:00
ce0a4e3a85
Use the gelu-erf activation. ( #969 )
2023-09-26 22:30:21 +01:00
0633c85514
Add leaky-relu in the activation enum. ( #858 )
2023-09-15 07:05:38 +01:00
49d3f7f708
Add support to flan-t5 ( #840 )
2023-09-13 19:27:20 +02:00
9daa6dbe87
Extract T5 module and add main function to use it ( #829 )
...
* Extract t5 out of musicgen
* Add main for t5 module
2023-09-13 07:14:05 +01:00
c78ce76501
Add a simple Module trait and implement it for the various nn layers ( #500 )
...
* Start adding the module trait.
* Use the module trait.
* Implement module for qmatmul.
2023-08-18 09:38:22 +01:00
b06e1a7e54
[nn] Move the Embedding and Activation parts. ( #116 )
...
* Share the Embedding and Activation parts.
* Tweak some activations.
2023-07-10 10:24:52 +01:00
