huggingface/candle
1
0
Fork 0
mirror of https://github.com/huggingface/candle.git synced 2025-06-19 19:58:35 +00:00
Code Issues Packages Projects Releases Wiki Activity

Documentation Pass for Models (#2617)

Browse Source 
* links in chinese_clip

* links for clip model

* add mod docs for flux and llava

* module doc for MMDIT and MIMI

* add docs for a few more modesl

* mod docs for bert naser and beit

* add module docs for convmixer colpali codegeex and chatglm

* add another series of moddocs

* add  fastvit-llama2_c

* module docs mamba -> mobileone

* module docs from moondream-phi3

* mod docs for quantized and qwen

* update to yi

* fix long names

* Update llama2_c.rs

* Update llama2_c_weights.rs

* Fix the link for mimi + tweaks

---------

Co-authored-by: Laurent Mazare <laurent.mazare@gmail.com>
This commit is contained in:
zachcp
2024-11-15 02:30:15 -05:00
committed by GitHub
parent 0ed24b9852
commit f689ce5d39
94 changed files with 1001 additions and 51 deletions
Download Patch File Download Diff File Expand all files Collapse all files

21
candle-transformers/src/models/recurrent_gemma.rs
View File

@ -1,5 +1,22 @@
// This implementation is based on the python version from huggingface/transformers.
// https://github.com/huggingface/transformers/blob/b109257f4fb8b1166e7c53cc5418632014ed53a5/src/transformers/models/recurrent_gemma/modeling_recurrent_gemma.py#L2
//! Recurrent Gemma model implementation
//!
//! Recurrent Gemma is a version of the Gemma language model that incorporates recurrent memory.
//! This allows the model to maintain state between predictions and have longer-range memory.
//!
//! Key characteristics:
//! - Real-gated linear recurrent units (RGLRU)
//! - 1D convolution for local context
//! - RMSNorm for layer normalization
//! - Rotary positional embeddings (RoPE)
//! - Grouped query attention
//!
//! References:
//! - [Gemma: Open Models Based on Gemini Technology](https://blog.google/technology/developers/gemma-open-models/)
//! - [Recurrent Memory model architecture](https://arxiv.org/abs/2402.00441)
//!
//! This implementation is based on the python version from huggingface/transformers.
//! https://github.com/huggingface/transformers/blob/b109257f4fb8b1166e7c53cc5418632014ed53a5/src/transformers/models/recurrent_gemma/modeling_recurrent_gemma.py#L2
//!
use candle::{DType, Device, IndexOp, Module, Result, Tensor, D};
use candle_nn::{linear_b as linear, Linear, VarBuilder};
use std::sync::Arc;