huggingface/candle
1
0
Fork 0
mirror of https://github.com/huggingface/candle.git synced 2025-06-19 03:54:56 +00:00
Code Issues Packages Projects Releases Wiki Activity

GQA support in the quantized model. (#555)

Browse Source 
* GQA support in the quantized model.

* Fix the reshaping.

* Fix the main llama model.

* Infer the proper gqa from the model kind.
This commit is contained in:
Laurent Mazare
2023-08-22 19:41:10 +01:00
committed by GitHub
parent 07067b01dc
commit f9ecc84477
2 changed files with 32 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
candle-examples/examples/llama/model.rs
View File

@ -291,7 +291,7 @@ impl CausalSelfAttention {
let x = x
.unsqueeze(2)?
.expand((b_sz, n_kv_head, n_rep, seq_len, head_dim))?
.reshape((b_sz, n_kv_head, n_rep, seq_len, head_dim))?;
.reshape((b_sz, n_kv_head * n_rep, seq_len, head_dim))?;
Ok(x)
}
}