huggingface/candle
1
0
Fork 0
mirror of https://github.com/huggingface/candle.git synced 2025-06-16 10:38:54 +00:00
Code Issues Packages Projects Releases Wiki Activity

Instructions for generating the tokenizer configs for marian-mt. (#1225)

Browse Source
This commit is contained in:
Laurent Mazare
2023-10-31 07:56:26 +01:00
committed by GitHub
parent 392a00a147
commit 7d0202710b
2 changed files with 1404 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

19
candle-examples/examples/marian-mt/README.md
View File

@ -17,3 +17,22 @@ cargo run --example marian-mt --release -- \
I know you are waiting for me. I will go through the forest, I will go through the
mountain. I cannot stay far from you any longer.</s>
```
## Generating the tokenizer.json files
You can use the following script to generate the `tokenizer.json` config files
from the hf-hub repos. This requires the `tokenizers` and `sentencepiece`
packages to be install and use the `convert_slow_tokenizer.py` script from this
directory.
```python
from convert_slow_tokenizer import MarianConverter
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("Helsinki-NLP/opus-mt-fr-en", use_fast=False)
fast_tokenizer = MarianConverter(tokenizer, index=0).converted()
fast_tokenizer.save(f"tokenizer-marian-base-fr.json")
fast_tokenizer = MarianConverter(tokenizer, index=1).converted()
fast_tokenizer.save(f"tokenizer-marian-base-en.json")
```

1385
candle-examples/examples/marian-mt/convert_slow_tokenizer.py Normal file
View File

File diff suppressed because it is too large Load Diff