Are you sure you want to delete this task? Once this task is deleted, it cannot be recovered.
You can not select more than 25 topics
Topics must start with a chinese character,a letter or number, can include dashes ('-') and can be up to 35 characters long.
taoht
4ce4f3a914
|
6 months ago | |
|---|---|---|
| .. | ||
| llama_vocab/llama_zh_hf | 7 months ago | |
| spm_13w | 6 months ago | |
| __init__.py | 6 months ago | |
| readme.txt | 6 months ago | |
| tokenlizer_test.py | 7 months ago | |
readme.txt
使用方法
# 1、Define tokenizer
from tokenizer.spm_13w.tokenizer import SpmTokenizer
vocab_file = '/path/to/spm.128k.model.1'
tokenizer = SpmTokenizer(vocab_file)
EOT = tokenizer.eot_id # 128298
EOD = tokenizer.eod_id # 128299
PAD = tokenizer.pad_id # 128297
vocab_size = tokenizer.vocab_size # 128320
# Tokenize input sentence to ids
input_sentence = "你今天中午吃的什么？"
input_id = tokenizer.tokenize(input_sentence)
# Decode output ids to sentence
output_sentence = tokenizer.detokenize(output_id])
# 【提醒】不需要提前分词
【注意】mPanGus实现翻译时输入组合方式为：
[src_language_id,src_language_id,src_language_id] + [input_id] + [translate_id, translate_id, translate_id]
需要配置:
end_token=128298
vocab_size=128320
top_p=1
eod_id=128298
padding_id=128297
use_pynative_op=1
use_past=true
per_batch_size=1
PengCheng.Mind 200B 脑海大模型是基于Transformer 架构的2010亿参数自回归式语言模型
Text Python
