Are you sure you want to delete this task? Once this task is deleted, it cannot be recovered.
ZhuiyiTechnology a79a412cca | 1 year ago | |
---|---|---|
clue | 2 years ago | |
multi-task | 1 year ago | |
LICENSE | 2 years ago | |
README.md | 1 year ago | |
train_mlm.py | 2 years ago |
RoFormer升级版,主要通过结构的简化来提升速度,并通过无监督预训练和有监督预训练的结合来提升效果,从而达到了速度与效果的“双赢”。
bert4keras >= 0.11.0
多任务训练代码参考 https://github.com/ZhuiyiTechnology/roformer-v2/tree/main/multi-task
Small版:两张3090(24G),先用无监督MLM训练了100万步(maxlen为512),然后有监督多任务训练了75万步(maxlen从64到512不等,取决于任务),batch_size为512,优化器为LAMB;
Base版:四张3090(24G),先用无监督MLM训练了100万步(maxlen为512),然后有监督多任务训练了75万步(maxlen从64到512不等,取决于任务),batch_size为512,优化器为LAMB;
Large版:两张A100(80G),先用无监督MLM训练了100万步(maxlen为512),然后有监督多任务训练了50万步(maxlen从64到512不等,取决于任务),batch_size为512,优化器为LAMB。
注:无监督的训练数据为280G,有监督的训练数据约为20G(77个标注数据集,构建了92个任务进行多任务训练,涵盖文本分类、文本匹配、阅读理解、信息抽取、指代消解等常见自然语言理解任务),large版的有监督训练步数更少,是因为20G的标注数据实在不够“喂饱”large级别的模型,继续训练下去出现了过拟合现象。
Bibtex:
@techreport{roformerv2,
title={RoFormerV2: A Faster and Better RoFormer - ZhuiyiAI},
author={Jianlin Su, Shengfeng Pan, Bo Wen, Yunfeng Liu},
year={2022},
url="https://github.com/ZhuiyiTechnology/roformer-v2",
}
No Description
Python
Dear OpenI User
Thank you for your continuous support to the Openl Qizhi Community AI Collaboration Platform. In order to protect your usage rights and ensure network security, we updated the Openl Qizhi Community AI Collaboration Platform Usage Agreement in January 2024. The updated agreement specifies that users are prohibited from using intranet penetration tools. After you click "Agree and continue", you can continue to use our services. Thank you for your cooperation and understanding.
For more agreement content, please refer to the《Openl Qizhi Community AI Collaboration Platform Usage Agreement》