ADBlur for Sentence Pair Classification
This is the implementation of [ADBlur] in Pytorch. I was strongly inspired by Hugging Face's code and I referred a lot to their codes
Requirements
Python > 3.6, fire, tqdm, tensorboardx,
tensorflow (for loading checkpoint file)
Overview
This contains 9 python files.
tokenization.py
: Tokenizers adopted from the original Google BERT's code
checkpoint.py
: Functions to load a model from tensorflow's checkpoint file
models.py
: Model classes for a general transformer
optim.py
: A custom optimizer (BertAdam class) adopted from Hugging Face's code
train.py
: A helper class for training and evaluation
utils.py
: Several utility functions
pretrain.py
: An example code for pre-training transformer
ADBlur.py & ADBlur_RoNERTa
: An example code for ADBlur framework
classify.py
: An example code for fine-tuning using pre-trained transformer
Datasets & PLMs
Download preprocessed datasets from MNLI & QQP & OOD evaluations and the parameters for
pre-trained BERT and RoBERTa
before fine-tuning.
Training on MNLI & HANS
- make sure that "total_steps" in train_mrpc.json is n_epochs*(num_data/batch_size)
python classify.py \
--train_cfg='config/train_mrpc.json',
--model_cfg='config/bert_base.json',
--train_file='glue_data/MNLI/train_aug.tsv',
--dev_file='glue_data/MNLI/ood_aug.tsv',
--iid_dev_file='glue_data/MNLI/dev_matched_aug.tsv',
--ood_ent_file='glue_data/MNLI/hans_eval_aug.tsv',
--ood_nent_file='glue_data/MNLI/hans_nen_eval_aug.tsv',
--model_file='save_model/model_steps_98176.pt',
--pretrain_file='uncased_L-12_H-768_A-12/bert_model.ckpt',
--data_parallel=True,
--vocab='uncased_L-12_H-768_A-12/vocab.txt',
--save_dir='save_model/bert-base/mnli',
--max_len =128,
--mode='train'):