PLMs_GNNs_in_NLP/ADBlur4SPC

• ADBlur for Sentence Pair Classification
• • Requirements
  • Overview
  • Datasets & PLMs
  • Training on MNLI & HANS

ADBlur for Sentence Pair Classification

This is the implementation of [ADBlur] in Pytorch. I was strongly inspired by Hugging Face's code and I referred a lot to their codes

Requirements

Python > 3.6, fire, tqdm, tensorboardx,
tensorflow (for loading checkpoint file)

Overview

This contains 9 python files.

• tokenization.py : Tokenizers adopted from the original Google BERT's code
• checkpoint.py : Functions to load a model from tensorflow's checkpoint file
• models.py : Model classes for a general transformer
• optim.py : A custom optimizer (BertAdam class) adopted from Hugging Face's code
• train.py : A helper class for training and evaluation
• utils.py : Several utility functions
• pretrain.py : An example code for pre-training transformer
• ADBlur.py & ADBlur_RoNERTa : An example code for ADBlur framework
• classify.py : An example code for fine-tuning using pre-trained transformer

Datasets & PLMs

Download preprocessed datasets from MNLI & QQP & OOD evaluations and the parameters for
pre-trained BERT and RoBERTa
before fine-tuning.

Training on MNLI & HANS

• make sure that "total_steps" in train_mrpc.json is n_epochs*(num_data/batch_size)

python classify.py \
    --train_cfg='config/train_mrpc.json',
    --model_cfg='config/bert_base.json',
    --train_file='glue_data/MNLI/train_aug.tsv',
    --dev_file='glue_data/MNLI/ood_aug.tsv',
    --iid_dev_file='glue_data/MNLI/dev_matched_aug.tsv',
    --ood_ent_file='glue_data/MNLI/hans_eval_aug.tsv',
    --ood_nent_file='glue_data/MNLI/hans_nen_eval_aug.tsv',
    --model_file='save_model/model_steps_98176.pt',
    --pretrain_file='uncased_L-12_H-768_A-12/bert_model.ckpt',
    --data_parallel=True,
    --vocab='uncased_L-12_H-768_A-12/vocab.txt',
    --save_dir='save_model/bert-base/mnli',
    --max_len =128,
    --mode='train'):