• MoCo
• • Requirements
  • Dataset
  • Usage
  • • Train MoCo
    • Linear Probing
  • Results

MoCo

A PyTorch implementation of MoCo based on CVPR 2020 paper Momentum Contrast for Unsupervised Visual Representation Learning.

Requirements

• Anaconda
• PyTorch

conda install pytorch=1.6.0 torchvision cudatoolkit=10.2 -c pytorch

Dataset

CIFAR10 dataset is used in this repo, the dataset will be downloaded into data directory by PyTorch automatically.

Usage

Train MoCo

python train.py --batch_size 1024 --epochs 50
optional arguments:
--feature_dim                 Feature dim for each image [default value is 128]
--m                           Negative sample number [default value is 4096]
--temperature                 Temperature used in softmax [default value is 0.5]
--momentum                    Momentum used for the update of memory bank [default value is 0.999]
--k                           Top k most similar images used to predict the label [default value is 200]
--batch_size                  Number of images in each mini-batch [default value is 256]
--epochs                      Number of sweeps over the dataset to train [default value is 500]

Linear Probing

Fix backbone weights and replace the projection head with classification head.

python linear.py --batch_size 1024 --epochs 200 --model_path ../moco_ep500_bs512_t0.5_m0.999_d128@200.pth
optional arguments:
--weights                  The pretrained model path [default value is 'results/128_4096_0.5_0.999_200_256_500_model.pth']
--batch_size                  Number of images in each mini-batch [default value is 256]
--epochs                      Number of sweeps over the dataset to train [default value is 100]

Results

There are some difference between this implementation and official implementation, the model (ResNet50) is trained on one NVIDIA V100 GPU:

1. No Gaussian blur used;
2. Adam optimizer with learning rate 1e-3 is used to replace SGD optimizer;
3. No Linear learning rate scaling used.

500 epochs for moco training , 100 epochs for linear probing.

EP	Acc
100	77.33
200	83.19
300	85.55
400	86.45
500	87.78