• Contents
• Flownet2 Description
• Model Architecture
• Dataset
• Environment Requirements
• Quick Start
• Script Description
• • Script and Sample Code
  • Script Parameters
  • Training Process
  • • Training
    • Distributed Training
  • Evaluation Process
  • • Evaluation
  • Export Process
  • • Export
• Model Description
• • Performance
  • • Training Performance
    • • FlowNet2 train on FlyingChairs
    • Inference Performance
    • • FlowNet2 infer on MpiSintelClean
• ModelZoo Homepage

Contents

• Contents
• Flownet2 Description
• Model Architecture
• Dataset
• Environment Requirements
• Quick Start
• Script Description
  • Script and Sample Code
  • Script Parameters
  • Training Process
    • Training
    • Distributed Training
  • Evaluation Process
    • Evaluation
  • Export Process
    • Export
• Model Description
  • Performance
    • Training Performance
      • FlowNet2 train on FlyingChairs
    • Inference Performance
      • FLowNet2 infer on MpiSintelClean
• ModelZoo Homepage

Flownet2 Description

FlowNet2.0, a deep network proposed in 2017, which performs end-to-end learning on optical flow data.
It is optimized based on the FlowNet network , The large improvements in quality and
speed are caused by three major contributions: first, it
focus on the training data and show that the schedule of
presenting data during training is very important. Second,
it develop a stacked architecture that includes warping
of the second image with intermediate optical flow. Third,
it elaborate on small displacements by introducing a subnetwork specializing on small motions.

Compared with the FLownet network, FlowNet 2.0 is only
marginally slower than the original FlowNet but decreases
the estimation error by more than 50%.

FlowNet2 paper：Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, Thomas Brox

FlowNet paper：Philipp Fischer, Alexey Dosovitskiy, Eddy Ilg, Philip Häusser, Caner Hazırbaş, Vladimir Golkov, Patrick van der Smagt, Daniel Cremers, Thomas Brox

Model Architecture

The FlowNet2 network is stacked by multiple flownet sub-modules. After the output of the previous network is processed by warp, it is used as the input of the second network.

The model structure is flowNet2CSS and FlowNet2SD two sub-networks fuse the output through the FlownetFusion network, and the entire large network structure formed is FLowNet2
The FlowNet2CSS network is a stack of FLowNet2C and two FLowNet2S. The specific structure can be further understood according to the paper

This source code provides the following model structure, which can be configured and used in the yaml file:

• FlowNet2S
• FlowNet2C
• FlowNet2CS
• FlowNet2CSS
• FlowNet2SD
• FlowNet2

Dataset

Dataset used: FlyingChairs

• Dataset size：31GB，22,872 pairs 512*384 colorful images
• Data format：PPM
  • Note：Data will be processed in src/dataset.py

Dataset used: ChairsSDHom

• Dataset size: 51GB, 21,668 pairs 512*384 colorful images
  • Train：20,965 pairs image
  • Test： 703 pairs image
• Data format：PNG
  • Note: Data will be processed in src/dataset.py

Dataset used: MpiSintel

• Dataset size: 536M, 1024 x 436 colorful images in 23 classes
  • MpiSintelClean：1150 images
  • MpiSintelFinal： 1150 images
• Data format：PNG
  • Note: Data will be processed in src/dataset.py

Environment Requirements

• Hardware（Ascend/GPU/CPU）
  • Prepare hardware environment with Ascend/GPU/CPU processor.
• Framework
  • MindSpore
• For more information, please check the resources below：
  • MindSpore Tutorials
  • MindSpore Python API

Quick Start

After installing MindSpore via the official website, you can start training and evaluation as follows:

• download pretrained parameter

  FlowNet2 620MB

  FlowNet2-C 149MB

  FlowNet2-CS 297MB

  FlowNet2-CSS 445MB

  FlowNet2-CSS-ft-sd 445MB

  FlowNet2-S 148MB

  FlowNet2-SD 173MB

• convert pretrained parameter (from pytorch pretrained parameter to mindspore pretained parameter,so the env should both installed torch and mindspore)
  convert pytorch pretrained parameter to mindspore pretrained parameter

  bash run_ckpt_convert.sh [PYTORCH_FILE_PATH] [MINDSPORE_FILE_PATH]
  # example:
  bash run_ckpt_convert.sh /path/to/flownet2.pth.tar /path/to/flownet2.ckpt
  

• config pretrained parameter path in yaml file

  pre_trained:     # whether use pretrained parameter file 'True' or 'False'
  pre_trained_ckpt_path: # pretrained checkpoint file path
  # 实例：
  pre_trained: True
  pre_trained_ckpt_path: /path/checkpoint/flownet2.ckpt
  

• config dataset name and path in yaml file

  train_data: [DATASET_NAME]  # Name of dataset, 'FlyingChairs' or 'MpiSintelFinal' or 'MpiSintelClean'
  train_data_path：[DATASET_PATH] # path of dataset
  # example：
  train_data: FlyingChairs
  train_data_path: /path/to/FlyingChairs_release/data
  

• running on GPU

  For running on GPU, please change device_target from Ascend to GPU in configuration file default_config.yaml

  # run training example
  export CUDA_VISIBLE_DEVICES=0
  python train.py > train.log 2>&1 &
  
  # run distributed training example
  bash scripts/run_train_gpu.sh 8 0,1,2,3,4,5,6,7
  
  # run evaluation example
  python eval.py --crop_type="Center" --eval_checkpoint_path=[EVAL_CHECKPOINT_PATH] > eval.log 2>&1 &  
  OR
  bash scripts/run_eval_gpu.sh [MpiSintelClean/MpiSintelFinal] [DATA_PATH] [MODEL_NAME] [CKPT_PATH] [DEVICE_ID]
  

We use FlyingChairs dataset by default. Your can also pass $dataset_type to the scripts so that select different datasets. For more details, please refer the specify script.

Script Description

Script and Sample Code

├── model_zoo
    ├── README.md                          // descriptions about all the models
    ├── flownet2
        ├── README.md                      // descriptions about flownet2
        ├── scripts
        │   ├── run_ckpt_convert.sh        // shell script for converting pytorch ckpt file to pickle file on GPU
        │   ├── run_eval_gpu.sh            // shell script for eval on GPU
        │   └── run_train_gpu.sh           // shell script for training on GPU
        ├── src
        │   ├── dataset.py                 // creating dataset
        │   ├── eval_callback.py           // eval callback when training
        │   ├── metric.py                  // metric to calculate mean error
        │   ├── model_utils
        │   │   ├── ckpt_convert.py         // convert pytorch ckpt file to pickle file
        │   │   ├── config.py               // parameter configuration
        │   │   ├── device_adapter.py       // device adapter
        │   │   ├── local_adapter.py        // local adapter
        │   │   ├── moxing_adapter.py       // moxing adapter
        │   │   ├── frame_utils.py          // utils to read files of dataset
        │   │   └── tools.py                // tools to match class with paratmeter from config
        │   ├── models.py                   // FlowNet2/FlowNet2CSS/FlowNet2CS/FlowNet2C/FlowNet2S/FlowNet2SD model
        │   └── submodels
        │       ├── FlowNetC.py             // FlowNetC model
        │       ├── FlowNetFusion.py        // FlowNetFusion model
        │       ├── FlowNetS.py             // FlowNetS model
        │       ├── FlowNetSD.py            // FlowNetSD model
        │       └── submodules.py           // submodules used in flownet model
        ├── default_config.yaml             // parameter configuration
        ├── requirements.txt                // requirements configuration
        ├── export.py                       // convert mindspore ckpt file to MINDIR file
        ├── eval.py                         // evaluation script
        └── train.py                        // training script

Script Parameters

Parameters for both training and evaluation can be set in config.py

• config for FLowNet2

    # ==============================================================================
    # Device
    device_target:          "GPU"
    device_id:              0
  
    # Dataset Setup
    crop_type:              Random                   # Type of cropping operation (Random and Center)
    crop_size:              [384, 512]               # (Height, Width) of image when training
    eval_size:              [256, 256]               # (Height, Width) of image when eval
  
    # Experiment Setup
    model:                  "FlowNet2"                # Name of model to be loaded
    rgb_max:                255                       # rgb channel used
    batchNorm:              False                     # boolean switch to whether add batchnorm before conv
    lr:                     1e-6                      # Learning rate
    num_parallel_workers:   2                         # Number of CPU worker used to load data
    max_rowsize:            2                         # Number of max rowsize used to load data
    batch_size:             2                         # Numbers of image pairs in a mini-batch
    epoch_size:             100                       # Total number of epochs
    pre_trained:            1                         # Load pretrained network
    pre_trained_ckpt_path:  "/path/flownet2.ckpt"     # Pretrained ckpt path
    seed:                   1                         # Seed for reproducibility
    is_dynamicLoss_scale:   0                         # Using dynamicLoss scale or fix scale
    scale:                  1024                      # Fix scale value
    weight_decay:           0.00001                   # Weight decay
    train_data:             "FlyingChairs"            # Train Dataset name
    train_data_path:        "/path/ds/FlyingChairs_release/data"       # Train Dataset path
  
    # Train Setup
    run_distribute:         1                         # Distributed training or not
    is_save_on_master:      1                         # Only save ckpt on master device
    save_checkpoint:        1                         # Is save ckpt while training
    save_ckpt_interval:     2                         # Saving ckpt interval
    keep_checkpoint_max:    20                        # Max ckpt file number
    save_checkpoint_path:   "/path/ckpt/"             # Ckpt save path
  
    # eval Setup
    eval_data:              "MpiSintelClean"           # Eval Dataset name
    eval_data_path:         "/home/shm/ds/training"    # Eval Dataset path
    eval_checkpoint_path:   "/path/flownet2.ckpt"      # Ckpt path used to eval
    run_evalCallback:       1                          # Is run evalCallBack while training
    eval_start_epoch:       1                          # EvalCallback start epoch
    eval_interval:          5                          # EvalCallback running interval
    save_best_ckpt:         1                          # Is save best ckpt
  
    # Export Setup
    mindir_file_name:        "Flownet2"                 # Save file path
    file_format:             "MINDIR"                   # Save file format
  
    # Modelarts Setup
    enable_modelarts:       0                           # Is training on modelarts
  

For more configuration details, please refer the script config.py.

Training Process

Training

• running on GPU

  export CUDA_VISIBLE_DEVICES=0
  python train.py > train.log 2>&1 &
  

  bash scripts/run_train_gpu.sh 1 0
  

  The python command above will run in the background, you can view the results through the file train.log.

  After training, you'll get some checkpoint files under the folder ${save_checkpoint_path}/ckpt_0/ by default.

• train.log for flyingchairs
  to be added

Distributed Training

• running on GPU

  bash scripts/run_train_gpu.sh 8 0,1,2,3,4,5,6,7
  

  The above shell script will run distribute training in the background. You can view the results through the file train.log.

• train.log for flyingchairs
  to be added

Evaluation Process

Evaluation

• evaluation on MpiSintelClean dataset when running on GPU

  Before running the command below, please check the checkpoint path used for evaluation. Please set the checkpoint path to be the absolute full path, e.g., "path/flownet2/ckpt/flownet2-125_390.ckpt".

  python eval.py --eval_data=[DATASET_NAME] --eval_data_path=[DATASET_PATH]/
  --model=[MODEL_NAME] --eval_checkpoint_path=[CHECKPOINT_PATH] --crop_type=Center> eval.log 2>&1 &  
  

  The above python command will run in the background. You can view the results through the file "eval.log". The accuracy of the test dataset will be as follows:

  # grep "mean error: " eval.log
  flownet2 mean error: 2.12
  

  OR,

  bash run_eval_gpu.sh run_eval_gpu.sh [MpiSintelClean/MpiSintelFinal] [DATA_PATH] [MODEL_NAME] [CKPT_PATH] [DEVICE_ID]
  

  The above python command will run in the background. You can view the results through the file "eval/eval.log". The accuracy of the test dataset will be as follows:

  # grep "mean error: " eval.log
  flownet2 mean error: 2.12
  

Export Process

Export

Before export model, you must modify the config file,
The config items you should modify are batch_size and pre_trained_ckpt_path.

python export.py --config_path [CONFIG_PATH]

Model Description

Performance

Training Performance

FlowNet2 train on FlyingChairs

Parameters	GPU
Model Version	Inception V1
Resource	NV SMX2 V100-32G
uploaded Date	04/05/2021 (month/day/year)
MindSpore Version	1.7.0
Dataset	FlyingChairs
Training Parameters	epoch=50, steps=5000, batch_size=8, lr=1e-6
Optimizer	Adam
Loss Function	L1loss
outputs	flow
Speed	1pc: 152 ms/step; 8pcs: 171 ms/step
Total time	8pcs: 8.8 hours
Parameters	162,518,834
Checkpoint for Fine tuning	260M (.ckpt file)
Scripts	flownet2 script

Inference Performance

FlowNet2 infer on MpiSintelClean

Parameters	GPU
Model Version	Inception V1
Resource	NV SMX2 V100-32G
Uploaded Date	04/05/2022 (month/day/year)
MindSpore Version	1.7.0
Dataset	MpiSintelClean
batch_size	8
outputs	flow
Mean Error	2.12

ModelZoo Homepage

Please check the official homepage.