mmediting/configs/video_interpolators/tof

• TOFlow (IJCV'2019)
• • Abstract
  • Results and models
  • Citation

TOFlow (IJCV'2019)

Video Enhancement with Task-Oriented Flow

Abstract

Many video enhancement algorithms rely on optical flow to register frames in a video sequence. Precise flow estimation is however intractable; and optical flow itself is often a sub-optimal representation for particular video processing tasks. In this paper, we propose task-oriented flow (TOFlow), a motion representation learned in a self-supervised, task-specific manner. We design a neural network with a trainable motion estimation component and a video processing component, and train them jointly to learn the task-oriented flow. For evaluation, we build Vimeo-90K, a large-scale, high-quality video dataset for low-level video processing. TOFlow outperforms traditional optical flow on standard benchmarks as well as our Vimeo-90K dataset in three video processing tasks: frame interpolation, video denoising/deblocking, and video super-resolution.

Results and models

Evaluated on RGB channels.
The metrics are PSNR / SSIM .

Method	Pretrained SPyNet	Vimeo90k-triplet	Download
tof_vfi_spynet_chair_nobn_1xb1_vimeo90k	spynet_chairs_final	33.3294 / 0.9465	model | log
tof_vfi_spynet_kitti_nobn_1xb1_vimeo90k	spynet_chairs_final	33.3339 / 0.9466	model | log
tof_vfi_spynet_sintel_clean_nobn_1xb1_vimeo90k	spynet_chairs_final	33.3170 / 0.9464	model | log
tof_vfi_spynet_sintel_final_nobn_1xb1_vimeo90k	spynet_chairs_final	33.3237 / 0.9465	model | log
tof_vfi_spynet_pytoflow_nobn_1xb1_vimeo90k	spynet_chairs_final	33.3426 / 0.9467	model | log

Note: These pretrained SPyNets don't contain BN layer since batch_size=1, which is consistent with https://github.com/Coldog2333/pytoflow.

Citation

@article{xue2019video,
  title={Video enhancement with task-oriented flow},
  author={Xue, Tianfan and Chen, Baian and Wu, Jiajun and Wei, Donglai and Freeman, William T},
  journal={International Journal of Computer Vision},
  volume={127},
  number={8},
  pages={1106--1125},
  year={2019},
  publisher={Springer}
}