#1 更新 'README.md'

Open
thefartchild wants to merge 1 commits from test into main
  1. +476
    -0
      README.md

+ 476
- 0
README.md View File

@@ -459,6 +459,482 @@ We provide two visualization tools for debugging.



#### Speed and quality tradeoff
We have reported some ablation experiments in our paper supplementary material.
Setting `N_iters`, `N_rand`, `num_voxels`, `rgbnet_depth`, `rgbnet_width` to larger values or setting `stepsize` to smaller values typically leads to better quality but need more computation.
The `weight_distortion` affects the training speed and quality as well.
Only `stepsize` is tunable in testing phase, while all the other fields should remain the same as training.

## Advanced data structure
- **Octree** — [Plenoxels: Radiance Fields without Neural Networks](https://alexyu.net/plenoxels/).
- **Hash** — [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://nvlabs.github.io/instant-ngp/).
- **Factorized components** — [TensoRF: Tensorial Radiance Fields](https://apchenstu.github.io/TensoRF/).

You will need them for scaling to a higher grid resolution. But we believe our simplest dense grid could still be your good starting point if you have other challenging problems to deal with.

## Acknowledgement
The code base is origined from an awesome [nerf-pytorch](https://github.com/yenchenlin/nerf-pytorch) implementation, but it becomes very different from the code base now.
# DirectVoxGO

Direct Voxel Grid Optimization (CVPR2022 Oral, [project page](https://sunset1995.github.io/dvgo/), [DVGO paper](https://arxiv.org/abs/2111.11215), [DVGO v2 paper](https://arxiv.org/abs/2206.05085)).

https://user-images.githubusercontent.com/2712505/153380311-19d6c3a1-9130-489a-af16-ad36c78f10a9.mp4

https://user-images.githubusercontent.com/2712505/153380197-991d1689-6418-499c-a192-d757f9a64b64.mp4

### Custom casual capturing
A [short guide](https://sunset1995.github.io/dvgo/tutor_forward_facing.html) to capture custom forward-facing scenes and rendering fly-through videos.

Below are two rgb and depth fly-through videos from custom captured scenes.

https://user-images.githubusercontent.com/2712505/174267754-619d4f81-dd04-4c50-ba7f-434774cb890e.mp4

### Features
- Speedup NeRF by replacing the MLP with the voxel grid.
- Simple scene representation:
- *Volume densities*: dense voxel grid (3D).
- *View-dependent colors*: dense feature grid (4D) + shallow MLP.
- Pytorch cuda extention built just-in-time for another 2--3x speedup.
- O(N) realization for the distortion loss proposed by [mip-nerf 360](https://jonbarron.info/mipnerf360/).
- The loss improves our training time and quality.
- We have released a self-contained pytorch package: [torch_efficient_distloss](https://github.com/sunset1995/torch_efficient_distloss).
- Consider a batch of 8192 rays X 256 points.
- GPU memory consumption: 6192MB => 96MB.
- Run times for 100 iters: 20 sec => 0.2sec.
- Supported datasets:
- *Bounded inward-facing*: [NeRF](https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1), [NSVF](https://dl.fbaipublicfiles.com/nsvf/dataset/Synthetic_NSVF.zip), [BlendedMVS](https://dl.fbaipublicfiles.com/nsvf/dataset/BlendedMVS.zip), [T&T (masked)](https://dl.fbaipublicfiles.com/nsvf/dataset/TanksAndTemple.zip), [DeepVoxels](https://drive.google.com/open?id=1ScsRlnzy9Bd_n-xw83SP-0t548v63mPH).
- *Unbounded inward-facing*: [T&T](https://drive.google.com/file/d/11KRfN91W1AxAW6lOFs4EeYDbeoQZCi87/view?usp=sharing), [LF](https://drive.google.com/file/d/1gsjDjkbTh4GAR9fFqlIDZ__qR9NYTURQ/view?usp=sharing), [mip-NeRF360](https://jonbarron.info/mipnerf360/).
- *Foward-facing*: [LLFF](https://drive.google.com/drive/folders/14boI-o5hGO9srnWaaogTU5_ji7wkX2S7).


### Installation
```
git clone git@github.com:sunset1995/DirectVoxGO.git
cd DirectVoxGO
pip install -r requirements.txt
```
[Pytorch](https://pytorch.org/) and [torch_scatter](https://github.com/rusty1s/pytorch_scatter) installation is machine dependent, please install the correct version for your machine.

<details>
<summary> Dependencies (click to expand) </summary>

- `PyTorch`, `numpy`, `torch_scatter`: main computation.
- `scipy`, `lpips`: SSIM and LPIPS evaluation.
- `tqdm`: progress bar.
- `mmcv`: config system.
- `opencv-python`: image processing.
- `imageio`, `imageio-ffmpeg`: images and videos I/O.
- `Ninja`: to build the newly implemented torch extention just-in-time.
- `einops`: torch tensor shaping with pretty api.
- `torch_efficient_distloss`: O(N) realization for the distortion loss.
</details>


## Directory structure for the datasets

<details>
<summary> (click to expand;) </summary>

data
├── nerf_synthetic # Link: https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1
│ └── [chair|drums|ficus|hotdog|lego|materials|mic|ship]
│ ├── [train|val|test]
│ │ └── r_*.png
│ └── transforms_[train|val|test].json
├── Synthetic_NSVF # Link: https://dl.fbaipublicfiles.com/nsvf/dataset/Synthetic_NSVF.zip
│ └── [Bike|Lifestyle|Palace|Robot|Spaceship|Steamtrain|Toad|Wineholder]
│ ├── intrinsics.txt
│ ├── rgb
│ │ └── [0_train|1_val|2_test]_*.png
│ └── pose
│ └── [0_train|1_val|2_test]_*.txt
├── BlendedMVS # Link: https://dl.fbaipublicfiles.com/nsvf/dataset/BlendedMVS.zip
│ └── [Character|Fountain|Jade|Statues]
│ ├── intrinsics.txt
│ ├── rgb
│ │ └── [0|1|2]_*.png
│ └── pose
│ └── [0|1|2]_*.txt
├── TanksAndTemple # Link: https://dl.fbaipublicfiles.com/nsvf/dataset/TanksAndTemple.zip
│ └── [Barn|Caterpillar|Family|Ignatius|Truck]
│ ├── intrinsics.txt
│ ├── rgb
│ │ └── [0|1|2]_*.png
│ └── pose
│ └── [0|1|2]_*.txt
├── deepvoxels # Link: https://drive.google.com/drive/folders/1ScsRlnzy9Bd_n-xw83SP-0t548v63mPH
│ └── [train|validation|test]
│ └── [armchair|cube|greek|vase]
│ ├── intrinsics.txt
│ ├── rgb/*.png
│ └── pose/*.txt
├── nerf_llff_data # Link: https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1
│ └── [fern|flower|fortress|horns|leaves|orchids|room|trex]
├── tanks_and_temples # Link: https://drive.google.com/file/d/11KRfN91W1AxAW6lOFs4EeYDbeoQZCi87/view?usp=sharing
│ └── [tat_intermediate_M60|tat_intermediate_Playground|tat_intermediate_Train|tat_training_Truck]
│ └── [train|test]
│ ├── intrinsics/*txt
│ ├── pose/*txt
│ └── rgb/*jpg
├── lf_data # Link: https://drive.google.com/file/d/1gsjDjkbTh4GAR9fFqlIDZ__qR9NYTURQ/view?usp=sharing
│ └── [africa|basket|ship|statue|torch]
│ └── [train|test]
│ ├── intrinsics/*txt
│ ├── pose/*txt
│ └── rgb/*jpg
├── 360_v2 # Link: https://jonbarron.info/mipnerf360/
│ └── [bicycle|bonsai|counter|garden|kitchen|room|stump]
│ ├── poses_bounds.npy
│ └── [images_2|images_4]
├── nerf_llff_data # Link: https://drive.google.com/drive/folders/14boI-o5hGO9srnWaaogTU5_ji7wkX2S7
│ └── [fern|flower|fortress|horns|leaves|orchids|room|trex]
│ ├── poses_bounds.npy
│ └── [images_2|images_4]
└── co3d # Link: https://github.com/facebookresearch/co3d
└── [donut|teddybear|umbrella|...]
├── frame_annotations.jgz
├── set_lists.json
└── [129_14950_29917|189_20376_35616|...]
├── images
│ └── frame*.jpg
└── masks
└── frame*.png
</details>



## GO

- Training
```bash
$ python run.py --config configs/nerf/lego.py --render_test
```
Use `--i_print` and `--i_weights` to change the log interval.
- Evaluation
To only evaluate the testset `PSNR`, `SSIM`, and `LPIPS` of the trained `lego` without re-training, run:
```bash
$ python run.py --config configs/nerf/lego.py --render_only --render_test \
--eval_ssim --eval_lpips_vgg
```
Use `--eval_lpips_alex` to evaluate LPIPS with pre-trained Alex net instead of VGG net.
- Render video
```bash
$ python run.py --config configs/nerf/lego.py --render_only --render_video
```
Use `--render_video_factor 4` for a fast preview.
- Reproduction: all config files to reproduce our results.
<details>
<summary> (click to expand) </summary>

$ ls configs/*
configs/blendedmvs:
Character.py Fountain.py Jade.py Statues.py

configs/nerf:
chair.py drums.py ficus.py hotdog.py lego.py materials.py mic.py ship.py

configs/nsvf:
Bike.py Lifestyle.py Palace.py Robot.py Spaceship.py Steamtrain.py Toad.py Wineholder.py

configs/tankstemple:
Barn.py Caterpillar.py Family.py Ignatius.py Truck.py

configs/deepvoxels:
armchair.py cube.py greek.py vase.py

configs/tankstemple_unbounded:
M60.py Playground.py Train.py Truck.py

configs/lf:
africa.py basket.py ship.py statue.py torch.py

configs/nerf_unbounded:
bicycle.py bonsai.py counter.py garden.py kitchen.py room.py stump.py

configs/llff:
fern.py flower.py fortress.py horns.py leaves.py orchids.py room.py trex.py
</details>

### Custom casually captured scenes
Coming soon hopefully.

### Development and tuning guide
#### Extention to new dataset
Adjusting the data related config fields to fit your camera coordinate system is recommend before implementing a new one.
We provide two visualization tools for debugging.
1. Inspect the camera and the allocated BBox.
- Export via `--export_bbox_and_cams_only {filename}.npz`:
```bash
python run.py --config configs/nerf/mic.py --export_bbox_and_cams_only cam_mic.npz
```
- Visualize the result:
```bash
python tools/vis_train.py cam_mic.npz
```
2. Inspect the learned geometry after coarse optimization.
- Export via `--export_coarse_only {filename}.npz` (assumed `coarse_last.tar` available in the train log):
```bash
python run.py --config configs/nerf/mic.py --export_coarse_only coarse_mic.npz
```
- Visualize the result:
```bash
python tools/vis_volume.py coarse_mic.npz 0.001 --cam cam_mic.npz
```

| Inspecting the cameras & BBox | Inspecting the learned coarse volume |
|:-:|:-:|
|![](figs/debug_cam_and_bbox.png)|![](figs/debug_coarse_volume.png)|



#### Speed and quality tradeoff
We have reported some ablation experiments in our paper supplementary material.
Setting `N_iters`, `N_rand`, `num_voxels`, `rgbnet_depth`, `rgbnet_width` to larger values or setting `stepsize` to smaller values typically leads to better quality but need more computation.
The `weight_distortion` affects the training speed and quality as well.
Only `stepsize` is tunable in testing phase, while all the other fields should remain the same as training.

## Advanced data structure
- **Octree** — [Plenoxels: Radiance Fields without Neural Networks](https://alexyu.net/plenoxels/).
- **Hash** — [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://nvlabs.github.io/instant-ngp/).
- **Factorized components** — [TensoRF: Tensorial Radiance Fields](https://apchenstu.github.io/TensoRF/).

You will need them for scaling to a higher grid resolution. But we believe our simplest dense grid could still be your good starting point if you have other challenging problems to deal with.

## Acknowledgement
The code base is origined from an awesome [nerf-pytorch](https://github.com/yenchenlin/nerf-pytorch) implementation, but it becomes very different from the code base now.
# DirectVoxGO

Direct Voxel Grid Optimization (CVPR2022 Oral, [project page](https://sunset1995.github.io/dvgo/), [DVGO paper](https://arxiv.org/abs/2111.11215), [DVGO v2 paper](https://arxiv.org/abs/2206.05085)).

https://user-images.githubusercontent.com/2712505/153380311-19d6c3a1-9130-489a-af16-ad36c78f10a9.mp4

https://user-images.githubusercontent.com/2712505/153380197-991d1689-6418-499c-a192-d757f9a64b64.mp4

### Custom casual capturing
A [short guide](https://sunset1995.github.io/dvgo/tutor_forward_facing.html) to capture custom forward-facing scenes and rendering fly-through videos.

Below are two rgb and depth fly-through videos from custom captured scenes.

https://user-images.githubusercontent.com/2712505/174267754-619d4f81-dd04-4c50-ba7f-434774cb890e.mp4

### Features
- Speedup NeRF by replacing the MLP with the voxel grid.
- Simple scene representation:
- *Volume densities*: dense voxel grid (3D).
- *View-dependent colors*: dense feature grid (4D) + shallow MLP.
- Pytorch cuda extention built just-in-time for another 2--3x speedup.
- O(N) realization for the distortion loss proposed by [mip-nerf 360](https://jonbarron.info/mipnerf360/).
- The loss improves our training time and quality.
- We have released a self-contained pytorch package: [torch_efficient_distloss](https://github.com/sunset1995/torch_efficient_distloss).
- Consider a batch of 8192 rays X 256 points.
- GPU memory consumption: 6192MB => 96MB.
- Run times for 100 iters: 20 sec => 0.2sec.
- Supported datasets:
- *Bounded inward-facing*: [NeRF](https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1), [NSVF](https://dl.fbaipublicfiles.com/nsvf/dataset/Synthetic_NSVF.zip), [BlendedMVS](https://dl.fbaipublicfiles.com/nsvf/dataset/BlendedMVS.zip), [T&T (masked)](https://dl.fbaipublicfiles.com/nsvf/dataset/TanksAndTemple.zip), [DeepVoxels](https://drive.google.com/open?id=1ScsRlnzy9Bd_n-xw83SP-0t548v63mPH).
- *Unbounded inward-facing*: [T&T](https://drive.google.com/file/d/11KRfN91W1AxAW6lOFs4EeYDbeoQZCi87/view?usp=sharing), [LF](https://drive.google.com/file/d/1gsjDjkbTh4GAR9fFqlIDZ__qR9NYTURQ/view?usp=sharing), [mip-NeRF360](https://jonbarron.info/mipnerf360/).
- *Foward-facing*: [LLFF](https://drive.google.com/drive/folders/14boI-o5hGO9srnWaaogTU5_ji7wkX2S7).


### Installation
```
git clone git@github.com:sunset1995/DirectVoxGO.git
cd DirectVoxGO
pip install -r requirements.txt
```
[Pytorch](https://pytorch.org/) and [torch_scatter](https://github.com/rusty1s/pytorch_scatter) installation is machine dependent, please install the correct version for your machine.

<details>
<summary> Dependencies (click to expand) </summary>

- `PyTorch`, `numpy`, `torch_scatter`: main computation.
- `scipy`, `lpips`: SSIM and LPIPS evaluation.
- `tqdm`: progress bar.
- `mmcv`: config system.
- `opencv-python`: image processing.
- `imageio`, `imageio-ffmpeg`: images and videos I/O.
- `Ninja`: to build the newly implemented torch extention just-in-time.
- `einops`: torch tensor shaping with pretty api.
- `torch_efficient_distloss`: O(N) realization for the distortion loss.
</details>


## Directory structure for the datasets

<details>
<summary> (click to expand;) </summary>

data
├── nerf_synthetic # Link: https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1
│ └── [chair|drums|ficus|hotdog|lego|materials|mic|ship]
│ ├── [train|val|test]
│ │ └── r_*.png
│ └── transforms_[train|val|test].json
├── Synthetic_NSVF # Link: https://dl.fbaipublicfiles.com/nsvf/dataset/Synthetic_NSVF.zip
│ └── [Bike|Lifestyle|Palace|Robot|Spaceship|Steamtrain|Toad|Wineholder]
│ ├── intrinsics.txt
│ ├── rgb
│ │ └── [0_train|1_val|2_test]_*.png
│ └── pose
│ └── [0_train|1_val|2_test]_*.txt
├── BlendedMVS # Link: https://dl.fbaipublicfiles.com/nsvf/dataset/BlendedMVS.zip
│ └── [Character|Fountain|Jade|Statues]
│ ├── intrinsics.txt
│ ├── rgb
│ │ └── [0|1|2]_*.png
│ └── pose
│ └── [0|1|2]_*.txt
├── TanksAndTemple # Link: https://dl.fbaipublicfiles.com/nsvf/dataset/TanksAndTemple.zip
│ └── [Barn|Caterpillar|Family|Ignatius|Truck]
│ ├── intrinsics.txt
│ ├── rgb
│ │ └── [0|1|2]_*.png
│ └── pose
│ └── [0|1|2]_*.txt
├── deepvoxels # Link: https://drive.google.com/drive/folders/1ScsRlnzy9Bd_n-xw83SP-0t548v63mPH
│ └── [train|validation|test]
│ └── [armchair|cube|greek|vase]
│ ├── intrinsics.txt
│ ├── rgb/*.png
│ └── pose/*.txt
├── nerf_llff_data # Link: https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1
│ └── [fern|flower|fortress|horns|leaves|orchids|room|trex]
├── tanks_and_temples # Link: https://drive.google.com/file/d/11KRfN91W1AxAW6lOFs4EeYDbeoQZCi87/view?usp=sharing
│ └── [tat_intermediate_M60|tat_intermediate_Playground|tat_intermediate_Train|tat_training_Truck]
│ └── [train|test]
│ ├── intrinsics/*txt
│ ├── pose/*txt
│ └── rgb/*jpg
├── lf_data # Link: https://drive.google.com/file/d/1gsjDjkbTh4GAR9fFqlIDZ__qR9NYTURQ/view?usp=sharing
│ └── [africa|basket|ship|statue|torch]
│ └── [train|test]
│ ├── intrinsics/*txt
│ ├── pose/*txt
│ └── rgb/*jpg
├── 360_v2 # Link: https://jonbarron.info/mipnerf360/
│ └── [bicycle|bonsai|counter|garden|kitchen|room|stump]
│ ├── poses_bounds.npy
│ └── [images_2|images_4]
├── nerf_llff_data # Link: https://drive.google.com/drive/folders/14boI-o5hGO9srnWaaogTU5_ji7wkX2S7
│ └── [fern|flower|fortress|horns|leaves|orchids|room|trex]
│ ├── poses_bounds.npy
│ └── [images_2|images_4]
└── co3d # Link: https://github.com/facebookresearch/co3d
└── [donut|teddybear|umbrella|...]
├── frame_annotations.jgz
├── set_lists.json
└── [129_14950_29917|189_20376_35616|...]
├── images
│ └── frame*.jpg
└── masks
└── frame*.png
</details>



## GO

- Training
```bash
$ python run.py --config configs/nerf/lego.py --render_test
```
Use `--i_print` and `--i_weights` to change the log interval.
- Evaluation
To only evaluate the testset `PSNR`, `SSIM`, and `LPIPS` of the trained `lego` without re-training, run:
```bash
$ python run.py --config configs/nerf/lego.py --render_only --render_test \
--eval_ssim --eval_lpips_vgg
```
Use `--eval_lpips_alex` to evaluate LPIPS with pre-trained Alex net instead of VGG net.
- Render video
```bash
$ python run.py --config configs/nerf/lego.py --render_only --render_video
```
Use `--render_video_factor 4` for a fast preview.
- Reproduction: all config files to reproduce our results.
<details>
<summary> (click to expand) </summary>

$ ls configs/*
configs/blendedmvs:
Character.py Fountain.py Jade.py Statues.py

configs/nerf:
chair.py drums.py ficus.py hotdog.py lego.py materials.py mic.py ship.py

configs/nsvf:
Bike.py Lifestyle.py Palace.py Robot.py Spaceship.py Steamtrain.py Toad.py Wineholder.py

configs/tankstemple:
Barn.py Caterpillar.py Family.py Ignatius.py Truck.py

configs/deepvoxels:
armchair.py cube.py greek.py vase.py

configs/tankstemple_unbounded:
M60.py Playground.py Train.py Truck.py

configs/lf:
africa.py basket.py ship.py statue.py torch.py

configs/nerf_unbounded:
bicycle.py bonsai.py counter.py garden.py kitchen.py room.py stump.py

configs/llff:
fern.py flower.py fortress.py horns.py leaves.py orchids.py room.py trex.py
</details>

### Custom casually captured scenes
Coming soon hopefully.

### Development and tuning guide
#### Extention to new dataset
Adjusting the data related config fields to fit your camera coordinate system is recommend before implementing a new one.
We provide two visualization tools for debugging.
1. Inspect the camera and the allocated BBox.
- Export via `--export_bbox_and_cams_only {filename}.npz`:
```bash
python run.py --config configs/nerf/mic.py --export_bbox_and_cams_only cam_mic.npz
```
- Visualize the result:
```bash
python tools/vis_train.py cam_mic.npz
```
2. Inspect the learned geometry after coarse optimization.
- Export via `--export_coarse_only {filename}.npz` (assumed `coarse_last.tar` available in the train log):
```bash
python run.py --config configs/nerf/mic.py --export_coarse_only coarse_mic.npz
```
- Visualize the result:
```bash
python tools/vis_volume.py coarse_mic.npz 0.001 --cam cam_mic.npz
```

| Inspecting the cameras & BBox | Inspecting the learned coarse volume |
|:-:|:-:|
|![](figs/debug_cam_and_bbox.png)|![](figs/debug_coarse_volume.png)|



#### Speed and quality tradeoff
We have reported some ablation experiments in our paper supplementary material.
Setting `N_iters`, `N_rand`, `num_voxels`, `rgbnet_depth`, `rgbnet_width` to larger values or setting `stepsize` to smaller values typically leads to better quality but need more computation.


Loading…
Cancel
Save