test
into main
@@ -459,6 +459,482 @@ We provide two visualization tools for debugging. | |||||
#### Speed and quality tradeoff | |||||
We have reported some ablation experiments in our paper supplementary material. | |||||
Setting `N_iters`, `N_rand`, `num_voxels`, `rgbnet_depth`, `rgbnet_width` to larger values or setting `stepsize` to smaller values typically leads to better quality but need more computation. | |||||
The `weight_distortion` affects the training speed and quality as well. | |||||
Only `stepsize` is tunable in testing phase, while all the other fields should remain the same as training. | |||||
## Advanced data structure | |||||
- **Octree** — [Plenoxels: Radiance Fields without Neural Networks](https://alexyu.net/plenoxels/). | |||||
- **Hash** — [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://nvlabs.github.io/instant-ngp/). | |||||
- **Factorized components** — [TensoRF: Tensorial Radiance Fields](https://apchenstu.github.io/TensoRF/). | |||||
You will need them for scaling to a higher grid resolution. But we believe our simplest dense grid could still be your good starting point if you have other challenging problems to deal with. | |||||
## Acknowledgement | |||||
The code base is origined from an awesome [nerf-pytorch](https://github.com/yenchenlin/nerf-pytorch) implementation, but it becomes very different from the code base now. | |||||
# DirectVoxGO | |||||
Direct Voxel Grid Optimization (CVPR2022 Oral, [project page](https://sunset1995.github.io/dvgo/), [DVGO paper](https://arxiv.org/abs/2111.11215), [DVGO v2 paper](https://arxiv.org/abs/2206.05085)). | |||||
https://user-images.githubusercontent.com/2712505/153380311-19d6c3a1-9130-489a-af16-ad36c78f10a9.mp4 | |||||
https://user-images.githubusercontent.com/2712505/153380197-991d1689-6418-499c-a192-d757f9a64b64.mp4 | |||||
### Custom casual capturing | |||||
A [short guide](https://sunset1995.github.io/dvgo/tutor_forward_facing.html) to capture custom forward-facing scenes and rendering fly-through videos. | |||||
Below are two rgb and depth fly-through videos from custom captured scenes. | |||||
https://user-images.githubusercontent.com/2712505/174267754-619d4f81-dd04-4c50-ba7f-434774cb890e.mp4 | |||||
### Features | |||||
- Speedup NeRF by replacing the MLP with the voxel grid. | |||||
- Simple scene representation: | |||||
- *Volume densities*: dense voxel grid (3D). | |||||
- *View-dependent colors*: dense feature grid (4D) + shallow MLP. | |||||
- Pytorch cuda extention built just-in-time for another 2--3x speedup. | |||||
- O(N) realization for the distortion loss proposed by [mip-nerf 360](https://jonbarron.info/mipnerf360/). | |||||
- The loss improves our training time and quality. | |||||
- We have released a self-contained pytorch package: [torch_efficient_distloss](https://github.com/sunset1995/torch_efficient_distloss). | |||||
- Consider a batch of 8192 rays X 256 points. | |||||
- GPU memory consumption: 6192MB => 96MB. | |||||
- Run times for 100 iters: 20 sec => 0.2sec. | |||||
- Supported datasets: | |||||
- *Bounded inward-facing*: [NeRF](https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1), [NSVF](https://dl.fbaipublicfiles.com/nsvf/dataset/Synthetic_NSVF.zip), [BlendedMVS](https://dl.fbaipublicfiles.com/nsvf/dataset/BlendedMVS.zip), [T&T (masked)](https://dl.fbaipublicfiles.com/nsvf/dataset/TanksAndTemple.zip), [DeepVoxels](https://drive.google.com/open?id=1ScsRlnzy9Bd_n-xw83SP-0t548v63mPH). | |||||
- *Unbounded inward-facing*: [T&T](https://drive.google.com/file/d/11KRfN91W1AxAW6lOFs4EeYDbeoQZCi87/view?usp=sharing), [LF](https://drive.google.com/file/d/1gsjDjkbTh4GAR9fFqlIDZ__qR9NYTURQ/view?usp=sharing), [mip-NeRF360](https://jonbarron.info/mipnerf360/). | |||||
- *Foward-facing*: [LLFF](https://drive.google.com/drive/folders/14boI-o5hGO9srnWaaogTU5_ji7wkX2S7). | |||||
### Installation | |||||
``` | |||||
git clone git@github.com:sunset1995/DirectVoxGO.git | |||||
cd DirectVoxGO | |||||
pip install -r requirements.txt | |||||
``` | |||||
[Pytorch](https://pytorch.org/) and [torch_scatter](https://github.com/rusty1s/pytorch_scatter) installation is machine dependent, please install the correct version for your machine. | |||||
<details> | |||||
<summary> Dependencies (click to expand) </summary> | |||||
- `PyTorch`, `numpy`, `torch_scatter`: main computation. | |||||
- `scipy`, `lpips`: SSIM and LPIPS evaluation. | |||||
- `tqdm`: progress bar. | |||||
- `mmcv`: config system. | |||||
- `opencv-python`: image processing. | |||||
- `imageio`, `imageio-ffmpeg`: images and videos I/O. | |||||
- `Ninja`: to build the newly implemented torch extention just-in-time. | |||||
- `einops`: torch tensor shaping with pretty api. | |||||
- `torch_efficient_distloss`: O(N) realization for the distortion loss. | |||||
</details> | |||||
## Directory structure for the datasets | |||||
<details> | |||||
<summary> (click to expand;) </summary> | |||||
data | |||||
├── nerf_synthetic # Link: https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1 | |||||
│ └── [chair|drums|ficus|hotdog|lego|materials|mic|ship] | |||||
│ ├── [train|val|test] | |||||
│ │ └── r_*.png | |||||
│ └── transforms_[train|val|test].json | |||||
│ | |||||
├── Synthetic_NSVF # Link: https://dl.fbaipublicfiles.com/nsvf/dataset/Synthetic_NSVF.zip | |||||
│ └── [Bike|Lifestyle|Palace|Robot|Spaceship|Steamtrain|Toad|Wineholder] | |||||
│ ├── intrinsics.txt | |||||
│ ├── rgb | |||||
│ │ └── [0_train|1_val|2_test]_*.png | |||||
│ └── pose | |||||
│ └── [0_train|1_val|2_test]_*.txt | |||||
│ | |||||
├── BlendedMVS # Link: https://dl.fbaipublicfiles.com/nsvf/dataset/BlendedMVS.zip | |||||
│ └── [Character|Fountain|Jade|Statues] | |||||
│ ├── intrinsics.txt | |||||
│ ├── rgb | |||||
│ │ └── [0|1|2]_*.png | |||||
│ └── pose | |||||
│ └── [0|1|2]_*.txt | |||||
│ | |||||
├── TanksAndTemple # Link: https://dl.fbaipublicfiles.com/nsvf/dataset/TanksAndTemple.zip | |||||
│ └── [Barn|Caterpillar|Family|Ignatius|Truck] | |||||
│ ├── intrinsics.txt | |||||
│ ├── rgb | |||||
│ │ └── [0|1|2]_*.png | |||||
│ └── pose | |||||
│ └── [0|1|2]_*.txt | |||||
│ | |||||
├── deepvoxels # Link: https://drive.google.com/drive/folders/1ScsRlnzy9Bd_n-xw83SP-0t548v63mPH | |||||
│ └── [train|validation|test] | |||||
│ └── [armchair|cube|greek|vase] | |||||
│ ├── intrinsics.txt | |||||
│ ├── rgb/*.png | |||||
│ └── pose/*.txt | |||||
│ | |||||
├── nerf_llff_data # Link: https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1 | |||||
│ └── [fern|flower|fortress|horns|leaves|orchids|room|trex] | |||||
│ | |||||
├── tanks_and_temples # Link: https://drive.google.com/file/d/11KRfN91W1AxAW6lOFs4EeYDbeoQZCi87/view?usp=sharing | |||||
│ └── [tat_intermediate_M60|tat_intermediate_Playground|tat_intermediate_Train|tat_training_Truck] | |||||
│ └── [train|test] | |||||
│ ├── intrinsics/*txt | |||||
│ ├── pose/*txt | |||||
│ └── rgb/*jpg | |||||
│ | |||||
├── lf_data # Link: https://drive.google.com/file/d/1gsjDjkbTh4GAR9fFqlIDZ__qR9NYTURQ/view?usp=sharing | |||||
│ └── [africa|basket|ship|statue|torch] | |||||
│ └── [train|test] | |||||
│ ├── intrinsics/*txt | |||||
│ ├── pose/*txt | |||||
│ └── rgb/*jpg | |||||
│ | |||||
├── 360_v2 # Link: https://jonbarron.info/mipnerf360/ | |||||
│ └── [bicycle|bonsai|counter|garden|kitchen|room|stump] | |||||
│ ├── poses_bounds.npy | |||||
│ └── [images_2|images_4] | |||||
│ | |||||
├── nerf_llff_data # Link: https://drive.google.com/drive/folders/14boI-o5hGO9srnWaaogTU5_ji7wkX2S7 | |||||
│ └── [fern|flower|fortress|horns|leaves|orchids|room|trex] | |||||
│ ├── poses_bounds.npy | |||||
│ └── [images_2|images_4] | |||||
│ | |||||
└── co3d # Link: https://github.com/facebookresearch/co3d | |||||
└── [donut|teddybear|umbrella|...] | |||||
├── frame_annotations.jgz | |||||
├── set_lists.json | |||||
└── [129_14950_29917|189_20376_35616|...] | |||||
├── images | |||||
│ └── frame*.jpg | |||||
└── masks | |||||
└── frame*.png | |||||
</details> | |||||
## GO | |||||
- Training | |||||
```bash | |||||
$ python run.py --config configs/nerf/lego.py --render_test | |||||
``` | |||||
Use `--i_print` and `--i_weights` to change the log interval. | |||||
- Evaluation | |||||
To only evaluate the testset `PSNR`, `SSIM`, and `LPIPS` of the trained `lego` without re-training, run: | |||||
```bash | |||||
$ python run.py --config configs/nerf/lego.py --render_only --render_test \ | |||||
--eval_ssim --eval_lpips_vgg | |||||
``` | |||||
Use `--eval_lpips_alex` to evaluate LPIPS with pre-trained Alex net instead of VGG net. | |||||
- Render video | |||||
```bash | |||||
$ python run.py --config configs/nerf/lego.py --render_only --render_video | |||||
``` | |||||
Use `--render_video_factor 4` for a fast preview. | |||||
- Reproduction: all config files to reproduce our results. | |||||
<details> | |||||
<summary> (click to expand) </summary> | |||||
$ ls configs/* | |||||
configs/blendedmvs: | |||||
Character.py Fountain.py Jade.py Statues.py | |||||
configs/nerf: | |||||
chair.py drums.py ficus.py hotdog.py lego.py materials.py mic.py ship.py | |||||
configs/nsvf: | |||||
Bike.py Lifestyle.py Palace.py Robot.py Spaceship.py Steamtrain.py Toad.py Wineholder.py | |||||
configs/tankstemple: | |||||
Barn.py Caterpillar.py Family.py Ignatius.py Truck.py | |||||
configs/deepvoxels: | |||||
armchair.py cube.py greek.py vase.py | |||||
configs/tankstemple_unbounded: | |||||
M60.py Playground.py Train.py Truck.py | |||||
configs/lf: | |||||
africa.py basket.py ship.py statue.py torch.py | |||||
configs/nerf_unbounded: | |||||
bicycle.py bonsai.py counter.py garden.py kitchen.py room.py stump.py | |||||
configs/llff: | |||||
fern.py flower.py fortress.py horns.py leaves.py orchids.py room.py trex.py | |||||
</details> | |||||
### Custom casually captured scenes | |||||
Coming soon hopefully. | |||||
### Development and tuning guide | |||||
#### Extention to new dataset | |||||
Adjusting the data related config fields to fit your camera coordinate system is recommend before implementing a new one. | |||||
We provide two visualization tools for debugging. | |||||
1. Inspect the camera and the allocated BBox. | |||||
- Export via `--export_bbox_and_cams_only {filename}.npz`: | |||||
```bash | |||||
python run.py --config configs/nerf/mic.py --export_bbox_and_cams_only cam_mic.npz | |||||
``` | |||||
- Visualize the result: | |||||
```bash | |||||
python tools/vis_train.py cam_mic.npz | |||||
``` | |||||
2. Inspect the learned geometry after coarse optimization. | |||||
- Export via `--export_coarse_only {filename}.npz` (assumed `coarse_last.tar` available in the train log): | |||||
```bash | |||||
python run.py --config configs/nerf/mic.py --export_coarse_only coarse_mic.npz | |||||
``` | |||||
- Visualize the result: | |||||
```bash | |||||
python tools/vis_volume.py coarse_mic.npz 0.001 --cam cam_mic.npz | |||||
``` | |||||
| Inspecting the cameras & BBox | Inspecting the learned coarse volume | | |||||
|:-:|:-:| | |||||
|![](figs/debug_cam_and_bbox.png)|![](figs/debug_coarse_volume.png)| | |||||
#### Speed and quality tradeoff | |||||
We have reported some ablation experiments in our paper supplementary material. | |||||
Setting `N_iters`, `N_rand`, `num_voxels`, `rgbnet_depth`, `rgbnet_width` to larger values or setting `stepsize` to smaller values typically leads to better quality but need more computation. | |||||
The `weight_distortion` affects the training speed and quality as well. | |||||
Only `stepsize` is tunable in testing phase, while all the other fields should remain the same as training. | |||||
## Advanced data structure | |||||
- **Octree** — [Plenoxels: Radiance Fields without Neural Networks](https://alexyu.net/plenoxels/). | |||||
- **Hash** — [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://nvlabs.github.io/instant-ngp/). | |||||
- **Factorized components** — [TensoRF: Tensorial Radiance Fields](https://apchenstu.github.io/TensoRF/). | |||||
You will need them for scaling to a higher grid resolution. But we believe our simplest dense grid could still be your good starting point if you have other challenging problems to deal with. | |||||
## Acknowledgement | |||||
The code base is origined from an awesome [nerf-pytorch](https://github.com/yenchenlin/nerf-pytorch) implementation, but it becomes very different from the code base now. | |||||
# DirectVoxGO | |||||
Direct Voxel Grid Optimization (CVPR2022 Oral, [project page](https://sunset1995.github.io/dvgo/), [DVGO paper](https://arxiv.org/abs/2111.11215), [DVGO v2 paper](https://arxiv.org/abs/2206.05085)). | |||||
https://user-images.githubusercontent.com/2712505/153380311-19d6c3a1-9130-489a-af16-ad36c78f10a9.mp4 | |||||
https://user-images.githubusercontent.com/2712505/153380197-991d1689-6418-499c-a192-d757f9a64b64.mp4 | |||||
### Custom casual capturing | |||||
A [short guide](https://sunset1995.github.io/dvgo/tutor_forward_facing.html) to capture custom forward-facing scenes and rendering fly-through videos. | |||||
Below are two rgb and depth fly-through videos from custom captured scenes. | |||||
https://user-images.githubusercontent.com/2712505/174267754-619d4f81-dd04-4c50-ba7f-434774cb890e.mp4 | |||||
### Features | |||||
- Speedup NeRF by replacing the MLP with the voxel grid. | |||||
- Simple scene representation: | |||||
- *Volume densities*: dense voxel grid (3D). | |||||
- *View-dependent colors*: dense feature grid (4D) + shallow MLP. | |||||
- Pytorch cuda extention built just-in-time for another 2--3x speedup. | |||||
- O(N) realization for the distortion loss proposed by [mip-nerf 360](https://jonbarron.info/mipnerf360/). | |||||
- The loss improves our training time and quality. | |||||
- We have released a self-contained pytorch package: [torch_efficient_distloss](https://github.com/sunset1995/torch_efficient_distloss). | |||||
- Consider a batch of 8192 rays X 256 points. | |||||
- GPU memory consumption: 6192MB => 96MB. | |||||
- Run times for 100 iters: 20 sec => 0.2sec. | |||||
- Supported datasets: | |||||
- *Bounded inward-facing*: [NeRF](https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1), [NSVF](https://dl.fbaipublicfiles.com/nsvf/dataset/Synthetic_NSVF.zip), [BlendedMVS](https://dl.fbaipublicfiles.com/nsvf/dataset/BlendedMVS.zip), [T&T (masked)](https://dl.fbaipublicfiles.com/nsvf/dataset/TanksAndTemple.zip), [DeepVoxels](https://drive.google.com/open?id=1ScsRlnzy9Bd_n-xw83SP-0t548v63mPH). | |||||
- *Unbounded inward-facing*: [T&T](https://drive.google.com/file/d/11KRfN91W1AxAW6lOFs4EeYDbeoQZCi87/view?usp=sharing), [LF](https://drive.google.com/file/d/1gsjDjkbTh4GAR9fFqlIDZ__qR9NYTURQ/view?usp=sharing), [mip-NeRF360](https://jonbarron.info/mipnerf360/). | |||||
- *Foward-facing*: [LLFF](https://drive.google.com/drive/folders/14boI-o5hGO9srnWaaogTU5_ji7wkX2S7). | |||||
### Installation | |||||
``` | |||||
git clone git@github.com:sunset1995/DirectVoxGO.git | |||||
cd DirectVoxGO | |||||
pip install -r requirements.txt | |||||
``` | |||||
[Pytorch](https://pytorch.org/) and [torch_scatter](https://github.com/rusty1s/pytorch_scatter) installation is machine dependent, please install the correct version for your machine. | |||||
<details> | |||||
<summary> Dependencies (click to expand) </summary> | |||||
- `PyTorch`, `numpy`, `torch_scatter`: main computation. | |||||
- `scipy`, `lpips`: SSIM and LPIPS evaluation. | |||||
- `tqdm`: progress bar. | |||||
- `mmcv`: config system. | |||||
- `opencv-python`: image processing. | |||||
- `imageio`, `imageio-ffmpeg`: images and videos I/O. | |||||
- `Ninja`: to build the newly implemented torch extention just-in-time. | |||||
- `einops`: torch tensor shaping with pretty api. | |||||
- `torch_efficient_distloss`: O(N) realization for the distortion loss. | |||||
</details> | |||||
## Directory structure for the datasets | |||||
<details> | |||||
<summary> (click to expand;) </summary> | |||||
data | |||||
├── nerf_synthetic # Link: https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1 | |||||
│ └── [chair|drums|ficus|hotdog|lego|materials|mic|ship] | |||||
│ ├── [train|val|test] | |||||
│ │ └── r_*.png | |||||
│ └── transforms_[train|val|test].json | |||||
│ | |||||
├── Synthetic_NSVF # Link: https://dl.fbaipublicfiles.com/nsvf/dataset/Synthetic_NSVF.zip | |||||
│ └── [Bike|Lifestyle|Palace|Robot|Spaceship|Steamtrain|Toad|Wineholder] | |||||
│ ├── intrinsics.txt | |||||
│ ├── rgb | |||||
│ │ └── [0_train|1_val|2_test]_*.png | |||||
│ └── pose | |||||
│ └── [0_train|1_val|2_test]_*.txt | |||||
│ | |||||
├── BlendedMVS # Link: https://dl.fbaipublicfiles.com/nsvf/dataset/BlendedMVS.zip | |||||
│ └── [Character|Fountain|Jade|Statues] | |||||
│ ├── intrinsics.txt | |||||
│ ├── rgb | |||||
│ │ └── [0|1|2]_*.png | |||||
│ └── pose | |||||
│ └── [0|1|2]_*.txt | |||||
│ | |||||
├── TanksAndTemple # Link: https://dl.fbaipublicfiles.com/nsvf/dataset/TanksAndTemple.zip | |||||
│ └── [Barn|Caterpillar|Family|Ignatius|Truck] | |||||
│ ├── intrinsics.txt | |||||
│ ├── rgb | |||||
│ │ └── [0|1|2]_*.png | |||||
│ └── pose | |||||
│ └── [0|1|2]_*.txt | |||||
│ | |||||
├── deepvoxels # Link: https://drive.google.com/drive/folders/1ScsRlnzy9Bd_n-xw83SP-0t548v63mPH | |||||
│ └── [train|validation|test] | |||||
│ └── [armchair|cube|greek|vase] | |||||
│ ├── intrinsics.txt | |||||
│ ├── rgb/*.png | |||||
│ └── pose/*.txt | |||||
│ | |||||
├── nerf_llff_data # Link: https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1 | |||||
│ └── [fern|flower|fortress|horns|leaves|orchids|room|trex] | |||||
│ | |||||
├── tanks_and_temples # Link: https://drive.google.com/file/d/11KRfN91W1AxAW6lOFs4EeYDbeoQZCi87/view?usp=sharing | |||||
│ └── [tat_intermediate_M60|tat_intermediate_Playground|tat_intermediate_Train|tat_training_Truck] | |||||
│ └── [train|test] | |||||
│ ├── intrinsics/*txt | |||||
│ ├── pose/*txt | |||||
│ └── rgb/*jpg | |||||
│ | |||||
├── lf_data # Link: https://drive.google.com/file/d/1gsjDjkbTh4GAR9fFqlIDZ__qR9NYTURQ/view?usp=sharing | |||||
│ └── [africa|basket|ship|statue|torch] | |||||
│ └── [train|test] | |||||
│ ├── intrinsics/*txt | |||||
│ ├── pose/*txt | |||||
│ └── rgb/*jpg | |||||
│ | |||||
├── 360_v2 # Link: https://jonbarron.info/mipnerf360/ | |||||
│ └── [bicycle|bonsai|counter|garden|kitchen|room|stump] | |||||
│ ├── poses_bounds.npy | |||||
│ └── [images_2|images_4] | |||||
│ | |||||
├── nerf_llff_data # Link: https://drive.google.com/drive/folders/14boI-o5hGO9srnWaaogTU5_ji7wkX2S7 | |||||
│ └── [fern|flower|fortress|horns|leaves|orchids|room|trex] | |||||
│ ├── poses_bounds.npy | |||||
│ └── [images_2|images_4] | |||||
│ | |||||
└── co3d # Link: https://github.com/facebookresearch/co3d | |||||
└── [donut|teddybear|umbrella|...] | |||||
├── frame_annotations.jgz | |||||
├── set_lists.json | |||||
└── [129_14950_29917|189_20376_35616|...] | |||||
├── images | |||||
│ └── frame*.jpg | |||||
└── masks | |||||
└── frame*.png | |||||
</details> | |||||
## GO | |||||
- Training | |||||
```bash | |||||
$ python run.py --config configs/nerf/lego.py --render_test | |||||
``` | |||||
Use `--i_print` and `--i_weights` to change the log interval. | |||||
- Evaluation | |||||
To only evaluate the testset `PSNR`, `SSIM`, and `LPIPS` of the trained `lego` without re-training, run: | |||||
```bash | |||||
$ python run.py --config configs/nerf/lego.py --render_only --render_test \ | |||||
--eval_ssim --eval_lpips_vgg | |||||
``` | |||||
Use `--eval_lpips_alex` to evaluate LPIPS with pre-trained Alex net instead of VGG net. | |||||
- Render video | |||||
```bash | |||||
$ python run.py --config configs/nerf/lego.py --render_only --render_video | |||||
``` | |||||
Use `--render_video_factor 4` for a fast preview. | |||||
- Reproduction: all config files to reproduce our results. | |||||
<details> | |||||
<summary> (click to expand) </summary> | |||||
$ ls configs/* | |||||
configs/blendedmvs: | |||||
Character.py Fountain.py Jade.py Statues.py | |||||
configs/nerf: | |||||
chair.py drums.py ficus.py hotdog.py lego.py materials.py mic.py ship.py | |||||
configs/nsvf: | |||||
Bike.py Lifestyle.py Palace.py Robot.py Spaceship.py Steamtrain.py Toad.py Wineholder.py | |||||
configs/tankstemple: | |||||
Barn.py Caterpillar.py Family.py Ignatius.py Truck.py | |||||
configs/deepvoxels: | |||||
armchair.py cube.py greek.py vase.py | |||||
configs/tankstemple_unbounded: | |||||
M60.py Playground.py Train.py Truck.py | |||||
configs/lf: | |||||
africa.py basket.py ship.py statue.py torch.py | |||||
configs/nerf_unbounded: | |||||
bicycle.py bonsai.py counter.py garden.py kitchen.py room.py stump.py | |||||
configs/llff: | |||||
fern.py flower.py fortress.py horns.py leaves.py orchids.py room.py trex.py | |||||
</details> | |||||
### Custom casually captured scenes | |||||
Coming soon hopefully. | |||||
### Development and tuning guide | |||||
#### Extention to new dataset | |||||
Adjusting the data related config fields to fit your camera coordinate system is recommend before implementing a new one. | |||||
We provide two visualization tools for debugging. | |||||
1. Inspect the camera and the allocated BBox. | |||||
- Export via `--export_bbox_and_cams_only {filename}.npz`: | |||||
```bash | |||||
python run.py --config configs/nerf/mic.py --export_bbox_and_cams_only cam_mic.npz | |||||
``` | |||||
- Visualize the result: | |||||
```bash | |||||
python tools/vis_train.py cam_mic.npz | |||||
``` | |||||
2. Inspect the learned geometry after coarse optimization. | |||||
- Export via `--export_coarse_only {filename}.npz` (assumed `coarse_last.tar` available in the train log): | |||||
```bash | |||||
python run.py --config configs/nerf/mic.py --export_coarse_only coarse_mic.npz | |||||
``` | |||||
- Visualize the result: | |||||
```bash | |||||
python tools/vis_volume.py coarse_mic.npz 0.001 --cam cam_mic.npz | |||||
``` | |||||
| Inspecting the cameras & BBox | Inspecting the learned coarse volume | | |||||
|:-:|:-:| | |||||
|![](figs/debug_cam_and_bbox.png)|![](figs/debug_coarse_volume.png)| | |||||
#### Speed and quality tradeoff | #### Speed and quality tradeoff | ||||
We have reported some ablation experiments in our paper supplementary material. | We have reported some ablation experiments in our paper supplementary material. | ||||
Setting `N_iters`, `N_rand`, `num_voxels`, `rgbnet_depth`, `rgbnet_width` to larger values or setting `stepsize` to smaller values typically leads to better quality but need more computation. | Setting `N_iters`, `N_rand`, `num_voxels`, `rgbnet_depth`, `rgbnet_width` to larger values or setting `stepsize` to smaller values typically leads to better quality but need more computation. | ||||
Dear OpenI User
Thank you for your continuous support to the Openl Qizhi Community AI Collaboration Platform. In order to protect your usage rights and ensure network security, we updated the Openl Qizhi Community AI Collaboration Platform Usage Agreement in January 2024. The updated agreement specifies that users are prohibited from using intranet penetration tools. After you click "Agree and continue", you can continue to use our services. Thank you for your cooperation and understanding.
For more agreement content, please refer to the《Openl Qizhi Community AI Collaboration Platform Usage Agreement》