NeRF Applications

KAIST CS479: Machine Learning for 3D Data

Instructor: Minhyuk Sung (mhsung [at] kaist.ac.kr)
TA: Yuseung Lee (phillip0701 [at] kaist.ac.kr)
Credit: Jaihoon Kim (jh27kim [at] kaist.ac.kr)

Abstract

NOTE: Most of the materials we provide here are heavily dependent on opensource framework.

• NeRFStudio: NeRFStudio is a opensource framework where you can try out various NeRF models
• SDFStudio: SDFStudio is built on top of NeRFStudio and provides various neural implicit surface reconstruction models.
• ThreeStudio: ThreeStudio is a framework for creating 3D contetns. They provide zero-shot, one-shot and few-shot generation models.

The goal of this task is for you to get familar with opensource NeRF based frameworks. The scope of NeRF research is too large to cover and here we have listed a few. Most of the tasks in this week are focused on installation, training and visualizing the results, and we left the most challenging task at the end, feel free to check out.

Table of Content

• Abstract
• Setup
• Tasks
  • Task 0. Run NeRFacto
  • Task 1. Try your own data
  • Task 2. Try other methods: Instant-NGP, Mip-NeRF, TensoRF
  • Task 3. Run NeuS
  • Task 4. Export mesh
  • Task 5. Run DreamFusion
  • Task 6. Try other methods: Magic3D, ProlificDreamer
  • Task 7. Extend framework
• Further Readings

Setup

If you have troubles with the latest version please refer to the commit ID which we employed.

• NeRFStudio: d78af5b5699cb343cb46b5163d464e455524040f
• SDFStudio: ef14f965296a8e3b37ce21e51f1256be4868038f
• ThreeStudio: 74ff18a3810cdea39a5341ef8652d4fdc3ffde9f

For task 7, we advise you to make your own environment.

conda create --name {ENVIRONMENT_NAME} -y python=3.8
conda activate {ENVIRONMENT_NAME}
python -m pip install --upgrade pip

# torch1.12.1+cu113
pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 --extra-index-url https://download.pytorch.org/whl/cu113
# or torch2.0.0+cu118
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118

Optioanlly, install ninja for faster build

pip install ninja

Install required packages

pip install -r requirements.txt

Tasks

Task 0. Run NeRFacto

Installation and training steps can be found at NeRFStudio github. After installation, you can run NeRFacto by running this command

# You can try different data 
ns-download-data nerfstudio --capture-name=poster
# Train model
ns-train nerfacto --data data/nerfstudio/poster

NeRFStudio has a variety of NeRF implementations and you can visualize the training process online. NeRFStudio project page More explanation of NeRFacto can be found at here

Once you successfully execute their code, you will be provided with a viewer link where you can navigate around the scene online.

Task 1. Try your own data

Run NeRFacto using your own data Use custom data on NeRFStudio We recommend using COLMAP with images or video. Note that when taking images they should overlap densely over each other.

Task 2. Try other methods: Instant-NGP, Mip-NeRF, TensoRF

Theoretical background and training examples can be found at InstantNGP, Mip-NeRF and TensoRF.

Task 3. Run NeuS

You can check available methods at SDFStudio project page.

We recommend creating a new environment at a separate directory. Follow the instructions to install required packages. SDFStudio github

You can check available dataset at SDFStudio dataset Select any dataset you prefer and train NeuSFacto.

Task 4. Export mesh

Once you have completed training, you can visualize implicit function by mesh extraction. Use the following command.

ns-extract-mesh --load-config outputs/neus-facto-dtu65/neus-facto/XXX/config.yml --output-path meshes/neus-facto-dtu65.ply

Example mesh of Brandenburg gate (takes ~12 hours using 8 GPUs)

Task 5. Run DreamFusion

Follow the instructions to install prerequisites. You can find more details at ThreeStudio github.

Run DreamFusion using either Stable Diffusion model or DeepFloy IF. Freely select text prompt you prefer and train the model.

Here's an example of DreamFusion output.

Task 6. (Optional) Try other methods: Magic3D, ProlificDreamer

(⚠ High VRAM usage) Train other methods available on ThreeStudio.

How do the results differ to vanilla DreamFusion ?

Example outputs of Magic3D and ProlificDreamer.

Task 7. (Optional) Extend framework

Pick any paper from here and extend NeRFStudio/SDFStudio/ThreeStudio.

Further Readings

If you are interested in this topic, we encourage you to check out the papers listed below.

• NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction (NeurIPS 2021)
• Volume Rendering of Neural Implicit Surfaces (NeurIPS 2021)
• DreamFusion: Text-to-3D using 2D Diffusion (ICLR 2023)
• Magic3D: High-Resolution Text-to-3D Content Creation (CVPR 2023)
• Nerfstudio: A Modular Framework for Neural Radiance Field Development (SIGGRAPH 2023)
• Zero-1-to-3: Zero-shot One Image to 3D Object (ICCV 2023)
• ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation (arXiv 2023)