Project Page | Paper | Data
Ethan Tseng, Grace Kuo, Seung-Hwan Baek, Nathan Matsuda, Andrew Maimone, Florian Schiffers, Praneeth Chakravarthula, Qiang Fu, Wolfgang Heidrich, Douglas Lanman, Felix Heide
This code implements a differentiable image formation and optimization model for neural étendue expansion. The experimental and simulation results from the manuscript and the supplementary information are reproducible with this implementation. The proposed framework is implemented completely in PyTorch without dependency on third-party libraries.
Download the data from the data repository before running the code. The data is organized using the same directory structure as this repository. The size of the data repository is around 1 GB.
The experimental results shown in Figure 2 can be reproduced by running the notebooks in the 'Experimental' folder. The notebook reads the raw capture data and performs intensity scaling (equivalent to laser power scaling) on the data. The code reproduces the experimental results for both monochromatic and trichomatic holograms. Each color channel of the trichromatic holograms was captured separately. The code will combine the independent color channel data into a single RGB image. The comparisons against random expanders [Kuo et al. 2020] and conventional holograms [Shi et al. 2021] can also be reproduced from the notebooks. The reconstructed holograms will be displayed within the notebook. Please see the instructions in the notebooks for more details.
The simulation results shown in Figure 3 can be reproduced by running the notebooks in the 'Simulation' folder. The notebooks run the gradient-based optimization described in the manuscript that generates étendue expanded holograms. These notebooks implement the inference step of the algorithm, that is, the neural étendue expander has already been trained and only the SLM pattern is optimized for each test image. The comparisons against random expanders [Kuo et al. 2020] can also be reproduced from the notebooks. The simulated holograms will be displayed within the notebook. Please see the instructions in the notebooks for more details.
The expected runtime of the optimization increases for higher étendue expansion ratios. For 4x étendue expansion the entire optimization should be completed in less than a minute for a single target image on an Nvidia A100 GPU. For 64x étendue expansion the entire optimization should be completed in a few minutes for a single target image on an Nvidia A100 GPU.
In addition, the virtual frequency analysis shown in Figure 3 can be reproduced by running the notebook 'Virtual_Freq.ipynb'. This notebook displays the virtual frequency of a dataset of natural images and the virtual frequency of the neural étendue expanders.
This code has been tested with Python 3.10.10 using PyTorch 2.0.0 running on Linux with an Nvidia A100 GPU with 10 GB RAM. The tested CUDA version is 12.0. An environment.yml file is included that provides the conda environment that we used to run this code. The total installation time when installing with conda should be a few minutes.
We installed the following library packages to run this code:
PyTorch >= 2.0.0
Numpy
Scipy
matplotlib
jupyter-notebook
If you find our work useful in your research, please cite:
@article{Tseng2024NeuralEtendueExpander,
author={Ethan Tseng, Grace Kuo, Seung-Hwan Baek, Nathan Matsuda, Andrew Maimone, Florian Schiffers, Praneeth Chakravarthula, Qiang Fu, Wolfgang Heidrich, Douglas Lanman, Felix Heide},
title={Neural \'{E}tendue Expander for Ultra-Wide-Angle High-Fidelity Holographic Display},
journal={Nature Communications},
year={2024},
month={Apr},
day={22},
volume={15},
number={1},
pages={2907}
}
For more information on holography and étendue expansion check out Holotorch and our SIGGRAPH 2022 course.
Our code is licensed under BSL-1. By downloading the software, you agree to the terms of this License.