problem while training

[ankan2709]

oject/gwang/ankan/zipnerf/data/tools/images
*** Loaded 58 images from disk
*** Loaded EXIF data for 58 images
*** Constructed COLMAP-to-world transform.
*** Constructed 120 render poses via ellipse path
*** Constructed train/test split: #train=15 #test=43
*** LLFF successfully loaded!
*** split=DataSplit.TRAIN
*** #images/poses/exposures=15
*** #camtoworlds=15
*** resolution=(2048, 3840)
I0202 13:09:14.556802 23456247948864 train.py:151] Optimization parameter sizes/counts:
I0202 13:09:14.583516 23456247948864 train.py:153] grid_0 6590464
I0202 13:09:14.583668 23456247948864 train.py:153] grid_0/grid_016 (16, 16, 16, 1)
I0202 13:09:14.583768 23456247948864 train.py:153] grid_0/grid_032 (32, 32, 32, 1)
I0202 13:09:14.583875 23456247948864 train.py:153] grid_0/grid_064 (64, 64, 64, 1)
I0202 13:09:14.584216 23456247948864 train.py:153] grid_0/grid_128 (128, 128, 128, 1)
I0202 13:09:14.584261 23456247948864 train.py:153] grid_0/hash_256 (2097152, 1)
I0202 13:09:14.584302 23456247948864 train.py:153] grid_0/hash_512 (2097152, 1)
I0202 13:09:14.585829 23456247948864 train.py:153] MLP_0 897
I0202 13:09:14.585915 23456247948864 train.py:153] MLP_0/Dense_0 832
I0202 13:09:14.585976 23456247948864 train.py:153] MLP_0/Dense_0/kernel (12, 64)
I0202 13:09:14.586011 23456247948864 train.py:153] MLP_0/Dense_0/bias (64,)
I0202 13:09:14.586069 23456247948864 train.py:153] MLP_0/Dense_1 65
I0202 13:09:14.586101 23456247948864 train.py:153] MLP_0/Dense_1/kernel (64, 1)
I0202 13:09:14.586128 23456247948864 train.py:153] MLP_0/Dense_1/bias (1,)
I0202 13:09:14.586176 23456247948864 train.py:153] grid_1 10784768
I0202 13:09:14.586209 23456247948864 train.py:153] grid_1/grid_0016 (16, 16, 16, 1)
I0202 13:09:14.586237 23456247948864 train.py:153] grid_1/grid_0032 (32, 32, 32, 1)
I0202 13:09:14.586264 23456247948864 train.py:153] grid_1/grid_0064 (64, 64, 64, 1)
I0202 13:09:14.586291 23456247948864 train.py:153] grid_1/grid_0128 (128, 128, 128, 1)
I0202 13:09:14.586317 23456247948864 train.py:153] grid_1/hash_0256 (2097152, 1)
I0202 13:09:14.586342 23456247948864 train.py:153] grid_1/hash_0512 (2097152, 1)
I0202 13:09:14.586367 23456247948864 train.py:153] grid_1/hash_1024 (2097152, 1)
I0202 13:09:14.586392 23456247948864 train.py:153] grid_1/hash_2048 (2097152, 1)
I0202 13:09:14.587173 23456247948864 train.py:153] MLP_1 1153
I0202 13:09:14.587485 23456247948864 train.py:153] MLP_1/Dense_0 1088
I0202 13:09:14.587525 23456247948864 train.py:153] MLP_1/Dense_0/kernel (16, 64)
I0202 13:09:14.587555 23456247948864 train.py:153] MLP_1/Dense_0/bias (64,)
I0202 13:09:14.587608 23456247948864 train.py:153] MLP_1/Dense_1 65
I0202 13:09:14.587638 23456247948864 train.py:153] MLP_1/Dense_1/kernel (64, 1)
I0202 13:09:14.587666 23456247948864 train.py:153] MLP_1/Dense_1/bias (1,)
I0202 13:09:14.587712 23456247948864 train.py:153] grid_2 59916288
I0202 13:09:14.587743 23456247948864 train.py:153] grid_2/grid_0016 (16, 16, 16, 4)
I0202 13:09:14.587779 23456247948864 train.py:153] grid_2/grid_0032 (32, 32, 32, 4)
I0202 13:09:14.587811 23456247948864 train.py:153] grid_2/grid_0064 (64, 64, 64, 4)
I0202 13:09:14.587838 23456247948864 train.py:153] grid_2/grid_0128 (128, 128, 128, 4)
I0202 13:09:14.587872 23456247948864 train.py:153] grid_2/hash_0256 (2097152, 4)
I0202 13:09:14.587897 23456247948864 train.py:153] grid_2/hash_0512 (2097152, 4)
I0202 13:09:14.587923 23456247948864 train.py:153] grid_2/hash_1024 (2097152, 4)
I0202 13:09:14.587947 23456247948864 train.py:153] grid_2/hash_2048 (2097152, 4)
I0202 13:09:14.587972 23456247948864 train.py:153] grid_2/hash_4096 (2097152, 4)
I0202 13:09:14.587997 23456247948864 train.py:153] grid_2/hash_8192 (2097152, 4)
I0202 13:09:14.588317 23456247948864 train.py:153] MLP_2 225877
I0202 13:09:14.588375 23456247948864 train.py:153] MLP_2/Dense_0 3264
I0202 13:09:14.588407 23456247948864 train.py:153] MLP_2/Dense_0/kernel (50, 64)
I0202 13:09:14.588435 23456247948864 train.py:153] MLP_2/Dense_0/bias (64,)
I0202 13:09:14.588488 23456247948864 train.py:153] MLP_2/Dense_1 65
I0202 13:09:14.588523 23456247948864 train.py:153] MLP_2/Dense_1/kernel (64, 1)
I0202 13:09:14.588550 23456247948864 train.py:153] MLP_2/Dense_1/bias (1,)
I0202 13:09:14.588595 23456247948864 train.py:153] MLP_2/Dense_2 16640
I0202 13:09:14.588625 23456247948864 train.py:153] MLP_2/Dense_2/kernel (64, 256)
I0202 13:09:14.588653 23456247948864 train.py:153] MLP_2/Dense_2/bias (256,)
I0202 13:09:14.588695 23456247948864 train.py:153] MLP_2/Dense_3 72704
I0202 13:09:14.588736 23456247948864 train.py:153] MLP_2/Dense_3/kernel (283, 256)
I0202 13:09:14.588765 23456247948864 train.py:153] MLP_2/Dense_3/bias (256,)
I0202 13:09:14.589181 23456247948864 train.py:153] MLP_2/Dense_4 65792
I0202 13:09:14.589230 23456247948864 train.py:153] MLP_2/Dense_4/kernel (256, 256)
I0202 13:09:14.589262 23456247948864 train.py:153] MLP_2/Dense_4/bias (256,)
I0202 13:09:14.589311 23456247948864 train.py:153] MLP_2/Dense_5 65792
I0202 13:09:14.589342 23456247948864 train.py:153] MLP_2/Dense_5/kernel (256, 256)
I0202 13:09:14.589369 23456247948864 train.py:153] MLP_2/Dense_5/bias (256,)
I0202 13:09:14.589750 23456247948864 train.py:153] MLP_2/Dense_6 1620
I0202 13:09:14.589790 23456247948864 train.py:153] MLP_2/Dense_6/kernel (539, 3)
I0202 13:09:14.589818 23456247948864 train.py:153] MLP_2/Dense_6/bias (3,)
I0202 13:09:15.044798 23456247948864 checkpoints.py:1101] Found no checkpoint files in /project/gwang/ankan/zipnerf/ckpt/tools with prefix checkpoint_
/home/ad892/anaconda3/envs/camp_zipnerf/lib/python3.11/site-packages/jax/_src/xla_bridge.py:945: UserWarning: jax.host_id has been renamed to jax.process_index. This alias will eventually be removed; please update your code.
warnings.warn(
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/mmfs1/project/gwang/ankan/zipnerf/train.py", line 557, in
app.run(main)
File "/home/ad892/anaconda3/envs/camp_zipnerf/lib/python3.11/site-packages/absl/app.py", line 308, in run
run_main(main, args)
File "/home/ad892/anaconda3/envs/camp_zipnerf/lib/python3.11/site-packages/absl/app.py", line 254, in run_main
sys.exit(main(argv))
^^^^^^^^^^
File "/mmfs1/project/gwang/ankan/zipnerf/train.py", line 210, in main
state, stats, rngs = train_pstep(rngs, state, batch, cameras, train_frac) # pytype: disable=wrong-arg-types # jnp-type
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mmfs1/project/gwang/ankan/zipnerf/internal/train_utils.py", line 476, in train_step
(, (stats, mutable_camera_params)), grad = loss_grad_fn(state.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mmfs1/project/gwang/ankan/zipnerf/internal/train_utils.py", line 357, in loss_fn
renderings, ray_history = model.apply(
^^^^^^^^^^^^
File "/mmfs1/project/gwang/ankan/zipnerf/internal/models.py", line 279, in call
ray_results = mlp(
^^^^
File "/mmfs1/project/gwang/ankan/zipnerf/internal/models.py", line 779, in call
raw_density, x = predict_density(means, covs, **predict_density_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mmfs1/project/gwang/ankan/zipnerf/internal/models.py", line 682, in predict_density
grid(
File "/mmfs1/project/gwang/ankan/zipnerf/internal/grid_utils.py", line 228, in call
values = self.param(f'{datastructure}{grid_size_str}', init_fn)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
flax.errors.ScopeParamNotFoundError: Could not find parameter named "grid_0000" in scope "/grid_1". (https://flax.readthedocs.io/en/latest/api_reference/flax.errors.html#flax.errors.ScopeParamNotFoundError)

For simplicity, JAX has removed its internal frames from the traceback of the following exception. Set JAX_TRACEBACK_FILTERING=off to include these.