Error running train/eval scripts

[jiadingfang]

Hi authors,

I'm trying to run your train/eval code, but I got the following error:

(LAR) [fjd@gpu10 LAR-Look-Around-and-Refer]$ python train_referit3d.py --transformer --model mmt_referIt3DNet -scannet-file $scanfile -referit3D-file $nr3dfile_csv --log-dir log/$exp_id --n-workers 2 --gpu 0 --unit-sphere-norm True --feat2d clsvecROI --mode evaluate --pretrained-path $pretrain_path/best_model.pth -load-dense False -load-imgs False --img-encoder False
1.13.1+cu117
Number of available GPUs is: 1
True
{'augment_with_sr3d': None,
'batch_size': 32,
'camaug': False,
'checkpoint_dir': 'log/test/06-20-2023-13-43-55/checkpoints',
'clspred3d': False,
'cluster_pid': None,
'cocoon': False,
'config_file': None,
'context_2d': None,
'context_info_2d_cached_file': None,
'context_obj': None,
'contrastiveloss': False,
'dgcnn_intermediate_feat_dim': [128, 128, 128, 128],
'dist_backend': 'nccl',
'dist_url': 'tcp://224.66.41.62:23456',
'eval_path': None,
'experiment_tag': None,
'feat2d': 'clsvecROI',
'feat2ddim': 2048,
'fine_tune': False,
'freeze_backbone': False,
'geo3d': False,
'gpu': 0,
'graph_out_dim': 128,
'img_encoder': False,
'imgaug': False,
'imgsize': 32,
'init_lr': 0.0005,
'knn': 7,
'lang_cls_alpha': 0.5,
'language_fusion': 'both',
'language_latent_dim': 768,
'load_dense': False,
'load_imgs': False,
'log_dir': 'log/test/06-20-2023-13-43-55',
'max_distractors': 51,
'max_seq_len': 24,
'max_test_objects': 88,
'max_train_epochs': 100,
'mentions_target_class_only': True,
'min_word_freq': 3,
'mmt_latent_dim': 768,
'mmt_mask': None,
'mode': 'evaluate',
'model': 'mmt_referIt3DNet',
'multiprocessing_distributed': False,
'n_workers': 2,
'obj_cls_alpha': 0.5,
'object_encoder': 'pnet_pp',
'object_latent_dim': 768,
'patience': 10,
'points_per_object': 1024,
'pretrained_path': '/share/data/2pals/fjd/SAT_release/SAT_clsvecROI_Nr3D/best_model.pth',
'random_seed': 2020,
'rank': -1,
'referit3D_file': '/share/data/2pals/fjd/referit3d/refer_it_3d/nr3d_train.csv',
'resume_path': None,
's_vs_n_weight': None,
'save_args': True,
'scannet_file': '/share/data/2pals/fjd/referit3d/keep_all_points_00_view_with_global_scan_alignment/keep_all_points_00_view_with_global_scan_alignment.pkl',
'sceneCocoonPath': None,
'sharetwoTrans': False,
'softtripleloss': False,
'tensorboard_dir': 'log/test/06-20-2023-13-43-55/tb_logs',
'train_scanRefer': False,
'train_vis_enc_only': False,
'transformer': True,
'tripleloss': False,
'twoStreams': False,
'twoTrans': False,
'unit_sphere_norm': True,
'vocab_file': None,
'warmup': True,
'word_dropout': 0.1,
'word_embedding_dim': 64,
'world_size': -1}
Use GPU: 0 for training
starting caching the pkl files.....
Loaded in RAM 707 scans
524 instance classes exist in these scans
#train/test scans: 1201 / 312
Finish caching the pkl files, Done
Dropping utterances without explicit mention to the target class 28716->28716
95-th percentile of token length for remaining (training) data is: 20.0
Dropping utterances with more than 24 tokens, 28716->28716
(mean) Random guessing among target-class test objects nan
Dropped 196 scans to reduce mem-foot-print.
Length of vocabulary, with min_word_freq=3 is 1288
511 training scans will be used.
90-th percentile of number of objects in the (training) scans is: 52.00
Traceback (most recent call last):
File "train_referit3d.py", line 445, in
main_worker(args.gpu, ngpus_per_node, args)
File "train_referit3d.py", line 95, in main_worker
mean_rgb, vocab = compute_auxiliary_data(referit_data, all_scans_in_dict, args)
File "/share/data/ripl/fjd/LAR-Look-Around-and-Refer/referit3d/in_out/neural_net_oriented.py", line 199, in compute_auxiliary_data
obj_cnt = objects_counter_percentile(testing_scan_ids, all_scans, prc)
File "/share/data/ripl/fjd/LAR-Look-Around-and-Refer/referit3d/in_out/neural_net_oriented.py", line 45, in objects_counter_percentile
return np.percentile(all_obs_len, prc)
File "<array_function internals>", line 6, in percentile
File "/share/data/ripl/fjd/miniconda3/envs/LAR/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3868, in percentile
a, q, axis, out, overwrite_input, interpolation, keepdims)
File "/share/data/ripl/fjd/miniconda3/envs/LAR/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3988, in _quantile_unchecked
interpolation=interpolation)
File "/share/data/ripl/fjd/miniconda3/envs/LAR/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3564, in _ureduce
r = func(a, **kwargs)
File "/share/data/ripl/fjd/miniconda3/envs/LAR/lib/python3.7/site-packages/numpy/lib/function_base.py", line 4098, in _quantile_ureduce_func
n = np.isnan(ap[-1])
IndexError: index -1 is out of bounds for axis 0 with size 0

I appreciate if you could provide any insights on why this error could happen and how to possibly solve it. Thanks!