Why do I get out of memory error when I run test_ips.sh?

[XiaoyiHan6]

Hello Dr.Yang
[>>>>>>>>>>>>>>>>>>>>>>> ] 22583/22609, 3.8 task/s, elapsed: 6005s, ETA: 7s
[>>>>>>>>>>>>>>>>>>>>>>> ] 22584/22609, 3.8 task/s, elapsed: 6005s, ETA: 7s
[>>>>>>>>>>>>>>>>>>>>>>> ] 22585/22609, 3.8 task/s, elapsed: 6006s, ETA: 6s
[>>>>>>>>>>>>>>>>>>>>>>> ] 22586/22609, 3.8 task/s, elapsed: 6006s, ETA: 6s
[>>>>>>>>>>>>>>>>>>>>>>> ] 22587/22609, 3.8 task/s, elapsed: 6007s, ETA: 6s
[>>>>>>>>>>>>>>>>>>>>>>> ] 22588/22609, 3.8 task/s, elapsed: 6007s, ETA: 6s
[>>>>>>>>>>>>>>>>>>>>>>> ] 22589/22609, 3.8 task/s, elapsed: 6008s, ETA: 5s
[>>>>>>>>>>>>>>>>>>>>>>> ] 22590/22609, 3.8 task/s, elapsed: 6009s, ETA: 5s
[>>>>>>>>>>>>>>>>>>>>>>> ] 22591/22609, 3.8 task/s, elapsed: 6009s, ETA: 5s
[>>>>>>>>>>>>>>>>>>>>>>> ] 22592/22609, 3.8 task/s, elapsed: 6010s, ETA: 5s
[>>>>>>>>>>>>>>>>>>>>>>> ] 22593/22609, 3.8 task/s, elapsed: 6010s, ETA: 4s
[>>>>>>>>>>>>>>>>>>>>>>> ] 22594/22609, 3.8 task/s, elapsed: 6011s, ETA: 4s
[>>>>>>>>>>>>>>>>>>>>>>> ] 22595/22609, 3.8 task/s, elapsed: 6011s, ETA: 4s
[>>>>>>>>>>>>>>>>>>>>>>> ] 22596/22609, 3.8 task/s, elapsed: 6012s, ETA: 3s
[>>>>>>>>>>>>>>>>>>>>>>> ] 22597/22609, 3.8 task/s, elapsed: 6013s, ETA: 3s
[>>>>>>>>>>>>>>>>>>>>>>> ] 22598/22609, 3.8 task/s, elapsed: 6013s, ETA: 3s
[>>>>>>>>>>>>>>>>>>>>>>> ] 22599/22609, 3.8 task/s, elapsed: 6014s, ETA: 3s
[>>>>>>>>>>>>>>>>>>>>>>> ] 22600/22609, 3.8 task/s, elapsed: 6014s, ETA: 2s
[>>>>>>>>>>>>>>>>>>>>>>> ] 22601/22609, 3.8 task/s, elapsed: 6015s, ETA: 2s
[>>>>>>>>>>>>>>>>>>>>>>> ] 22602/22609, 3.8 task/s, elapsed: 6015s, ETA: 2s
[>>>>>>>>>>>>>>>>>>>>>>> ] 22603/22609, 3.8 task/s, elapsed: 6016s, ETA: 2s
[>>>>>>>>>>>>>>>>>>>>>>> ] 22604/22609, 3.8 task/s, elapsed: 6017s, ETA: 1s
[>>>>>>>>>>>>>>>>>>>>>>> ] 22605/22609, 3.8 task/s, elapsed: 6017s, ETA: 1s
[>>>>>>>>>>>>>>>>>>>>>>> ] 22606/22609, 3.8 task/s, elapsed: 6018s, ETA: 1s
[>>>>>>>>>>>>>>>>>>>>>>> ] 22607/22609, 3.8 task/s, elapsed: 6018s, ETA: 1s
[>>>>>>>>>>>>>>>>>>>>>>> ] 22608/22609, 3.8 task/s, elapsed: 6019s, ETA: 0s
[>>>>>>>>>>>>>>>>>>>>>>>>] 22609/22609, 3.8 task/s, elapsed: 6019s, ETA: 0s/data02/archive01/scv9240/pvsg/vidor/frames/1006_4580824633/0002.png is annotated as all background!
/data02/run01/scv9240/code/OpenPVSG-main/datasets/datasets/utils.py:81: RuntimeWarning: divide by zero encountered in scalar divide
iou = int_area / union
/data02/archive01/scv9240/pvsg/vidor/frames/1006_4580824633/0000.png is annotated as all background!
/data02/archive01/scv9240/pvsg/vidor/frames/1006_4580824633/0003.png is annotated as all background!
/data02/archive01/scv9240/pvsg/vidor/frames/1006_4580824633/0001.png is annotated as all background!
/var/spool/slurmd/job1129607/slurm_script: line 27: 38811 Killed python -u tools/test.py ${CONFIG} ${CHECKPOINT} --work-dir=${WORK_DIR} --eval PQ
slurmstepd: error: Detected 1 oom-kill event(s) in StepId=1129607.batch cgroup. Some of your processes may have been killed by the cgroup out-of-memory handler.

When I execute "sbatch --gpus=4 ./scripts/test/test_ips.sh" statement, I get this error. ( “sbatch” ，This is because I used the server of Beijing Super-Computing Center.) I tried to --gpus=1/2/3/4/8，but I got the same error. However, I found that the utilization of my gpus was very low.

1. Can you tell me Why this happened? (I guess that if your code didn't clear some features and suddenly crashed halfway through the program?)
2. Can you tell me How to solve this problem?

Best regards!

Yours sincerely,

XY.Han