Overwhelmed storage with too many read requests when running on > 14,000 MPI processes

[fyli16]

Hello, I am trying some large scale runs with 14336 (256 nodes x 56 cores/node) MPI processes on the Frontera machine of TACC (https://tacc.utexas.edu/systems/frontera/). My jobs were cancelled by admin in the middle of the run, who complained that my job was overwhelming the storage targets with too many read requests and asked us to reduce IO. Is this problem normal for running SMILEI at this scale? I tried to halve the size of the problem and run with 128 nodes (7168 MPI) and had no issues (no complaints and no cancellation of my jobs).

Here is my job script and input file:
input.txt
job.txt
It's 2D Cartesian simulations and the diagnostics include simple probe (at selected points), fields on grids, and particlebinning.

Any solutions/comments are appreciated.

[charlesprouveur]

I cannot comment on the policy of this admin, but i have never seen the number of reads being an issue. SMILEI is regularly run with more than 14000 MPI processes on other systems and the issue in the case of IOs is the allocated space in the scratch partition. Overall it is a good policy to reduce as much as you can the number of IOs to improve the performance. If you want to reduce the number of MPI processes, a possibility would be to use OMP_NUM_THREADS (for instance use 256 x 14 tasks with 256 x 56 cores and OMP_NUM_THREADS=4).

Note: this issue has been converted into a discussion as this is not a bug issue (smilei was running fine, the admin killed it)

[fyli16]

Thanks for your comments! I will try out OMP and see if it helps. Could you elaborate on the issue of IOs? It is related to that too many MPI tried to access disk writing (into a single file) concurrently?

[charlesprouveur]

10 years ago, 10000 tasks accessing a file could bring down a file system. Frontera does not seem that old to me so I am a bit surprised. Nowadays I would say it is more of a performance issue, Bigger IOs should only decrease the performance of your run in order to write in parallel the same hdf5 file. Again, in my personal experience with production runs, 14000 is not outlandish

[fyli16]

It worked by using OMP_NUM_THREADS>1. Thanks again for your help! @charlesprouveur