job_submission for MiNoPy

[falkamelung]

• We used to have the job_submission command in the log file. I used to have:

run_workflow.bash /scratch/05861/tg851601/HaitiSenDT142 --append --dostep mintpy --tmp
minopyApp.py /scratch/05861/tg851601/HaitiSenDT142/modified_template/HaitiSenDT142.template --dir minopy --jobfiles
20211213:18-12 * run_workflow.bash /scratch/05861/tg851601/HaitiSenDT142 --append --dostep minopy --tmp
20211214:09-12 * run_workflow.bash /scratch/05861/tg851601/HaitiSenDT142 --minopy --start 3
20211214:105050 * job_submission.py --template /scratch/05861/tg851601/HaitiSenDT142/modified_template/HaitiSenDT142.template /scratch/05861/tg851601/HaitiSenDT142/minopy/run_files/run_03_minopy_generate_ifgram --outdir /scratch/05861/tg851601/HaitiSenDT142/minopy/run_files --numBursts 3 --writeonly
20211214:110004 * job_submission.py --template /scratch/05861/tg851601/HaitiSenDT142/modified_template/HaitiSenDT142.template /scratch/05861/tg851601/HaitiSenDT142/minopy/run_files/run_03_minopy_generate_ifgram --outdir /scratch/05861/tg851601/HaitiSenDT142/minopy/run_files --numBursts 3 --writeonly
20211214:110023 * job_submission.py --template /scratch/05861/tg851601/HaitiSenDT142/modified_template/HaitiSenDT142.template /scratch/05861/tg851601/HaitiSenDT142/minopy/run_files/run_03_minopy_generate_ifgram --outdir /scratch/05861/tg851601/HaitiSenDT142/minopy/run_files --numBursts 3 --writeonly
20211214:110038 * job_submission.py --template /scratch/05861/tg851601/HaitiSenDT142/modified_template/HaitiSenDT142.template /scratch/05861/tg851601/HaitiSenDT142/minopy/run_files/run_03_minopy_generate_ifgram --outdir /scratch/05861/tg851601/HaitiSenDT142/minopy/run_files --numBursts 3 --writeonly
20211214:110054 * job_submission.py --template /scratch/05861/tg851601/HaitiSenDT142/modified_template/HaitiSenDT142.template /scratch/05861/tg851601/HaitiSenDT142/minopy/run_files/run_03_minopy_generate_ifgram --outdir /scratch/05861/tg851601/HaitiSenDT142/minopy/run_files --numBursts 3 --writeonly
20211214:110107 * job_submission.py --template /scratch/05861/tg851601/HaitiSenDT142/modified_template/HaitiSenDT142.template /scratch/05861/tg851601/HaitiSenDT142/minopy/run_files/run_03_minopy_generate_ifgram --outdir /scratch/05861/tg851601/HaitiSenDT142/minopy/run_files --numBursts 3 --writeonly
20211214:110118 * job_submission.py --template /scratch/05861/tg851601/HaitiSenDT142/modified_template/HaitiSenDT142.template /scratch/05861/tg851601/HaitiSenDT142/minopy/run_files/run_03_minopy_generate_ifgram --outdir /scratch/05861/tg851601/HaitiSenDT142/minopy/run_files --numBursts 3 --writeonly

• run_02_minopy times out if I process 7 years of data. Are we taking the number of acquisitions into account for wall time calculation?
• run_02_minopy times out and we get emails for re-running. Shall we resubmit without lengthening the walltime? The danger is that if time for just one patch is too long it never will finish, right? We also could suppress sending emails for this step.
  -make recommendation on how to check from progress of run_02 step (when it is running you are concerned that it just runs and will never finish....)
• minopy.interferograms.referenceDate : option for snowfree (last image in August/February for northern/southern Hemisphere). Or something with max coherence
• For wall time calculation take number of dates into account for appropriate steps
• I used to get lots of timeouts for areas including water. We need an option to mask out water (e.g. preprocessing option to generate water mask). In addition, can't we have an option in the template file to lengthen wall times for known problem cases? e.g. minsar.walltime_factor or minsar.walltime_factor.minopy_unwrap_ifgram=1.2. `create_runfiles.py would run at the end a fumction to adjust the walltimes.
• A user always wants to analyze an area as big as possible. We should give some advice on how big is reasonable given the available computer resources.

### Low Priority:
- for `--tmp` and no option given write jobfiles into `run_files_tmp` (that makes it consistent with minsar)
- rename --numBursts to something else

[falkamelung]

MinoPy walltime issues

• run_01 is too short for multiple bursts (Trouble generating slcStack for SLC from too many bursts because of timeout MiNoPy#68 )
• run_09* is too long: run_09_mintpy_timeseries_correction_0.job :#SBATCH -t 3:36:00 for unittestGalapagos