enable cycling ensemble runs based on GETKF

[guoqing-noaa]

One can run the cycling ensemble with cycling deterministic at the same time, and then the ensemble part will get the ioda observation files from the deterministic run directly. Under this situation, set DO_IODA=false for ensemble runs.

One can run the cycling ensemble independently. Under this situation, set DO_IODA=true.

[SamuelDegelia-NOAA]

@guoqing-noaa Thank you for immense work in getting this functionality added. I just tested the workflow on the 27 May retro case. The GETKF tasks complete successfully but I am seeing crashes for the fcst step on the 0300 UTC cycle.

----------------------------------------------------------------------
Beginning MPAS-atmosphere Error Log File for task       1 of      40
    Opened at 2025/03/03 22:26:38
----------------------------------------------------------------------

ERROR: Error in compute_layer_mean: pressure should increase with index

Have you or anyone else running the workflow experienced this error before, or is this new with the ensemble DA steps added?

[guoqing-noaa]

@SamuelDegelia-NOAA Thanks for testing. We got the same model crash. I have not had time to debug the issue. But the problem should come for DA (i.e., GETKF updating ensembles).

If you set save posterior ensemble: false in parm/getkf_solver.yaml, the workflow will run through and the ensemble forecast does not crash.

There is something wrong with the DA settings, and we need to figure it out.
But the workflow itself is correct. Do you agree?

[SamuelDegelia-NOAA]

@guoqing-noaa Yes I would agree that something is wrong with the DA settings if setting save posterior ensemble: false allows the workflow to keep running. I will look into this. Other than that, the GETKF tasks in the workflow look good.

[guoqing-noaa]

@guoqing-noaa Yes I would agree that something is wrong with the DA settings if setting save posterior ensemble: false allows the workflow to keep running. I will look into this. Other than that, the GETKF tasks in the workflow look good.

@SamuelDegelia-NOAA It is great that you can look at the GETKF DA configurations!
If it is fine with you, you may approve this PR so it may be merged for more testing and ready for upcoming GETKF DA change PRs. Thanks!

[SamuelDegelia-NOAA]

Yes that is fine by me, thanks again!

[guoqing-noaa]

For information, I tested this branch for regular (no GETKF) runs on Hera/Jet/Gaea and all worked well.