Add an updated offline domain check for satellite radiances

[SamuelDegelia-NOAA]

This PR adds a new version of the offline domain check for satellite radiances. The modifications were originally done by @delippi and @xyzemc to account for satellite radiances being 2D variables in the IODA file (channel, location) instead of 1D.

Modifications are also made here to speed up the section of the code that slices the original 2D array to retrieve the locations contained within the domain. The original version took over an hour to process large dump files. Now we use the itemgetter function from the operator library (e.g., g.variables[var][:,idy] = itemgetter(*inside_indices)(invar[:,idy])) which is much faster. The script can now process global satellite datasets in 15-20 seconds.

Example usage:

python offline_domain_check_satrad.py -g invariant.nc -o gdas.t00z.atms_all_npp.tm00.nc -s 0.1 -f

[delippi]

@SamuelDegelia-NOAA & @xyzemc very nice result and thanks for taking this on! I figured it could be done and wasn't too difficult. I'm glad to see this as a separate utility from the other offline domain check that we've been using for conventional data. It just seems easier to have them separated.

Do you have a resulting scatter plot map and has the resulting ioda*_dc.nc file been tested and work as expected?

[SamuelDegelia-NOAA]

@delippi Yes thank you for the reminder! Here is the scatter plot. It matches what @xyzemc showed from the original code that took ~1 hour. So it looks like things are working as expected.

[xyzemc]

Very nice work.

[delippi]

It is really good to have this utility to facilitate testing when the online domain check doesn't want to cooperate (actually it isn't the domain check cooperating but particular QC filters which might not be configured to look at the qcflags correct). I just want to make sure that we are only using these tools when the offline domain check doesn't work first. It seems that at least some radiance data isn't having trouble using the qcflags set by the online domain check. So if the online domain check works for a particular data set we should use that and not the offline domain check which is meant only to be a temporary solution so that we can do our tests. That being said, I found the problem in ufo as to why the online domain check hasn't been working for conventional obs and will have a fix for that soon. With that fix, I don't think there should be any issues using the online domain check for any data set. But if there are any remaining issues with any data sets, I think it should be fairly easy to pinpoint the problem in ufo and make that fix. Again, I think it is still useful to have this utility available.

[SamuelDegelia-NOAA]

@delippi I agree that the offline domain check should be a temporary route. If you are able to fix the issue with the conventional obs and the online domain check then that would be great!

One thing we will need to think about is whether we still need the offline domain check for GETKF. The solver becomes very slow based on the number of obs read in. And since the online domain check cannot work with reduce obs space yet, keeping all of those obs in memory could make the GETKF too slow. But maybe this could be solved by also adding a Bounds Check that throws out any observations outside a lat/lon range. I believe Bounds Check is a pre-filter, so it should be able to work with reduce obs space. That could go a long way to reducing the slow down from keeping lots of obs outside the domain (e.g., satellite obs) in memory. But I will plan to do some testing once you fix the bug in UFO!

[delippi]

@SamuelDegelia-NOAA, I've already tested the fix for conventional obs and it does work. It is actually a very simple fix! I just need to work on the PR with a few other updates to that same QC filter. I agree that making clever use of pre-filters and reduce obs space could help remove the need to use these tools. Sounds like a good idea and I think it will work.

[SamuelDegelia-NOAA]

PS: I have a working version of the offline domain check that uses a concave hull to reject fewer observations inside the domain. This update doesn't need any special python modules and is only about ~15s slower than the convex hull approach. But considering that we are close to having the online domain check working, it might not be worth committing this new check. We can talk about this on Thursday.