filter null ocean colour data #122
Conversation
twsearle
force-pushed
the
feature/filter-null-oceancolour
branch
from
73f737e
to
64e7d4f
Compare
| @@ -0,0 +1,15 @@ | |||
| where: | |||
| varno: [288,] | |||
| query: initial_obsvalue is not null | |||
There was a problem hiding this comment.
Is the indentation correct for this line? In https://github.com/JCSDA-internal/ioda/pull/1345/files the query is indented the same as varno.
I would be using this in the AMM7 sith change, so that we can compare the outputs across OPS and JOPA.
I think I can't put it in ioda as it doesn't have a test. I hesitate to put it in as the change was only just removed from ioda. Another option would be to put it in jjdocs/jjaux. I honestly don't know what the preferred protocol is for this or what others are doing. As I say I think ideally, we wouldn't have these large ODB files with mostly missing data so I would like to resolve it upstream. |
|
Would it work if it were inserted into the ODB->ODB2 conversion a bit like is done in https://github.com/MetOffice/jjdocs/blob/develop/docs/share/jopa/ocean/apps/ocean_ops_codb_odb_to_odb2/opt/rose-app-o-ts.conf#L2? If not, I'm happy to approve this. |
I will give it a go! That at least should be a bit better in performance. Ideally we don't want it in the ODB in the first place, but this does seem like a better place to put it thanks. |
|
Looks like using the ODB->ODB2 step works closing this one. |
Description
75% of ocean colour observations are null values in ODB files. We can improve performance and comparibility with the OPS by filtering these observations from the ODB file before JOPA runs. Ideally, we would do this during creation of the ODB file, however I am not sure how to do this, and this may not be in scope for the ODB creation executables.
I attempted to add this to the base query file, however ODB is unable to handle files with a different number of valid
varno's. My change was removed as it caused various crashes on different platforms:https://github.com/JCSDA-internal/ioda/pull/1345
As an alternative, I am temporarily adding the query file to orca-jedi on the assumption that we only need Chlorophyll data (true for PS46) so that we filter down to non-null values.
In the future, I hope that upstream changes by @adammaycock or @DJDavies2 might render this file unnecessary.
Impact
Enables comparison of OPS and JOPA ocean colour outputs.