Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add 187/287 METAR config templates #218

Draft
wants to merge 1 commit into
base: develop
Choose a base branch
from
Draft

Add 187/287 METAR config templates #218

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
194 changes: 194 additions & 0 deletions rrfs-test/validated_yamls/templates/obtype_config/adpsfc_airTemperature_187.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,194 @@
- obs space:
name: adpsfc_airTemperature_187
distribution:
name: "@DISTRIBUTION@"
halo size: 100e3
obsdatain:
engine:
type: H5File
obsfile: "@OBSFILE@"
obsdataout:
engine:
type: H5File
obsfile: hofx_adpsfc_airTemperature_187.nc4
allow overwrite: true
io pool:
max pool size: 1
observed variables: [airTemperature]
simulated variables: [airTemperature]

obs operator:
name: Composite
components:
- name: VertInterp
variables:
- name: airTemperature
vertical coordinate: air_pressure
observation vertical coordinate: pressure
observation vertical coordinate group: MetaData
interpolation method: log-linear

obs error:
covariance model: diagonal

obs localizations:
- localization method: Horizontal Gaspari-Cohn
lengthscale: 300e3 # orig

obs filters:
# ------------------
# airTemperature (187)
# ------------------
# Reject all obs with ObsType /= 187, QualityMarker = 4 -15
- filter: RejectList
where:
- variable: ObsType/airTemperature
is_in: 187
- variable: QualityMarker/airTemperature
is_in: 4-15

# Time window filter
- filter: Domain Check
where:
- variable:
name: MetaData/timeOffset
minvalue: -900
maxvalue: 900

# Temporal thinning filter
- filter: Temporal Thinning
min_spacing: PT30M
seed_time: *analysisDate
category_variable:
name: MetaData/stationIdentification
Comment on lines +58 to +63
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@spanNOAA, are you finding that this is needed to match ob counts and increments? I found that for mesonet that this was actually detrimental for my results. Are you mainly using this to remove duplicate observations? If so, there could be other things we could try to handle that. I don't think the data is temporally thinned in GSI, so this "might" not be the best. Just something to consider.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found that removing duplicate METAR observations actually improved my results. In my case, eliminating them improved the analysis, as the duplicates may add additional value at the same location even if the information was outdated or from a significantly later time.

I believe the detrimental results when applying temporal thinning to mesonet observations may be due to the smaller observation window used. For mesonet data, a ±6 minute window is applied, while for METAR data, a larger ±15 minute window is used. This is because some METAR stations have less frequent observations. As a result, applying a smaller window may cause some stations to be dropped, while others might have multiple observations within the ±15 minute window.

Furthermore, GSI applies a form of "temporal thinning" when the l_closeobs option is enabled. This method retains only the observation closest to the analysis time and discards the others. Below is the relevant code:

if (l_closeobs) then
    if(abs(data(itime,k)-hr_offset)<abs(data(itime,l)-hr_offset)) then
        muse(l)=.false.
    else
        muse(k)=.false.
    endif
else
    tfact=min(one,abs(data(itime,k)-data(itime,l))/dfact1)
    dup(k)=dup(k)+one-tfact*tfact*(one-dfact)
    dup(l)=dup(l)+one-tfact*tfact*(one-dfact)
endif

Temporal thinning works well for temperature, humidity, and wind observations. However, I've had an issue with temporal thinning for surface pressure in my case. Specifically, surface observations in Canada were being dropped when temporal thinning was enabled. Theoretically, there should always be at least one observation at each station. I'm still investigating the cause of this issue.

30M spacing
image
0M spacing
image

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should try to remove the duplicated observations from surface stations. @delippi Could you provide more details on how this filter detriment your MESONET results?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here are some runs I just made for mesonet using the temporal filter. The time window is +/-6 min. "No filter" gives better observation counts (counts are in the image; lower left corner), but increments look largely the same. The difference (between no filter and PT01M) is hard to spot but mainly just over west Texas. No temporal filter just seems better to me. PT30M definitely seems not good for this case. It could be argued that PT00M or PT01M would be fine.

No Temporal filter: (This gives different result from PT00M)
fv3jedi_vs_gsi_increment_airTemperature_msonet_airTemperature_188_None
PT01M:
fv3jedi_vs_gsi_increment_airTemperature_msonet_airTemperature_188_PT01M
PT30M:
fv3jedi_vs_gsi_increment_airTemperature_msonet_airTemperature_188_PT30M


# Initial error assignment
- filter: Perform Action
filter variables:
- name: airTemperature
where:
- variable:
name: ObsType/airTemperature
is_in: 187
action:
name: assign error
error function:
name: ObsFunction/ObsErrorModelStepwiseLinear
options:
xvar:
name: MetaData/pressure
xvals: [110000, 105000, 100000, 95000, 90000, 85000, 80000, 75000, 70000, 65000, 60000, 55000, 50000, 45000, 40000, 35000, 30000, 25000, 20000, 15000, 10000, 7500, 5000, 4000, 3000, 2000, 1000, 500, 400, 300, 200, 100, 0]
errors: [2.2585, 2.2585, 2.2585, 2.2585, 2.2585, 2.2585, 2.2585, 2.2585, 2.2585, 2.2585, 2.2585, 2.2585, 2.2585, 2.2585, 2.2585, 2.2585, 2.2585, 2.2585, 2.2585, 2.2585, 2.2585, 2.2585, 2.2585, 2.2585, 2.2585, 2.2585, 2.2585, 2.2585, 2.2585, 2.2585, 2.2585, 2.2585, 2.2585]

# error inflation based on pressure check (setupt.f90)
- filter: Perform Action
filter variables:
- name: airTemperature
where:
- variable: ObsType/airTemperature
is_in: 187
action:
name: inflate error
inflation variable:
name: ObsFunction/ObsErrorFactorPressureCheck
options:
variable: airTemperature
inflation factor: 8.0
# The new feature "surface observation error ramp"
# needs to be added to UFO to align with the ramp
# options for temperature and humidity in GSI.
#surface observation error ramp: true

# Create temporary ObsErrorData
- filter: Variable Assignment
assignments:
- name: TempObsErrorData/airTemperature
type: float
function:
name: ObsFunction/Arithmetic
options:
variables:
- name: ObsErrorData/airTemperature

# Set ObsError set "error parameter" if < "max value"
- filter: Perform Action
filter variables:
- name: airTemperature
action:
name: assign error
error parameter: 1.0
where:
- variable:
name: ObsErrorData/airTemperature
maxvalue: 1.0
- variable:
name: ObsErrorData/airTemperature
value: is_valid

# Set ObsError set "error parameter" if > "min value"
- filter: Perform Action
filter variables:
- name: airTemperature
action:
name: assign error
error parameter: 3.0
where:
- variable:
name: ObsErrorData/airTemperature
minvalue: 3.0
- variable:
name: ObsErrorData/airTemperature
value: is_valid

# Gross Error Check
- filter: Background Check
filter variables:
- name: airTemperature
threshold: 5.0
where:
- variable: ObsType/airTemperature
is_in: 187
- variable: QualityMarker/airTemperature
is_not_in: 3
action:
name: reject

# Gross Error Check
- filter: Background Check
filter variables:
- name: airTemperature
threshold: 3.5
where:
- variable: ObsType/airTemperature
is_in: 187
- variable: QualityMarker/airTemperature
is_in: 3
action:
name: reject

# Re-assign err ObsErrorData <--- TempObsErrorData
# after gross error check.
- filter: Perform Action
filter variables:
- name: airTemperature
action:
name: assign error
error function: TempObsErrorData/airTemperature
where:
- variable:
name: TempObsErrorData/airTemperature
value: is_valid

#- filter: Print Filter Data
# message: Printing filter data
# summary: true
# variables:
# - variable: MetaData/latitude
# - variable: MetaData/longitude
# - variable: MetaData/pressure
# - variable: ObsType/airTemperature
# - variable: ObsValue/airTemperature
# - variable: QCflagsData/airTemperature

#- filter: GOMsaver
# filename: ./data/geovals/msonet_geovals_rrfs.nc4
Loading
Loading