dual python embedding with gridded data and point_obs not working

[Wesley-J-Davis]

So for some reason this command is retaining the values set in the default configuration file for point stat instead of overwriting them with the values set in the python embedding.

singularity run /p/work1/daviswj/MET_11_0_1_BASE/met_v11.0.1.simg point_stat PYTHON_NUMPY PYTHON_NUMPY=/p/home/daviswj/src/naaps_verify/aero_python/stable_py/naaps_verify/interactive_demos/UofW_reader.py /p/home/daviswj/src/naaps_verify/aero_python/stable_py/testing_data/PointStatConfig_NAAPS2 -outdir ./testing_data/ -v 5

How I define fcst and obs in my point-stat config file:

fcst = {
   file_type = PYTHON_NUMPY;
   field = [ { name = "lib/naaps2_reader_MET.py";} ];
}

obs = {
   file_type = PYTHON_NUMPY;
   level = 0;
}

The error I get when running:

ERROR : PointStatConfInfo::process_config() -> The number of verification tasks in "obs.field" (5) must be non-zero and match the number in "fcst.field" (1).

The fcst and obs dictionary when examined through verbosity 5 output shows that I have spfh, tmp, hgt, ugrd,and vgrd in my observation data dictionary.

These are the settings in the default point stat config:

fcst = {
   field = [
      {
        name       = "SPFH";
        level      = [ "P500" ];
        cat_thresh = [ >80.0 ];
      },

      {
        name       = "TMP";
        level      = [ "P500" ];
        cat_thresh = [ >273.0 ];
      },

      {
        name       = "HGT";
        level      = [ "P500" ];
        cat_thresh = [ >0.0 ];
      },

      {
        name       = "UGRD";
        level      = [ "P500" ];
        cat_thresh = [ >5.0 ];
      },

      {
        name       = "VGRD";
        level      = [ "P500" ];
        cat_thresh = [ >5.0 ];
      }
   ];

}
obs = fcst;

[CPKalb]

Hi Wesley:

The obs dictionary in your configuration does not have a field array. I think this may be what is causing MET to use the values found in the default file, as the field needs to be specified for both the fcst and obs dictionaries. For point observations with python embedding, you want to use the actual variable name and level of your point data. I'm assuming your python script UofW_reader.py puts data in MET's 11 column format. Here, the variable name and level are given in columns 7 and 8 (var and level). Without seeing the python embedding script, I don't know exactly what the field name and level should be set to. However, the name and level should be set to values that are stored in columns 7 and 8 respectively (you could have more than one variable name and level in these columns and MET will pull the appropriate one.

As an example, if you have temperature at level 0, and column 7 has values of TMP, and column 8 has values of 0, then the configuration would look like this:

obs = {
   file_type = PYTHON_NUMPY;
   field = [ { name = "TMP"; level="0";} ];
}

Does this help solve the error?

Christina

[Wesley-J-Davis]

Strictly doing that in my configs returns this error:

Traceback (most recent call last):
File "/usr/local/share/met/wrappers/write_tmp_point_nc.py", line 56, in
tmp_point_obs.put_data(met_in.met_point_data)
File "/usr/local/share/met/python/met_point_obs.py", line 245, in put_data
self.use_var_id = point_obs_dict['use_var_id']
TypeError: list indices must be integers or slices, not str 

I ended up getting it to work before I left for the day yesterday but had to follow the instructions laid out in this discussion:

#2391

Specifically adding the following lines of code to my point reader. (I'm using version 11.0.1 in a singularity container)

from met_point_obs import convert_point_data
...
out_data=df[['typ','sid','vld','lat','lon','elv','var','lvl','hgt','qc','obs']]
point_data=out_data.values.tolist()
print(point_data)
met_point_data = convert_point_data(point_data)

[CPKalb]

Great! I'm going to lock the discussion. If you have additional questions, feel free to start a new discussion.

Christina