sum_stat.py error #2037

Ho-ChunHuang-NOAA · 2023-02-06T20:20:27Z

Ho-ChunHuang-NOAA
Feb 6, 2023

I am plotting the time series of FBAR and OBAR and receive a error message today. The same code/script did not generate this error message. Can you help to resolve it.

::::::
::::::
WARNING: discarding series member with case 2022-06-12 12:00:00 480000 for ('V70C55_BC', 'CONUS')
WARNING: event equalization removed 1008 rows
==== End Python output ====
==== Start Python error ====
Traceback (most recent call last):
File "/opt/metviewer_python/METcalcpy//metcalcpy/sum_stat.py", line 305, in
SUM_STAT.calculate_stats()
File "/opt/metviewer_python/METcalcpy//metcalcpy/sum_stat.py", line 145, in calculate_stats
self.aggregate_special_fields('1')
File "/opt/metviewer_python/METcalcpy//metcalcpy/sum_stat.py", line 238, in aggregate_special_fields
aggregated_values = aggregated_values.append(point_data)
File "/opt/metviewer_python/METviewer_py3.8.6/lib/python3.8/site-packages/pandas/core/frame.py", line 9747, in append
warnings.warn(
FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
==== End Python error ====
Python time 0:00:00.956

::::::::::::::::
::::::::::::::::

I turn the event equal = true but the graphic still plotting the data from 06/01 to 06/14. The V70C55_BC data points starting 06/15.
https://www.emc.ncep.noaa.gov/mmb/hchuang/ftp/CONUS_CMAQ_PM25_TIME_SERIES_DAY2_12Z_202206.png

Answered by TatianaBurek

Feb 8, 2023

@Ho-ChunHuang-NOAA
The Python problem with sum_stat.py script should be fixed.

View full answer

Ho-ChunHuang-NOAA · 2023-02-07T01:09:13Z

Ho-ChunHuang-NOAA
Feb 7, 2023
Author

A figure been produced only when event-equalizer is False, but with following warning
/opt/metviewer_python/METcalcpy//metcalcpy/sum_stat.py:238: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
aggregated_values = aggregated_values.append(point_data)

Additional; Python error, along with the one in previous reporting, with event-equalizer turn on, this time no figures is produced.

==== Start Python error ====
Traceback (most recent call last):
File "/opt/metviewer_python/METplotpy//metplotpy/plots/line/line.py", line 782, in
main()
File "/opt/metviewer_python/METplotpy//metplotpy/plots/line/line.py", line 772, in main
plot = Line(docs)
File "/opt/metviewer_python/METplotpy//metplotpy/plots/line/line.py", line 77, in init
self.input_df = self._read_input_data()
File "/opt/metviewer_python/METplotpy//metplotpy/plots/line/line.py", line 115, in _read_input_data
return pd.read_csv(self.config_obj.parameters['stat_input'], sep='\t',
File "/opt/metviewer_python/METviewer_py3.8.6/lib/python3.8/site-packages/pandas/util/_decorators.py", line 211, in wrapper
return func(*args, **kwargs)
File "/opt/metviewer_python/METviewer_py3.8.6/lib/python3.8/site-packages/pandas/util/_decorators.py", line 331, in wrapper
return func(*args, **kwargs)
File "/opt/metviewer_python/METviewer_py3.8.6/lib/python3.8/site-packages/pandas/io/parsers/readers.py", line 950, in read_csv
return _read(filepath_or_buffer, kwds)
File "/opt/metviewer_python/METviewer_py3.8.6/lib/python3.8/site-packages/pandas/io/parsers/readers.py", line 605, in _read
parser = TextFileReader(filepath_or_buffer, **kwds)
File "/opt/metviewer_python/METviewer_py3.8.6/lib/python3.8/site-packages/pandas/io/parsers/readers.py", line 1442, in init
self._engine = self._make_engine(f, self.engine)
File "/opt/metviewer_python/METviewer_py3.8.6/lib/python3.8/site-packages/pandas/io/parsers/readers.py", line 1735, in _make_engine
self.handles = get_handle(
File "/opt/metviewer_python/METviewer_py3.8.6/lib/python3.8/site-packages/pandas/io/common.py", line 856, in get_handle
handle = open(
FileNotFoundError: [Errno 2] No such file or directory: '/data/mv_data/ho-chun.huang/data/CONUS_CMAQ_O3_TIME_SERIES_DAY2_12Z_0615_0630.data'
==== End Python error ====

0 replies

Ho-ChunHuang-NOAA · 2023-02-07T16:48:11Z

Ho-ChunHuang-NOAA
Feb 7, 2023
Author

I went to MetViewer website and am able to produce the time series with the same dataset (case 20230207_164448; 20230207_164730)

1 reply

jprestop Feb 7, 2023
Maintainer

Hi @Ho-ChunHuang-NOAA. Does this mean that all issues are resolved or are you still encountering problems? Thank you for any clarification you can provide.

Ho-ChunHuang-NOAA · 2023-02-07T17:53:45Z

Ho-ChunHuang-NOAA
Feb 7, 2023
Author

Metviewer is fine but batch jobs for time series job continue to have problems.

…

-- Ho-Chun Huang, Ph.D. Physical Scientist III, Contractor with Lynker in Support of NOAA/NWS/NCEP/EMC, U.S. Department of Commerce 5830 University Research Ct., Rm. 2792 College Park, MD 20740 ***@***.*** ***@***.***> 301-683-3958

On Tue, Feb 7, 2023 at 12:52 PM jprestop ***@***.***> wrote: Hi @Ho-ChunHuang-NOAA <https://github.com/Ho-ChunHuang-NOAA>. Does this mean that all issues are resolved or are you still encountering problems? Thank you for any clarification you can provide. — Reply to this email directly, view it on GitHub <#2037 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALPHE3FE25HYIAZQ2VKMSD3WWKDUDANCNFSM6AAAAAAUTDZOVE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

1 reply

jprestop Feb 7, 2023
Maintainer

Thank you @Ho-ChunHuang-NOAA. Unfortunately, @TatianaBurek is out of the office, but I am tagging her on this so that she can follow up when she returns.

Ho-ChunHuang-NOAA · 2023-02-07T18:14:23Z

Ho-ChunHuang-NOAA
Feb 7, 2023
Author

Hi, Thank you. I need the function to be returned as soon as possible. (will wait for Tatiana's return) We are in the evaluation period for the AQM implementation. I need all the support from the DTC, especially on batch jobs to function normally for the graphic generation. Thanks.

…

-- Ho-Chun Huang, Ph.D. Physical Scientist III, Contractor with Lynker in Support of NOAA/NWS/NCEP/EMC, U.S. Department of Commerce 5830 University Research Ct., Rm. 2792 College Park, MD 20740 ***@***.*** ***@***.***> 301-683-3958

On Tue, Feb 7, 2023 at 12:55 PM jprestop ***@***.***> wrote: Thank you @Ho-ChunHuang-NOAA <https://github.com/Ho-ChunHuang-NOAA>. Unfortunately, @TatianaBurek <https://github.com/TatianaBurek> is out of the office, but I am tagging her on this so that she can follow up when she returns. — Reply to this email directly, view it on GitHub <#2037 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALPHE3FJCBU7NYUVRV2ZVQDWWKEALANCNFSM6AAAAAAUTDZOVE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

0 replies

TatianaBurek · 2023-02-07T18:20:06Z

TatianaBurek
Feb 7, 2023
Collaborator

Ho-ChunHuang-NOAA could you please provide the XML that you use for the batch plotting

0 replies

Ho-ChunHuang-NOAA · 2023-02-07T22:14:39Z

Ho-ChunHuang-NOAA
Feb 7, 2023
Author

@TatianaBurek

XML for ozmax8 no plot generated
plot_cmaq_o3_time_series.xml

XML for o3 even=true, plot generated

plot_cmaq_o3_time_series.xml

CONUS_CMAQ_O3_TIME_SERIES_DAY2_12Z_202206.png
but it should have no data before 6/15. The figure is like event=false although xml said true

XML for pm25 event=true, plot generated

plot_cmaq_pm25_time_series.xml

CONUS_CMAQ_PM25_TIME_SERIES_DAY2_12Z_202206.png
but it should have no data before 6/15. The figure is like event=false although xml said true

0 replies

TatianaBurek · 2023-02-08T16:25:30Z

TatianaBurek
Feb 8, 2023
Collaborator

@Ho-ChunHuang-NOAA
I was able to reproduce the issue with plots cmaq_pm25_time_series and cmaq_ozmax8_time_series and will work to fix it. As a work around, you can use Rscript to create these plots by specifying
<execution_type>Rscript</execution_type>
in the xml or uncheck "Use Python" check box in the "Common" tab
Here are the Rscript plots
https://metviewer.nws.noaa.gov/output/plots/plot_cmaq_pm25_time_series.png
https://metviewer.nws.noaa.gov/output/plots/plot_cmaq_o3_time_series.png
For plot_cmaq_ozmax8_time_series the dates start from 2022-06-16 10:00
You can find jobs configurations in the History panel

0 replies

TatianaBurek · 2023-02-08T16:56:43Z

TatianaBurek
Feb 8, 2023
Collaborator

@Ho-ChunHuang-NOAA
The Python problem with sum_stat.py script should be fixed.

0 replies

Ho-ChunHuang-NOAA · 2023-02-08T17:14:08Z

Ho-ChunHuang-NOAA
Feb 8, 2023
Author

@TatianaBurek

Thank you very much fixing this problem. The time-series now is functioned normally, with event_equal=true and false.

You can close this discussion

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sum_stat.py error #2037

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 9 comments 2 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

sum_stat.py error #2037

Ho-ChunHuang-NOAA Feb 6, 2023

Replies: 9 comments · 2 replies

Ho-ChunHuang-NOAA Feb 7, 2023 Author

Ho-ChunHuang-NOAA Feb 7, 2023 Author

jprestop Feb 7, 2023 Maintainer

Ho-ChunHuang-NOAA Feb 7, 2023 Author

jprestop Feb 7, 2023 Maintainer

Ho-ChunHuang-NOAA Feb 7, 2023 Author

TatianaBurek Feb 7, 2023 Collaborator

Ho-ChunHuang-NOAA Feb 7, 2023 Author

TatianaBurek Feb 8, 2023 Collaborator

TatianaBurek Feb 8, 2023 Collaborator

Ho-ChunHuang-NOAA Feb 8, 2023 Author

Ho-ChunHuang-NOAA
Feb 6, 2023

Replies: 9 comments 2 replies

Ho-ChunHuang-NOAA
Feb 7, 2023
Author

Ho-ChunHuang-NOAA
Feb 7, 2023
Author

jprestop Feb 7, 2023
Maintainer

Ho-ChunHuang-NOAA
Feb 7, 2023
Author

jprestop Feb 7, 2023
Maintainer

Ho-ChunHuang-NOAA
Feb 7, 2023
Author

TatianaBurek
Feb 7, 2023
Collaborator

Ho-ChunHuang-NOAA
Feb 7, 2023
Author

TatianaBurek
Feb 8, 2023
Collaborator

TatianaBurek
Feb 8, 2023
Collaborator

Ho-ChunHuang-NOAA
Feb 8, 2023
Author