Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enhance TCMPR plotter to read TCDiag lines and filter by one column and plot another column #342

Closed
9 of 26 tasks
bikegeek opened this issue Apr 27, 2023 · 2 comments · Fixed by #446
Closed
9 of 26 tasks

Enhance TCMPR plotter to read TCDiag lines and filter by one column and plot another column #342

bikegeek opened this issue Apr 27, 2023 · 2 comments · Fixed by #446
Assignees
@bikegeek
Labels
alert: NEED ACCOUNT KEY Need to assign an account key to this issue alert: NEED CYCLE ASSIGNMENT Need to assign to a release development cycle METplotpy: Plots priority: high High Priority requestor: DTC/T&E General DTC Testing and Evaluation work required: FOR OFFICIAL RELEASE Required to be completed in the official release for the assigned milestone type: enhancement Improve something that it is currently doing
Milestone
METplotpy-3.0.0

Comments

@bikegeek
Copy link
Collaborator

bikegeek commented Apr 27, 2023 •
edited
Loading

Describe the Enhancement

Add support to the TCMPR plotting code to read TCDiag lines.
Allow filtering by a specified column and plotting of another specified column. Append these TCDiag lines to the TC MPR data.

This work ties in with this use case:
NCAR/TCDiag-Realtime#5

Time Estimate

Estimate the amount of work required here.
Issues should represent approximately 1 to 3 days of work.

Sub-Issues

Consider breaking the enhancement down into sub-issues.

  • Read in the TCDiag line type into a pandas dataframe
  • Allow user to select one column of data and another column to plot via configuration file
  • Append the TCDiag lines to the TCMPR data. Refer to the MET documentation. In Table 24.2 TCMPR, column 85 will be the DIAG_SOURCE from the TCDiag line type, column 86 will be the TRACK_SOURCE, etc.
  • Filter the TCDiag data by diagnostic values to stratify track and intensity errors by diagnostic value (such as shear). Allow the user to specify this diagnostic in the configuration file.

These will not be finished for this issue, moving them to this issue: #442

  • *Allow user to specify threshold values in the config file, ie. when wind shear is >30 kts, plot the distribution of errors.
  • Support any other filtering, customizing the plots by allowing coloring of points based on diagnostic type.

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

Define the source of funding and account keys here or state NONE.

Define the Metadata

Assignee

  • Select engineer(s) Minna Win
  • Select scientist(s) Jonathan Vigh & Kathryn Newman

Labels

  • Select component(s)
  • Select priority
  • Select requestor(s)

Projects and Milestone

  • Select Repository and/or Organization level Project(s) or add alert: NEED PROJECT ASSIGNMENT label
  • Select Milestone as the next official version or Future Versions

Define Related Issue(s)

Consider the impact to the other METplus components.

Enhancement Checklist

See the METplus Workflow for details.

  • Complete the issue definition above, including the Time Estimate and Funding Source.
  • Fork this repository or create a branch of develop.
    Branch name: feature_<Issue Number>_<Description>
  • Complete the development and test your changes.
  • Add/update log messages for easier debugging.
  • Add/update unit tests.
  • Add/update documentation.
  • Push local changes to GitHub.
  • Submit a pull request to merge into develop.
    Pull request: feature <Issue Number> <Description>
  • Define the pull request metadata, as permissions allow.
    Select: Reviewer(s) and Development issues
    Select: Repository level development cycle Project for the next official release
    Select: Milestone as the next official version
  • Iterate until the reviewer(s) accept and merge your changes.
  • Delete your fork or branch.
  • Close this issue.
@bikegeek bikegeek added type: enhancement Improve something that it is currently doing priority: high High Priority alert: NEED ACCOUNT KEY Need to assign an account key to this issue alert: NEED CYCLE ASSIGNMENT Need to assign to a release development cycle requestor: DTC/T&E General DTC Testing and Evaluation work required: FOR OFFICIAL RELEASE Required to be completed in the official release for the assigned milestone METplotpy: Plots labels Apr 27, 2023
@bikegeek bikegeek added this to the METplotpy-2.1.0 milestone Apr 27, 2023
@bikegeek bikegeek moved this from 📋 Backlog to 🔖 Ready in METplus-Analysis-5.1.0 Development Apr 27, 2023
@bikegeek bikegeek mentioned this issue Apr 27, 2023
Open
@bikegeek
Copy link
Collaborator Author

Refer to this issue:
#233

@bikegeek bikegeek mentioned this issue Oct 31, 2023
Open
38 tasks
This was referenced Nov 1, 2023
Closed
Open
bikegeek added a commit that referenced this issue Mar 31, 2024
@bikegeek
issue #342 added two more settings to accomodate plotting for TCDiag …
2f7118e 
…data
@bikegeek bikegeek moved this from 🔖 Ready to 🏗 In progress in METplus-Analysis-6.0.0 Development Apr 3, 2024
@bikegeek bikegeek mentioned this issue May 8, 2024
Open
24 tasks
@bikegeek
Copy link
Collaborator Author

bikegeek commented May 8, 2024

created new issue to address the handling of thresholds and changing marker colors based on diagnostic type. These are beyond the allotted time:

#442

@bikegeek bikegeek linked a pull request May 22, 2024 that will close this issue
Feature 342 tcdiag tcmpr plotter #446
Merged
15 tasks
hankenstein2 added a commit that referenced this issue May 29, 2024
@bikegeek @hankenstein2
Feature 342 tcdiag tcmpr plotter (#446)
5c3cb76 
* Issue #383 modifications to support multiple plot types, list_stat_1 values

* issue #383 modifications to support generating more than one plot type

* issue #383 provide supporting plotting multiple plot types with single config file

* issue #383 provide supporting plotting multiple plot types with single config file

* issue #383 provide supporting plotting multiple plot types with single config file

* clean up comments

* issue #383 provide supporting plotting multiple plot types with single config file

* issue #383 provide supporting plotting multiple plot types with single config file

* issue #383 provide supporting plotting multiple plot types with single config file

* issue #383 provide supporting plotting multiple plot types with single config file

* fix colors and legends

* issue #383 add logging to replace printing to stdout

* issue #383 add logging support

* issue #383 added logging support

* issue #383 modifications to support plotting multiple plot types using one config file and logging to replace printing to stdout

* issue #383 modifications to support multiple plot types in one config file, logging support added to replace print statements

* issue #383 modifications to support multiple plot types in a single config file and logging to replace print statements

* issue #383 modifications to support multiple plot types in one config file and logging to replace print statements

* issue #383 replace print statements with logging

* Issue #383 logging support

* issue #383 modifications to support multiple plot types in a single config file and logging replaces printing to stdout

* issue #383 modifications to support plotting multiple plot types defined in a single config file

* issue #383 modifications to support multiple plot types in a single config file and added logging

* issue #383 add more checking and modifications to support multiple plot types defined in a single config file.  Also added logging

* issue #383 modifications to support plotting multiple plot types defined in a single config file and added logging

* issue #383 update the plot name to include the plot type

* issue #383 TCMPR plot documentation initial content

* changed file permissions

* Added tcmpr_plots to the Table of Contents

* issue #383 plot images added for TCMPR plotter

* Updated plot to match updates to User's Guide

* updates to match User's Guide

* Update the config files for box plot and relperf plots to create only the TK_ERR plot so the y-axis can be more specific.

* Delete docs/Users_Guide/figure/RELPERF_SAMPLE_DATA_ABS(AMAX_WIND-BMAX_WIND)_relperf.png

not relevant

* Delete docs/Users_Guide/figure/BOXPLOT_SAMPLE_DATA_ABS(AMAX_WIND-BMAX_WIND)_boxplot.png

not relevant

* sample data for TCMPR plotter

* issue #383 config for all seven plot types

* issue #383 removed unused figure, replaced with TK_ERR figure for boxplot

* issue #383 removed hard-coded paths

* issue #383 rearrange content for clarity

* issue #383 fix incomplete sentences

* issue #383 added the baseline_file and column_info_file

* issue #383 added instructions for the baseline_file and column_info_file settings

* issue #383 basic system tests for TCMPR plotting

* System tests for TCMPR plotting

* Issue #383 include the tcmpr plotting system tests

* issue #383 explicitly set hfip_bsln to 'no'

* explicitly set hfip_bsln to no in testing

* change comparison syntax for hfip_bsln check

* change file size testing

* change file size testing with assert False for mismatch

* change file size testing-check mean line plots

* comment out file size testing, they are not consistent when run inside containers

* issue #383 Explicitly state that the TCMPR data must have all columns labelled

* issue #342 added two more settings to accomodate plotting for TCDiag data

* Add support for creating line plot

* Support for reading in the tcst reformatted file

* replace printing to stdout with logging

* create log directory if one doesn't already exist

* Added two more settings for the point plot to support generating line plot for TCDiag data

* plot_list replaced with plot_type_list

* Added two settings to support line plot for TCDiag data

* modify formatting

* Refactor to allow user to create either a scatter or line plot.  Decrease opacity to enable better visualization of overlapping points

* reformatted TCDiag from TC-Pairs output, to be used for testing

* Config file for generating TCDiag simple time series for all forecasts and a single initialization

* check for identical length of x- and y-values for line plot

* pull out plot generating code from the main and into it's own function to make this more usable for METplus use cases

* Clean up comments and formatting of the create_plot method

* replace existing logic with code that behaves more like unix mkdir -p to create directories for output

* Delete plots/config directory

* Update tcmpr_config.py

Fixing sonar qube complaint

---------

Co-authored-by: Hank Fisher <[email protected]>
@bikegeek bikegeek closed this as completed Jun 4, 2024
@github-project-automation github-project-automation bot moved this from 🏗 In progress to ✅ Done in METplus-Analysis-6.0.0 Development Jun 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bikegeek bikegeek

Labels
alert: NEED ACCOUNT KEY Need to assign an account key to this issue alert: NEED CYCLE ASSIGNMENT Need to assign to a release development cycle METplotpy: Plots priority: high High Priority requestor: DTC/T&E General DTC Testing and Evaluation work required: FOR OFFICIAL RELEASE Required to be completed in the official release for the assigned milestone type: enhancement Improve something that it is currently doing
Projects
METplus-Analysis-6.0.0 Development
Status: 🏁 Done
Milestone
METplotpy-3.0.0
Development

Successfully merging a pull request may close this issue.

Feature 342 tcdiag tcmpr plotter dtcenter/METplotpy
3 participants
@bikegeek @jprestop @TatianaBurek