Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Addition of coverage tool to unit tests in workflow #21

Merged
merged 21 commits into from
Jul 12, 2024
Merged

Addition of coverage tool to unit tests in workflow #21

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
f337fcf
Added orchestrated tests to workflow
fmalatino May 21, 2024
a137551
Linting
fmalatino May 21, 2024
63ddb26
Adding FV3_DACEMODE variable to tests
fmalatino May 21, 2024
ebf24de
Added data download and environment variables to translate workflow
fmalatino May 21, 2024
d2cdc8b
Updated setup.py to point to updated NDSL
fmalatino May 22, 2024
cb94382
Updates to translate test to remove gt_cache between tests
fmalatino May 22, 2024
6675f76
Changed setup.py back to track 2024.04.00 release of NDSL
fmalatino May 22, 2024
5a3c1e7
Moving NDSL reference back to develop
fmalatino May 23, 2024
918ff32
Merge branch 'develop' into update/workflow
fmalatino May 25, 2024
5dced20
Increased number of omp threads for testing
fmalatino May 28, 2024
4ae14e1
Adding in dace mode and loglevel environment variables for tests
fmalatino May 28, 2024
a74ee82
Addition of environment variables to have settings be explicit
fmalatino May 28, 2024
ecd8d2a
Changed translate tests in translate.yml to use 1 omp thread
fmalatino May 29, 2024
0828d72
Changed reference to NDSL in setup.py to latest release version
fmalatino May 29, 2024
10ed431
Added coverage package to testing workflow
fmalatino May 29, 2024
453e6b7
Linting
fmalatino May 29, 2024
614aa47
Re-linting
fmalatino May 29, 2024
0f45171
Fixed translate tests to use one rank
fmalatino May 31, 2024
9d3dc6b
Merged develop into update/workflow
fmalatino Jul 9, 2024
c670a1b
Removed translate.yml
fmalatino Jul 9, 2024
c8e244a
Updated minimum python version requirement in doc of setup.py
fmalatino Jul 9, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 43 additions & 0 deletions .github/workflows/translate.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
name: "Translate test (subset)"
on:
push:
pull_request:
types: [opened, synchronize, reopened, ready_for_review, labeled, unlabeled]

Expand Down Expand Up @@ -83,3 +84,45 @@ jobs:
--which_modules=DynCore \
--threshold_overrides_file=./tests/savepoint/translate/overrides/standard.yaml \
./tests/savepoint
- name: Orchestrated dace:cpu FvTp2d
run: |
rm -rf .gt_cache_FV3_A
export FV3_DACEMODE=BuildAndRun
export PACE_FLOAT_PRECISION=64
export PACE_TEST_N_THRESHOLD_SAMPLES=0
export OMP_NUM_THREADS=1
export PACE_LOGLEVEL=Debug
pytest \
-v -s --data_path=./test_data/8.1.3/c12_6ranks_standard/dycore \
--backend=dace:cpu \
--which_modules=FvTp2d \
--threshold_overrides_file=./tests/savepoint/translate/overrides/standard.yaml \
./tests/savepoint
- name: Orchestrated dace:cpu D_SW
run: |
rm -rf .gt_cache_FV3_A
export FV3_DACEMODE=BuildAndRun
export PACE_FLOAT_PRECISION=64
export PACE_TEST_N_THRESHOLD_SAMPLES=0
export OMP_NUM_THREADS=1
export PACE_LOGLEVEL=Debug
pytest \
-v -s --data_path=./test_data/8.1.3/c12_6ranks_standard/dycore \
--backend=dace:cpu \
--which_modules=D_SW \
--threshold_overrides_file=./tests/savepoint/translate/overrides/standard.yaml \
./tests/savepoint
- name: Orchestrated dace:cpu Remapping
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the bare minimum for orchestration. We should probably run some pySHiELD code and the full FVDynamics case as well

Copy link
Contributor Author

@fmalatino fmalatino May 29, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As of commit 453e6b7 the omp threads have been rolled back to 1, the NDSL version is back at 2024.04.00, and added coverage to the test suite. I timed the current test suite to take 5601.23 s (~93.35 mins). Would it make sense to add in the pySHiELD code and FVDynamics case in another PR?

Coverage result:

Name                                                       Stmts   Miss Branch BrPart  Cover
--------------------------------------------------------------------------------------------
pyFV3/_config.py                                             281     42     84      2    80%
pyFV3/dycore_state.py                                        120     39     52      1    64%
pyFV3/initialization/analytic_init.py                         20     11      6      0    35%
pyFV3/initialization/init_utils.py                           139     92     12      0    31%
pyFV3/initialization/test_cases/initialize_baroclinic.py      89     67      4      0    24%
pyFV3/initialization/test_cases/initialize_tc.py             164    164      2      0     0%
pyFV3/stencils/a2b_ord4.py                                   252    125     98     12    47%
pyFV3/stencils/c_sw.py                                       174    120     68      3    24%
pyFV3/stencils/d2a2c_vect.py                                 252    136    112      8    39%
pyFV3/stencils/d_sw.py                                       323    132    140     18    51%
pyFV3/stencils/del2cubed.py                                   90     45     40      0    36%
pyFV3/stencils/delnflux.py                                   252    129    155      8    37%
pyFV3/stencils/divergence_damping.py                         201     68     74      7    57%
pyFV3/stencils/dyn_core.py                                   277    122    114     11    46%
pyFV3/stencils/fillz.py                                       78     57     50      0    21%
pyFV3/stencils/fv_dynamics.py                                188    145     80      0    18%
pyFV3/stencils/fv_subgridz.py                                377    331     90      0    14%
pyFV3/stencils/fvtp2d.py                                      83     16     24      2    74%
pyFV3/stencils/fxadv.py                                      141     81     66      3    30%
pyFV3/stencils/map_single.py                                  59     21     16      0    59%
pyFV3/stencils/mapn_tracer.py                                 24      1      8      2    91%
pyFV3/stencils/moist_cv.py                                    52     33     20      0    43%
pyFV3/stencils/neg_adj3.py                                   188    169    106      0     8%
pyFV3/stencils/nh_p_grad.py                                   51     22     10      0    48%
pyFV3/stencils/pe_halo.py                                     11      8     10      0    14%
pyFV3/stencils/pk3_halo.py                                    24     10     10      0    41%
pyFV3/stencils/ppm.py                                         40     26     20      0    30%
pyFV3/stencils/ray_fast.py                                    94     67     84      0    21%
pyFV3/stencils/remap_profile.py                              318    276    220      2    11%
pyFV3/stencils/remapping.py                                  173     77     88      8    41%
pyFV3/stencils/riem_solver3.py                                74     45     30      1    31%
pyFV3/stencils/riem_solver_c.py                               57     31     28      0    33%
pyFV3/stencils/saturation_adjustment.py                      525    400    200      1    30%
pyFV3/stencils/sim1_solver.py                                 77     64     58      0    11%
pyFV3/stencils/temperature_adjust.py                          18     13     10      0    18%
pyFV3/stencils/tracer_2d_1l.py                                94     71     36      0    21%
pyFV3/stencils/updatedzc.py                                   72     36     28      1    45%
pyFV3/stencils/updatedzd.py                                   84     31     24      1    54%
pyFV3/stencils/xppm.py                                       169    122     78      3    32%
pyFV3/stencils/xtp_u.py                                       35     27     16      0    24%
pyFV3/stencils/yppm.py                                       170    122     78      3    33%
pyFV3/stencils/ytp_v.py                                       35     27     16      0    24%
pyFV3/testing/map_single.py                                   15     15      2      0     0%
pyFV3/testing/translate_dyncore.py                            55      3     16      1    94%
pyFV3/testing/translate_fvdynamics.py                         81     52     24      0    30%
pyFV3/testing/validation.py                                   89     62     20      0    27%
pyFV3/utils/functional_validation.py                          31     25      4      0    17%
pyFV3/version.py                                               2      0      0      0   100%
pyFV3/wrappers/geos_wrapper.py                               275    275     32      0     0%
--------------------------------------------------------------------------------------------
TOTAL                                                       6493   4053   2563     98    33%

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

93 mins and we are only touching 33% of the code? That's... disappointing. Is the coverage report aggregating everything correctly since all the test are done one by one?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, this is very slow, could you try using more openmp threads?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OpenMP won't help, the execution is not slow part, the build is. Frank restricted to one rank test (e.g. testing the data generated on rank 0 instead of the entire cube) and that should take care of the slowness of the test overall

run: |
rm -rf .gt_cache_FV3_A
export FV3_DACEMODE=BuildAndRun
export PACE_FLOAT_PRECISION=64
export PACE_TEST_N_THRESHOLD_SAMPLES=0
export OMP_NUM_THREADS=1
export PACE_LOGLEVEL=Debug
pytest \
-v -s --data_path=./test_data/8.1.3/c12_6ranks_standard/dycore \
--backend=dace:cpu \
--which_modules=Remapping \
--threshold_overrides_file=./tests/savepoint/translate/overrides/standard.yaml \
./tests/savepoint
2 changes: 1 addition & 1 deletion setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
]

test_requirements = ["pytest", "pytest-subtests", "serialbox"]
ndsl_requirements = ["ndsl @ git+https://github.com/NOAA-GFDL/NDSL.git@2024.04.00"]
ndsl_requirements = ["ndsl @ git+https://github.com/NOAA-GFDL/NDSL.git@develop"]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bad reference version.

develop_requirements = test_requirements + ndsl_requirements + ["pre-commit"]

extras_requires = {
Expand Down
Loading