Skip to content

Commit

Permalink
Issue 66: simplify and split CI actions (#67)
Browse files Browse the repository at this point in the history
* switch to more basic pre-commit from pyrenew

* remove concurrency

* Update .pre-commit-config.yaml

* Add back some hooks

* Update .pre-commit-config.yaml

* Delete .pre-commit-config_max.yaml

* add spelling pre-commit hook

* move script because not a "hook" anymore

* Create render-README.yaml

* Update render-README.yaml

* Update README.md

* Update .pre-commit-config.yaml

* revert pre-commit file minus readme render and with security hook

* pre-commit fixes

* minor test changes in quarto readme

---------

Co-authored-by: AFg6K7h4fhy2 <[email protected]>
  • Loading branch information
SamuelBrand1 and AFg6K7h4fhy2 authored Feb 11, 2025
1 parent ae479f0 commit 10d2ab4
Show file tree
Hide file tree
Showing 16 changed files with 344 additions and 103 deletions.
6 changes: 0 additions & 6 deletions .github/workflows/pre-commit.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,4 @@ jobs:
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- name: Set up Quarto
uses: quarto-dev/quarto-actions/setup@v2
- name: Install Poetry
run: pip install poetry
- name: Install packages
run: poetry install --with dev
- uses: pre-commit/[email protected]
23 changes: 23 additions & 0 deletions .github/workflows/render-README.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
name: render-readme

on:
pull_request:
push:
branches: [main]

jobs:
render-readme:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- name: Set up Quarto
uses: quarto-dev/quarto-actions/setup@v2
- name: Install Poetry
run: pip install poetry
- name: Install packages
run: poetry install --with dev
- name: Render README
run: poetry run bash scripts/render-quarto-readme.sh
77 changes: 55 additions & 22 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,6 @@ repos:
- id: end-of-file-fixer
- id: mixed-line-ending
- id: trailing-whitespace

- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v5.0.0
hooks:
Expand Down Expand Up @@ -56,49 +55,83 @@ repos:
- id: check-executables-have-shebangs
files: \.sh$
################################################################################
# QUARTO README RENDERING
# PYTHON
################################################################################
- repo: local
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.9.4
hooks:
- id: render-quarto-readme
name: "render README.qmd as README.md"
entry: "./scripts/hooks/render-quarto-readme.sh"
language: script
always_run: true
files: "README.(q|)md"
# "currently, the Ruff formatter does not sort imports.
# In order to both sort imports and format, call
# the Ruff linter and then the formatter:"
- id: ruff
args: [
"check",
"--select",
# isort
"I",
"--fix"]
# run ruff linter; the Ruff Linter is an extremely fast Python linter
# designed as a drop-in replacement for Flake8 (plus dozens of plugins),
# isort, pydocstyle, pyupgrade, autoflake, and more
- id: ruff
args: [
# ambiguous variable name: {name}
"--ignore=E741",
# do not assign a lambda expression, use a def
"--ignore=E731",
# found useless expression. ignore since .qmd displays
"--ignore=B018",
# {name} is too complex ({complexity} > {max_complexity})
# note: ignored on select repositories
"--ignore=C901",
# E and W: pycodestyle, standard PEP8 errors and pycodestyle warnings.
# F: pyflakes warnings (e.g., unused variables, undefined names).,
# B: flake8-bugbear (useful best practices).
# SIM: flake8-simplify
# C90: McCabe complexity (cyclomatic complexity).
# UP: pyupgrade, Python version compatibility
"--select=E,W,F,B,C90,UP,SIM",
# linter checks for lines, but doesn't fix, default is 88
"--line-length=79",
# lint all files in the current directory, and fix any fixable errors.
"--fix"]
# run the ruff-formatter; the Ruff formatter is an extremely fast
# Python code formatter designed as a drop-in replacement for Black
- id: ruff-format
args: [
"--line-length=79",
]
################################################################################
# PYTHON
# SECURITY
################################################################################
- repo: https://github.com/PyCQA/isort
rev: 5.13.2
hooks:
- id: isort
args: ["--profile", "black", "--line-length", "79"]
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.8.3
- repo: https://github.com/Yelp/detect-secrets
rev: v1.5.0
hooks:
- id: ruff
args: ["--ignore=E741", "--ignore=E731", "--fix"]
# must first run
# detect-secrets scan > .secrets.baseline
- id: detect-secrets
args: ["--baseline", ".secrets.baseline"]
exclude: package.lock.json
################################################################################
# GITHUB ACTIONS
################################################################################
- repo: https://github.com/rhysd/actionlint
rev: v1.7.4
rev: v1.7.7
hooks:
- id: actionlint
################################################################################
# SPELLING
################################################################################
- repo: https://github.com/crate-ci/typos
rev: typos-dict-v0.11.37
rev: typos-dict-v0.12.4
hooks:
- id: typos
args: ["--force-exclude"]
################################################################################
# COMMIT MESSAGES
################################################################################
- repo: https://github.com/commitizen-tools/commitizen
rev: v4.1.0
rev: v4.1.1
hooks:
- id: commitizen
- repo: https://github.com/jorisroovers/gitlint
Expand Down
112 changes: 112 additions & 0 deletions .secrets.baseline
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
{
"version": "1.4.0",
"plugins_used": [
{
"name": "ArtifactoryDetector"
},
{
"name": "AWSKeyDetector"
},
{
"name": "AzureStorageKeyDetector"
},
{
"name": "Base64HighEntropyString",
"limit": 4.5
},
{
"name": "BasicAuthDetector"
},
{
"name": "CloudantDetector"
},
{
"name": "DiscordBotTokenDetector"
},
{
"name": "GitHubTokenDetector"
},
{
"name": "HexHighEntropyString",
"limit": 3.0
},
{
"name": "IbmCloudIamDetector"
},
{
"name": "IbmCosHmacDetector"
},
{
"name": "JwtTokenDetector"
},
{
"name": "KeywordDetector",
"keyword_exclude": ""
},
{
"name": "MailchimpDetector"
},
{
"name": "NpmDetector"
},
{
"name": "PrivateKeyDetector"
},
{
"name": "SendGridDetector"
},
{
"name": "SlackDetector"
},
{
"name": "SoftlayerDetector"
},
{
"name": "SquareOAuthDetector"
},
{
"name": "StripeDetector"
},
{
"name": "TwilioKeyDetector"
}
],
"filters_used": [
{
"path": "detect_secrets.filters.allowlist.is_line_allowlisted"
},
{
"path": "detect_secrets.filters.common.is_ignored_due_to_verification_policies",
"min_level": 2
},
{
"path": "detect_secrets.filters.heuristic.is_indirect_reference"
},
{
"path": "detect_secrets.filters.heuristic.is_likely_id_string"
},
{
"path": "detect_secrets.filters.heuristic.is_lock_file"
},
{
"path": "detect_secrets.filters.heuristic.is_not_alphanumeric_string"
},
{
"path": "detect_secrets.filters.heuristic.is_potential_uuid"
},
{
"path": "detect_secrets.filters.heuristic.is_prefixed_with_dollar_sign"
},
{
"path": "detect_secrets.filters.heuristic.is_sequential_string"
},
{
"path": "detect_secrets.filters.heuristic.is_swagger_file"
},
{
"path": "detect_secrets.filters.heuristic.is_templated_secret"
}
],
"results": {},
"generated_at": "2023-09-24T19:52:08Z"
}
29 changes: 0 additions & 29 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@
render a GitHub README, see
<https://quarto.org/docs/output-formats/gfm.html>
-->

<!-- ```{python}
#| echo: false
import polars as pl
Expand Down Expand Up @@ -34,9 +33,7 @@ Notes:
[pyrenew-hew](https://github.com/CDCgov/pyrenew-hew).

<details>

<summary>

A Tentative Utilities Diagram
</summary>

Expand Down Expand Up @@ -384,9 +381,7 @@ given the expectation that no one would ever call it; however, for
reproducibility’s sake, the following is included here)

<details>

<summary>

Some Of The Forecast Code
</summary>

Expand Down Expand Up @@ -762,9 +757,7 @@ CDC use of GitHub does not imply an endorsement of any one particular
service, product, or enterprise.

<details>

<summary>

Rules, Policy, And Collaboration
</summary>

Expand All @@ -776,14 +769,10 @@ Rules, Policy, And Collaboration
- [Code of Conduct](./rules-and-policies/code-of-conduct.md)

</details>

<details>

<summary>

Public Domain Standard Notice
</summary>

This repository constitutes a work of the United States Government and
is not subject to domestic copyright protection under 17 USC § 105. This
repository is in the public domain within the United States, and
Expand All @@ -794,11 +783,8 @@ contributions to this repository will be released under the CC0
dedication. By submitting a pull request you are agreeing to comply with
this waiver of copyright interest.
</details>

<details>

<summary>

License Standard Notice
</summary>

Expand All @@ -821,25 +807,18 @@ http://www.apache.org/licenses/LICENSE-2.0.html
The source code forked from other open source projects will inherit its
license.
</details>

<details>

<summary>

Privacy Standard Notice
</summary>

This repository contains only non-sensitive, publicly available data and
information. All material and community participation is covered by the
[Disclaimer](DISCLAIMER.md) and [Code of Conduct](code-of-conduct.md).
For more information about CDC’s privacy policy, please visit
[http://www.cdc.gov/other/privacy.html](https://www.cdc.gov/other/privacy.html).
</details>

<details>

<summary>

Contributing Standard Notice
</summary>

Expand All @@ -857,27 +836,19 @@ through CDC including this GitHub page may be subject to applicable
federal law, including but not limited to the Federal Records Act, and
may be archived. Learn more at <http://www.cdc.gov/other/privacy.html>.
</details>

<details>

<summary>

Records Management Standard Notice
</summary>

This repository is not a source of government records, but is a copy to
increase collaboration and collaborative potential. All government
records will be published through the [CDC web
site](http://www.cdc.gov).
</details>

<details>

<summary>

Additional Standard Notices
</summary>

Please refer to [CDC’s Template
Repository](https://github.com/CDCgov/template) for more information
about [contributing to this
Expand Down
2 changes: 1 addition & 1 deletion README.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -230,7 +230,7 @@ print(idata_w_dates["observed_data"]["obs"]["obs_dim_0"][:15])
```{python}
# idata without dates as coordinates
idata_wo_dates = forecasttools.nhsn_flu_forecast_wo_dates
print(idata_wo_dates["observed_data"]["obs"]["obs_dim_0"][:15])
print(idata_wo_dates["observed_data"]["obs"]["obs_dim_0"][:20])
```

The forecast was generated following the creation of `nhsn_hosp_flu.csv` (see previous section) by running `data.py` with the following added:
Expand Down
4 changes: 3 additions & 1 deletion forecasttools/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,9 @@
"location_table.parquet"
)
location_table = pl.read_parquet(location_table_path)
united_states = location_table.filter(pl.col("is_state")).get_column("long_name").to_list()
united_states = (
location_table.filter(pl.col("is_state")).get_column("long_name").to_list()
)

# load example flusight submission
example_flusight_submission_path = importlib.resources.files(
Expand Down
Loading

0 comments on commit 10d2ab4

Please sign in to comment.