From df141574d8c12c825f239f5f85d546d030c9bc93 Mon Sep 17 00:00:00 2001 From: "Patrick J. Roddy" Date: Tue, 22 Oct 2024 12:00:27 +0100 Subject: [PATCH] Add `markdownlint` and tidy up `markdown` (#465) --- .markdownlint.yaml | 2 + .pre-commit-config.yaml | 6 + CONTRIBUTING.md | 2 + README.md | 174 +++++++++--------- docs/pages/benchmarking-profiling.md | 10 +- docs/pages/ci.md | 10 +- docs/pages/community.md | 2 +- docs/pages/docs.md | 13 +- docs/pages/libraries/bindings.md | 8 +- docs/pages/libraries/clis.md | 4 +- docs/pages/libraries/guis.md | 2 +- docs/pages/libraries/index.md | 2 +- docs/pages/libraries/jupyter-notebooks.md | 8 +- docs/pages/libraries/logging.md | 2 +- docs/pages/libraries/parallel-async.md | 10 +- docs/pages/linting.md | 6 +- docs/pages/packaging.md | 10 +- docs/pages/refactoring.md | 2 +- docs/pages/templates.md | 8 +- docs/pages/testing.md | 6 +- docs/pages/virtual.md | 2 +- tests/test_package_generation.py | 1 + tutorial.md | 88 +++++---- .../.markdownlint.yaml | 2 + .../.pre-commit-config.yaml | 6 + {{cookiecutter.project_slug}}/LICENSE.md | 2 + {{cookiecutter.project_slug}}/README.md | 11 -- {{cookiecutter.project_slug}}/docs/LICENSE.md | 2 + {{cookiecutter.project_slug}}/docs/index.md | 2 + 29 files changed, 218 insertions(+), 185 deletions(-) create mode 100644 .markdownlint.yaml create mode 100644 {{cookiecutter.project_slug}}/.markdownlint.yaml diff --git a/.markdownlint.yaml b/.markdownlint.yaml new file mode 100644 index 00000000..02907b2b --- /dev/null +++ b/.markdownlint.yaml @@ -0,0 +1,2 @@ +--- +MD013: false diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 8427b380..ba5792d5 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -9,6 +9,12 @@ repos: - id: ruff-format args: - --config=pyproject.toml + - repo: https://github.com/igorshubovych/markdownlint-cli + rev: v0.42.0 + hooks: + - id: markdownlint-fix + args: + - --dot - repo: https://github.com/Lucas-C/pre-commit-hooks rev: v1.5.5 hooks: diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 6f8f2f64..3bc57e5f 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,3 +1,5 @@ +# Contributing Guide + This template and our [recommendation pages][website] were made by [research software engineers] at [UCL's Centre for Advanced Research Computing][UCL ARC]. We made it with research software projects in mind, but whoever you are, we hope diff --git a/README.md b/README.md index fb33fd93..b4b9b988 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,9 @@ +
UCL ARC Python tooling logo

UCL ARC Python Recommendations

+ This repository collects the [UCL ARC] recommendations for a research software project in Python. It contains a template for new Python packages and a @@ -73,80 +75,86 @@ We also have a detailed [tutorial](tutorial.md) that has been given in a couple The tutorial goes into much more pedagogical detail, it both describes using the template to create a package and how to use the newly created package with some of the tools included. -1. Install [cookiecutter] in a Conda or `venv` environment (commented lines for - Conda example). - - ``` - # conda create --channel conda-forge --name new-env-name - # conda activate new-env-name - # conda install pip - pip install cookiecutter - ``` - -2. Run cookiecutter in the desired directory - ``` - cookiecutter gh:ucl-arc/python-tooling - ``` - If you have this repo locally (this may be the case if you are developing), - you can run the following: - ``` - cookiecutter /path/to/your/checkout/of/python-tooling - ``` -3. A series of questions will pop up to configure the project. Type the answer - or hit return to use the default option (shown in parenthesis). - - Note that these project variables are defined in the `cookiecutter.json` - file. - -4. This will create a specific directory structure. - - For example, for a project with the following variables: - - ``` - project_name [Python Template]: PROJECT_NAME - project_slug [python-template]: PROJECT_SLUG - package_name [python_template]: PACKAGE_NAME - ``` - - we will get a project folder named `PROJECT_SLUG`, structured like this: - - ``` - PROJECT_SLUG - ├── ... - ├── README.md - ├── pyproject.toml - ├── src - │ └── PACKAGE_NAME - │ └── __init__.py - └── tests - └── test_dummy.py - ``` - - And the `PROJECT_NAME` will appear in the README.md as the human-readable - name of the project. - - ``` - cat PROJECT_SLUG/README.md - # PROJECT_NAME - ... - ``` - -5. To work on your project, initialise a Git repository and _install_ it in - editable mode. - ``` - cd PROJECT_SLUG - git init - python -m pip install -e ".[dev]" - ``` -6. Build your package - ``` - python -m build - ``` +1. Install [cookiecutter] in a Conda or `venv` environment (commented lines for + Conda example). + + ```sh + # conda create --channel conda-forge --name new-env-name + # conda activate new-env-name + # conda install pip + pip install cookiecutter + ``` + +2. Run cookiecutter in the desired directory + + ```sh + cookiecutter gh:ucl-arc/python-tooling + ``` + + If you have this repo locally (this may be the case if you are developing), + you can run the following: + + ```sh + cookiecutter /path/to/your/checkout/of/python-tooling + ``` + +3. A series of questions will pop up to configure the project. Type the answer + or hit return to use the default option (shown in parenthesis). + + Note that these project variables are defined in the `cookiecutter.json` + file. + +4. This will create a specific directory structure. + + For example, for a project with the following variables: + + ```yaml + project_name [Python Template]: PROJECT_NAME + project_slug [python-template]: PROJECT_SLUG + package_name [python_template]: PACKAGE_NAME + ``` + + we will get a project folder named `PROJECT_SLUG`, structured like this: + + ```sh + PROJECT_SLUG + ├── ... + ├── README.md + ├── pyproject.toml + ├── src + │ └── PACKAGE_NAME + │ └── __init__.py + └── tests + └── test_dummy.py + ``` + + And the `PROJECT_NAME` will appear in the README.md as the human-readable + name of the project. + + ```sh + cat PROJECT_SLUG/README.md + # PROJECT_NAME + ... + ``` + +5. To work on your project, initialise a Git repository and _install_ it in + editable mode. + + ```sh + cd PROJECT_SLUG + git init + python -m pip install -e ".[dev]" + ``` + +6. Build your package + + ```sh + python -m build + ``` ## Notes for developers -
-Click to expand... +
Click to expand... First, clone repository @@ -157,7 +165,7 @@ First, clone repository - Clone the repository by typing (or copying) the following line in a terminal at your selected path in your machine: -``` +```sh git clone git@github.com:UCL-ARC/python-tooling.git cd python-tooling ``` @@ -166,7 +174,7 @@ cd python-tooling - To create and remove your virtual environment - ``` + ```sh conda create -n ptoolingVE pip -c conda-forge conda activate ptoolingVE conda deactivate ptoolingVE @@ -176,7 +184,7 @@ cd python-tooling - To run template in the same path of this repo. We do a test cookiecut of a dummy package and install it to ensure the template setup works. - ``` + ```sh cookiecutter . cd python-template git init @@ -185,13 +193,13 @@ cd python-tooling - To run cookiecutter using a specific branch of the template: - ``` + ```sh cookiecutter https://github.com/UCL-ARC/python-tooling --checkout ``` - To run the tests locally: - ``` + ```sh pytest -s ``` @@ -203,13 +211,13 @@ sub-directory, and are written in markdown. To build the webpage locally (for testing) -1. [Install jekyll](https://jekyllrb.com/docs/installation) -2. Run `bundle install` from the `docs/` directory of this repository to - install dependencies. -3. Run `bundle exec jekyll serve` from the root directory of this repository. - This should fire up a local web server and tell you its address. By default - the server will automatically refresh the HTML pages if any changes are made - to the markdown sources. +1. [Install jekyll](https://jekyllrb.com/docs/installation) +2. Run `bundle install` from the `docs/` directory of this repository to + install dependencies. +3. Run `bundle exec jekyll serve` from the root directory of this repository. + This should fire up a local web server and tell you its address. By default + the server will automatically refresh the HTML pages if any changes are made + to the markdown sources. See the [jekyll docs](https://jekyllrb.com/docs) for more info. diff --git a/docs/pages/benchmarking-profiling.md b/docs/pages/benchmarking-profiling.md index 93d9f7c3..dd2521ee 100644 --- a/docs/pages/benchmarking-profiling.md +++ b/docs/pages/benchmarking-profiling.md @@ -3,29 +3,29 @@ title: Benchmarking and profiling layout: default --- -# Benchmarking +## Benchmarking | Name | Short description | 🚦 | | -------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :-: | | [asv](https://asv.readthedocs.io/en/stable/) | A tool for benchmarking Python packages over their lifetime. Allows you to write benchmarks and then run them against every commit in the repository, to identify where performance increased or decreased. Comparative benchmarks can also be run, which can be useful for [running them in CI using GitHub runners](https://labs.quansight.org/blog/2021/08/github-actions-benchmarks). | 🟢 | -# Profiling +## Profiling -## Time +### Time | Name | Short description | 🚦 | | ------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | :-: | | [pyinstrument](https://pyinstrument.readthedocs.io/en/stable) | Python profiler. Tells you how long individual lines of code take to run, so you can focus on the slowest part of your program to speed it up. | 🟢 | | [line_profiler](https://pypi.org/project/line-profiler/) | A tool for line-by-line profiling of functions. | 🟠 | -## Memory +### Memory | Name | Short description | 🚦 | | ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :-: | | [memray](https://bloomberg.github.io/memray/) | Tracks and reports memory allocations, both in Python code and in compiled extension modules. It also has a [plugin](https://pytest-memray.readthedocs.io/en/latest/) for easy integration with pytest. Only works on Linux and macOS. | 🟠 | | [memory_profiler](https://pypi.org/project/memory-profiler/) | No longer actively maintained. A Python module for monitoring memory consumption of a process alongside line-by-line analysis of memory consumption. Might be a useful alternative to memray if you need to do memory profiling on Windows. | 🟠 | -## General/other tools +### General/other tools | Name | Short description | 🚦 | | -------------------------------------------------- | ------------------------------------------------------------------------------------------------- | :-: | diff --git a/docs/pages/ci.md b/docs/pages/ci.md index d11834d5..68c8d0dd 100644 --- a/docs/pages/ci.md +++ b/docs/pages/ci.md @@ -3,7 +3,7 @@ title: Continuous integration layout: default --- -# Continuous integration (CI) +## Continuous integration (CI) | Name | Short description | 🚦 | | ------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | :-: | @@ -13,12 +13,11 @@ layout: default | [Travis CI](https://docs.travis-ci.com/) | Continuous integration and continuous delivery platform. | 🟠 | | [pre-commit.ci](https://pre-commit.ci/) | A bot that adds a pre-commit job to your GitHub Actions CI, and can automatically fix most trivial linting failures. Free for open-source projects. | 🟢 | -
- 🟢 explanation +
🟢 explanation We have many projects using GitHub CI and, it has good integration with GitHub itself, and is free for public repositories (with limited free monthly minutes for private repositories).
-# Coverage monitoring +## Coverage monitoring These services report and track test code coverage over time. They render the code with highlighting to show which lines are not executed by tests. See @@ -30,7 +29,6 @@ during tests. | [Codecov](https://docs.codecov.com/docs) | Hosted service to report code coverage metrics. Occasionally slow to update after a report is updated, can be configured to add extra CI checks. This service is probably more widely used and is [free for both open-source and private projects](https://about.codecov.io/pricing/). | 🟢 | | [Coveralls](https://docs.coveralls.io/) | Hosted service to report code coverage metrics. Very similar to codecov and we don't strongly recommend one over the other. This service is only [free for open-source projects](https://coveralls.io/pricing). | 🟢 | -
- 🟢 explanation +
🟢 explanation Both services are similar, so both 🟢.
diff --git a/docs/pages/community.md b/docs/pages/community.md index 85716264..01cb7292 100644 --- a/docs/pages/community.md +++ b/docs/pages/community.md @@ -3,7 +3,7 @@ title: Community layout: default --- -# Community building +## Community building There are many platforms to build a community for users and developers. We recommend you choose one, and not more than one. If you are creating a new diff --git a/docs/pages/docs.md b/docs/pages/docs.md index 87636315..7315af7a 100644 --- a/docs/pages/docs.md +++ b/docs/pages/docs.md @@ -3,7 +3,7 @@ title: Documentation layout: default --- -# Documentation +## Documentation With Python, as with many other languages, it's very common to automatically create a web page with documentation. This can include reference for the API @@ -23,7 +23,7 @@ If you're using GitHub, one option is to host your docs on [GitHub pages]. [our template]: https://github.com/UCL-ARC/python-tooling?tab=readme-ov-file#using-this-template [template-docs-dot-yaml]: https://github.com/UCL-ARC/python-tooling/blob/main/%7B%7Bcookiecutter.project_slug%7D%7D/.github/workflows/docs.yml -## Documentation build tools +### Documentation build tools | Name | Short description | 🚦 | | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | :-: | @@ -32,8 +32,7 @@ If you're using GitHub, one option is to host your docs on [GitHub pages]. | [gitbook] | General documentation builder; integrates with GitHub. | 🟠 | | [pdoc] | Auto-generates API documentation from docstrings, beginner friendly but with less of a plugin ecosystem than others. | 🟠 | -
-More details about Sphinx +
More details about Sphinx We marginally recommend [MkDocs] over [Sphinx] due to it's ease of use, preference for Markdown, and more support for a variety of docstring styles. @@ -44,7 +43,7 @@ a [Sphinx extension](#sphinx-extensions) that does not have an equivalent [MkDocs plugin](https://github.com/mkdocs/catalog), or if you are part of a community that heavily uses [Sphinx] then we recommend you use that instead. -### See also +#### See also - Our internal discussions about which to recommend ([#16](https://github.com/UCL-ARC/python-tooling/issues/16) and @@ -60,13 +59,13 @@ community that heavily uses [Sphinx] then we recommend you use that instead. [gitbook]: https://www.gitbook.com/ [pdoc]: https://pdoc.dev/ -## MkDocs plugins +### MkDocs plugins | Name | Short description | 🚦 | | ------------------------------------------------------------- | -------------------------------------------- | :-: | | [mkdocstrings-python](https://mkdocstrings.github.io/python/) | Automatically generates API reference pages. | 🟢 | -## Sphinx extensions +### Sphinx extensions | Name | Short description | 🚦 | | -------------------------------------------------------------------- | ---------------------------------------------------------------- | :-: | diff --git a/docs/pages/libraries/bindings.md b/docs/pages/libraries/bindings.md index 6512ad51..05839831 100644 --- a/docs/pages/libraries/bindings.md +++ b/docs/pages/libraries/bindings.md @@ -4,9 +4,9 @@ layout: default parent: Recommended libraries --- -# Bindings +## Bindings -## C/C++ +### C/C++ | Name | Short description | 🚦 | | --------------------------------------------------------------------------------------- | ---------------------------------------------------------------- | :-: | @@ -14,13 +14,13 @@ parent: Recommended libraries | [ctypes](https://docs.python.org/3.8/library/ctypes.html) | Native python method for calling functions in shared C libraries | 🟠 | | [pybind11](https://github.com/pybind/pybind11) | Bindings to C++ with a steep learning curve | 🟠 | -## Rust +### Rust | Name | Short description | 🚦 | | ------------------------------------ | ---------------------------------------------------------------------------------------------- | :-: | | [pyO3](https://github.com/PyO3/pyo3) | Straightforward bindings to rust with support for [packaging](https://github.com/PyO3/maturin) | 🟢 | -## Fortran +### Fortran | Name | Short description | 🚦 | | ---------------------------------------------------------------- | -------------------------------------- | :-: | diff --git a/docs/pages/libraries/clis.md b/docs/pages/libraries/clis.md index 34511d17..bff605e0 100644 --- a/docs/pages/libraries/clis.md +++ b/docs/pages/libraries/clis.md @@ -4,7 +4,7 @@ layout: default parent: Recommended libraries --- -# Command-line interfaces +## Command-line interfaces | Name | Short description | 🚦 | | ----------------------------------------------------------- | ------------------------------------------------------------------------------ | :-: | @@ -13,7 +13,7 @@ parent: Recommended libraries | [argparse](https://docs.python.org/3/library/argparse.html) | Python's builtin CLI system uses object configuration. | 🟠 | | [optparse](https://docs.python.org/3/library/optparse.html) | A now-deprecated CLI system built into Python. | 🔴 | -## Other useful tools for CLIs +### Other useful tools for CLIs | Name | Short description | 🚦 | | ------------------------------- | ----------------------- | :-: | diff --git a/docs/pages/libraries/guis.md b/docs/pages/libraries/guis.md index 5d7aefa2..d68b8049 100644 --- a/docs/pages/libraries/guis.md +++ b/docs/pages/libraries/guis.md @@ -4,7 +4,7 @@ layout: default parent: Recommended libraries --- -# Graphical user interface toolkits +## Graphical user interface toolkits | Name | Short description | 🚦 | | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :-: | diff --git a/docs/pages/libraries/index.md b/docs/pages/libraries/index.md index cd390993..42919b36 100644 --- a/docs/pages/libraries/index.md +++ b/docs/pages/libraries/index.md @@ -4,7 +4,7 @@ layout: default has_children: true --- -# Recommended libraries +## Recommended libraries Python has a large ecosystem of packages and modules beyond what is built in to Python. Here is a list of libraries and toolkits that we've used in our projects, and can recommend (or might recommend that you avoid!). diff --git a/docs/pages/libraries/jupyter-notebooks.md b/docs/pages/libraries/jupyter-notebooks.md index b96ac906..ffb78563 100644 --- a/docs/pages/libraries/jupyter-notebooks.md +++ b/docs/pages/libraries/jupyter-notebooks.md @@ -4,7 +4,7 @@ layout: default parent: Recommended libraries --- -# Jupyter notebooks +## Jupyter notebooks We generally recommend packaging reusable code components into Python modules where possible. However occasionally we support researchers who prefer a @@ -12,7 +12,7 @@ notebook environment or projects which want to provide examples and tutorials in notebook format. Notebooks can also be a valid alternative to Python scripts for running and recording the results of numerical experiments for example. -## Live executable environments +### Live executable environments | Name | Short description | 🚦 | | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | :-: | @@ -20,7 +20,7 @@ running and recording the results of numerical experiments for example. | [Google Colab](https://colab.google/) | Colab is a hosted Jupyter Notebook service that requires no setup to use and provides free access to computing resources, including GPUs and TPUs. | 🟢 | | [GitHub codespaces with JupyterLab](https://docs.github.com/en/codespaces/developing-in-a-codespace/getting-started-with-github-codespaces-for-machine-learning#opening-your-codespace-in-jupyterlab) | A codespace is a development environment that is hosted in the cloud and can run [Python and Jupyter notebooks](https://github.com/github/codespaces-jupyter) | 🟠 | -## Linting +### Linting Many of [our recommended linters](linting) don't work out-of-the box on Jupyter notebooks, however [nbQA] has been found to work well. This is a translation @@ -36,7 +36,7 @@ more reliably than [black] via [nbQA]. | [pre-commit] | Has cell output cleanup hooks (if desired). Also found to work well with [nbQA]. | 🟢 | | `nbqa black` | [black] via [nbQA]. | 🟠 | -## Other +### Other | Name | Short description | 🚦 | | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | :-: | diff --git a/docs/pages/libraries/logging.md b/docs/pages/libraries/logging.md index e21d0070..586b595e 100644 --- a/docs/pages/libraries/logging.md +++ b/docs/pages/libraries/logging.md @@ -4,7 +4,7 @@ layout: default parent: Recommended libraries --- -# Logging +## Logging | Name | Short description | 🚦 | | --------------------------------------------------------- | -------------------------------------------------------------------- | :-: | diff --git a/docs/pages/libraries/parallel-async.md b/docs/pages/libraries/parallel-async.md index 080118dc..fe47f3cc 100644 --- a/docs/pages/libraries/parallel-async.md +++ b/docs/pages/libraries/parallel-async.md @@ -4,7 +4,7 @@ layout: default parent: Recommended libraries --- -# Parallel and asynchronous processing +## Parallel and asynchronous processing Python has a good ecosystem of libraries for parallelising the processing of tasks, as well as asynchronous processing. @@ -20,7 +20,7 @@ a specific interface or parallelisation scheme. Possibly due to the nature of the research problem, the high-performance computing resources available or simply due to pre-existing code using a library like [pandas]. -## Process-based (and thread-based) parallelism +### Process-based (and thread-based) parallelism | Name | Short description | 🚦 | | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :-: | @@ -31,7 +31,7 @@ simply due to pre-existing code using a library like [pandas]. | [mpi4py] | Support for MPI based parallelism. | 🟠 | | [threading] | The standard library module for multi-threading. Due to the _global interpreter lock_ [currently][PEP703] only one thread can execute Python code at a time. | 🔴 | -## Compiler-based parallelism +### Compiler-based parallelism | Name | Short description | 🚦 | | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :-: | @@ -39,14 +39,14 @@ simply due to pre-existing code using a library like [pandas]. | [numba] | [Support for parallelism via `jit(parallel=True)`](https://numba.readthedocs.io/en/stable/user/parallel.html). | 🟠 | | [jax] | [Support for parallelising NumPy / scientific computing like operations using functional transforms](https://jax.readthedocs.io/en/latest/jax-101/06-parallelism.html). | 🟠 | -## Asynchronous processing +### Asynchronous processing | Name | Short description | 🚦 | | -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :-: | | [asyncio] | Python standard library for asynchronous programming with tasks run in a single-threaded event loop. Used for [cooperative multitasking](https://en.wikipedia.org/wiki/Cooperative_multitasking). | 🟠 | | [concurrent.futures] | Another Python standard library for asynchronous processing. Provides a common interface for thread and process based concurrency as an alternative to using `multiprocess(ing)` or `threading` directly. | 🟠 | -## See also +### See also - This [Stack Overflow post](https://stackoverflow.com/a/61360215) is a nice summary of what each of [threading], [multiprocessing], [asyncio] and diff --git a/docs/pages/linting.md b/docs/pages/linting.md index 5c5df1de..8559605f 100644 --- a/docs/pages/linting.md +++ b/docs/pages/linting.md @@ -3,13 +3,13 @@ title: Linting layout: default --- -# Linting +## Linting See [here for an example configuration](https://github.com/UCL-ARC/python-tooling/blob/main/%7B%7Bcookiecutter.project_slug%7D%7D/.pre-commit-config.yaml) for some of these. -## Code formatting +### Code formatting | Name | Short description | 🚦 | | -------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :-: | @@ -27,7 +27,7 @@ for some of these. | [yapf](https://github.com/google/yapf) | Google formatter. | 🟠 | | [flake8](https://flake8.pycqa.org/en/latest/) | Linter which complains if code doesn't follow a rule. Does not support modern `pyproject.toml` configuration. | 🔴 | -## Type checking +### Type checking | Name | Short description | 🚦 | | ---------------------------------------------- | ----------------------------------------------------------------------------- | :-: | diff --git a/docs/pages/packaging.md b/docs/pages/packaging.md index 9da0ecdb..58c0714a 100644 --- a/docs/pages/packaging.md +++ b/docs/pages/packaging.md @@ -3,7 +3,7 @@ title: Packaging layout: default --- -# General packaging +## General packaging A common question facing developers is "_How much code should go into a package?_". Where code to solve a research problem might be large and perform @@ -23,7 +23,7 @@ software tasks. Typically there is a separate git repository per package, and we recommend you stick to this. You can always add the packages as dependencies to a higher-level package which is effectively the same, but much easier to reuse. -# Packaging tools +## Packaging tools | Name | Short description | 🚦 | | --------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | --- | @@ -34,14 +34,14 @@ a higher-level package which is effectively the same, but much easier to reuse. | [bump2version](https://pypi.org/project/bump2version/) | Tool for version-bumping your software. No longer maintained | 🔴 | | [bump-my-version](https://github.com/callowayproject/bump-my-version) | Tool for version-bumping your software. Superseded by setuptools_scm | 🔴 | -# Building +## Building | Name | Short description | 🚦 | | ----------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | --- | | [build](https://pypa-build.readthedocs.io/en/stable/) | Straightforward tool to build a Python package. | 🟢 | | [cibuildwheel](https://cibuildwheel.readthedocs.io) | Builds python wheels for the main operating systems on continuous integration runs (e.g. GitHub actions). | 🟠 | -## Package configuration file +### Package configuration file | Name | Short description | 🚦 | | -------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --- | @@ -49,7 +49,7 @@ a higher-level package which is effectively the same, but much easier to reuse. | [setup.py](https://packaging.python.org/en/latest/guides/distributing-packages-using-setuptools/) | Strongly coupled with setuptools and therefore not recommended. | 🟠 | | [setup.cfg](https://packaging.python.org/en/latest/guides/distributing-packages-using-setuptools/) | An ini file that contains defaults for setup.py commands. | 🟠 | -## Conda +### Conda These tools are helpful if you're looking to publish your package on [conda-forge](https://conda-forge.org/), so users can install it through the diff --git a/docs/pages/refactoring.md b/docs/pages/refactoring.md index c606753b..9d381e05 100644 --- a/docs/pages/refactoring.md +++ b/docs/pages/refactoring.md @@ -3,7 +3,7 @@ title: Refactoring tools layout: default --- -# Refactoring Tools +## Refactoring Tools | Name | Short description | 🚦 | | ----------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | --- | diff --git a/docs/pages/templates.md b/docs/pages/templates.md index dbe2d7f5..83da2e9b 100644 --- a/docs/pages/templates.md +++ b/docs/pages/templates.md @@ -4,9 +4,9 @@ layout: default nav_order: 2 --- -# Package templates +## Package templates -## Recommended ARC template +### Recommended ARC template The [UCL-ARC/python-tooling](https://github.com/UCL-ARC/python-tooling) repository contains our recommended package template for new ARC projects. It @@ -20,7 +20,7 @@ If you're making a package for a community that already has a template in general use (some examples are listed below) we recommend using their template instead. -## Community-specific templates +### Community-specific templates If you're making a package within one of these communities, we recommend using their package template. @@ -31,7 +31,7 @@ their package template. | [SciKit-Surgery](https://github.com/SciKit-Surgery/PythonTemplate) | Cookiecutter template developed by the Wellcome EPSRC Centre for Interventional and Surgical Sciences. | | [Scientific Python](https://github.com/scientific-python/cookie) | Cookiecutter template developed by [SciKit-HEP](https://github.com/scikit-hep) but now adopted by the more general Scientific Python community. | -## Template engines +### Template engines Tools that can be used for creating your own package template. diff --git a/docs/pages/testing.md b/docs/pages/testing.md index 757cf567..ddc784ed 100644 --- a/docs/pages/testing.md +++ b/docs/pages/testing.md @@ -3,9 +3,9 @@ title: Testing layout: default --- -# Testing +## Testing -## Test runners +### Test runners | Name | Short description | 🚦 | | ----------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :-: | @@ -13,7 +13,7 @@ layout: default | [tox](https://tox.wiki/en/latest/index.html) | A framework that allows running tests and packaging in different environments. | 🟢 | | [unittest](https://docs.python.org/dev/library/unittest.html#module-unittest) | Python's built in framework for writing and running tests. Encourages use of classes as test fixtures. | 🟠 | -## pytest plugins +### pytest plugins | Name | Short description | 🚦 | | ---------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :-: | diff --git a/docs/pages/virtual.md b/docs/pages/virtual.md index 70613ff4..6814a577 100644 --- a/docs/pages/virtual.md +++ b/docs/pages/virtual.md @@ -3,7 +3,7 @@ title: Virtual environments layout: default --- -# Virtual environments +## Virtual environments | Name | Short description | 🚦 | | ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :-: | diff --git a/tests/test_package_generation.py b/tests/test_package_generation.py index 71925b63..95f79a5a 100644 --- a/tests/test_package_generation.py +++ b/tests/test_package_generation.py @@ -58,6 +58,7 @@ def test_package_generation( pathlib.Path(".github/workflows/linting.yml"), pathlib.Path(".github/workflows/tests.yml"), pathlib.Path(".gitignore"), + pathlib.Path(".markdownlint.yaml"), pathlib.Path(".pre-commit-config.yaml"), pathlib.Path("CITATION.cff"), pathlib.Path("LICENSE.md"), diff --git a/tutorial.md b/tutorial.md index 35c1af06..bb14cd01 100644 --- a/tutorial.md +++ b/tutorial.md @@ -7,7 +7,7 @@ In this tutorial we will go through in detail the steps required to set-up a Pyt ## ⚙️ Setting up dependencies for using template -
Click to expand... +
Click to expand... To use the template you will need to have at least the following software tools installed, @@ -26,44 +26,52 @@ An easy way to get all the software you need installed is using the tool [Conda] 1. Download and install [Miniconda](https://docs.anaconda.com/free/miniconda/) following the operating system specific instructions at either - - https://docs.anaconda.com/free/miniconda/miniconda-install/ for a graphical installer - - or https://docs.anaconda.com/free/miniconda/#quick-command-line-install for installation via the command line. + - for a graphical installer + - or for installation via the command line. If you already have Miniconda (or Anaconda) installed you can skip this step. 2. Once Miniconda is installed, you need to open a terminal window: -- On Windows: open the Start menu from the taskbar, type `miniconda` in the search field, then click `Anaconda Prompt (miniconda3)` from the results (or `Anaconda Prompt (anaconda3)` if using a previous Anaconda installation). -- On MacOS: click the Launchpad icon in the Dock, type `Terminal` in the search field, then click `Terminal` from the results. -- On Linux: open the default terminal application installed in your distribution. + - On Windows: open the Start menu from the taskbar, type `miniconda` in the search field, then click `Anaconda Prompt (miniconda3)` from the results (or `Anaconda Prompt (anaconda3)` if using a previous Anaconda installation). + - On MacOS: click the Launchpad icon in the Dock, type `Terminal` in the search field, then click `Terminal` from the results. + - On Linux: open the default terminal application installed in your distribution. 3. Once you have a terminal window open you should see text `(base)` as part of your [command prompt](https://en.wikipedia.org/wiki/Command-line_interface#Command_prompt) if Minconda has been installed and set up correctly, indicating you currently have the default `base` environment active. If this is the case you should run - ``` + + ```sh conda create -y -n python-tooling -c conda-forge cookiecutter git gh pre-commit tox ``` + to create a new environment named `python-tooling` in to which will be installed the packages necessary for creating and using a package using the `UCL-ARC/python-tooling` cookiecutter template. + 4. To check that all the dependencies have installed correctly in the new environment first activate the environment by running - ``` + + ```sh conda activate python-tooling ``` + at which point you should see `(base)` in the command prompt change to `(python-tooling)`, Then try running each of the following commands in turn, one at a time, - ``` + + ```sh cookiecutter --version gh --version git --version pre-commit --version tox --version ``` + For each command you should see some text outputted to the terminal giving details of the installed versions of the applications - the output itself is not important as long as you do not see any error messages. + 5. If you also want to try out creating a GitHub repository for the package you will need to [sign-up for a free GitHub account](https://github.com/join) if you don't already have one. Once you have a GitHub account, open a terminal window - you can either use the same one as previously if you still have it open, or open a new terminal window as described in step 2 and then activate the `python-tooling` Conda environment by running - ``` + ```sh conda activate python-tooling ``` Once you have a terminal window with the `python-tooling` environment active (you should see `(python-tooling)` in your command prompt) run - ``` + ```sh gh auth login ``` @@ -84,30 +92,36 @@ We will first go through the steps for creating a new package using the `UCL-ARC 1. Open a terminal window: -- On Windows: open the Start menu from the taskbar, type `miniconda` in the search field, then click `Anaconda Prompt (miniconda3)` from the results. -- On MacOS: click the Launchpad icon in the Dock, type `Terminal` in the search field, then click `Terminal` from the results. -- On Linux: open the default terminal application installed in your distribution. + - On Windows: open the Start menu from the taskbar, type `miniconda` in the search field, then click `Anaconda Prompt (miniconda3)` from the results. + - On MacOS: click the Launchpad icon in the Dock, type `Terminal` in the search field, then click `Terminal` from the results. + - On Linux: open the default terminal application installed in your distribution. 2. In the opened terminal window change the working directory to the path you wish to create the package in using the `cd` (change directory) command. 3. Activate the `python-tooling` Conda environment you previously created ([see instructions above](#%EF%B8%8F-setting-up-dependencies-for-using-template)) by running - ``` + + ```sh conda activate python-tooling ``` + You should now see the text `(python-tooling)` in your [command prompt](https://en.wikipedia.org/wiki/Command-line_interface#Command_prompt). If you installed, or already had installed, the tools listed above in the set up instructions at a system level you can skip this step. + 4. To begin creating the package run - ``` + + ```sh cookiecutter gh:ucl-arc/python-tooling ``` + You will then be shown a series of prompts at the command line asking for details of the project and package. You can choose to use the default placeholder value (shown in parenthesis `()` in prompt) for any question by hitting `Enter`. If you already have a specific project in mind you want to set up a package for using the template you can use this project's details, otherwise you can just use the placeholder values. You should choose `Y` (yes) to the questions on whether to initialise Git repository and automatically deploy HTML documentation to GitHub Pages to allow you to complete the follow on exercises which rely on these options being enabled. For the prompt asking for the GitHub user or organization name to be owner of repository you should supply your GitHub user name. + 5. Once you have completed all the cookiecutter prompts some additional instructions will be printed to screen (which we will come back to in the next sections) and your new package will be generated in a directory named `{project_slug}` in the current working directory (where `{project_slug}` is the value entered for the `'Slugified' project name...`[^slug] prompt, this will be `python-template` if you used the default placeholder values). You can see the directory tree of files generated by running (if on Linux or MacOS) - ``` + ```sh tree {project_slug} ``` or on Windows - ``` + ```sh tree /F {project_slug} ``` @@ -122,7 +136,7 @@ We will first go through the steps for creating a new package using the `UCL-ARC Try viewing the content of some of the files generated by running - ``` + ```sh cat {path_to_file} ``` @@ -136,7 +150,7 @@ The package you created in the previous section will have been initialised local When you completed setting up the package using the `cookiecutter` command you should have seen some additional instructions printed to screen including, providing you have the [GitHub CLI](https://cli.github.com/) `gh` installed, a message of the form -``` +```sh GitHub CLI detected, you can create a repo with the following: gh repo create {github_user}/{project_slug} -d "{project_description}" --public -r origin --source {project_slug} @@ -148,7 +162,7 @@ If you get an error message at this point it may be because you have not install If the command runs successfully you should see a message of the form -``` +```sh ✓ Created repository {github_user}/{project_slug} on GitHub https://github.com/{github_user}/{project_slug} ✓ Added remote https://github.com/{github_user}/{project_slug}.git @@ -158,7 +172,7 @@ A repository should have been created at the printed URL which you should be abl To commit the package files generated by `cookiecutter` locally on the default `main` branch and also push this commit to the repository on GitHub, run each of the commands below in turn, replacing the `{project_slug}` placeholder with the relevant value you used when creating package (`python-template` if you used default) -``` +```sh cd {project_slug} git add . git commit -m "Initial commit" @@ -175,7 +189,7 @@ Depending on your account settings, you may find that the `Documentation` workfl Now that the repository is synchronised to GitHub, a few additional steps are required to configure the repository for the HTML documentation to be automatically deployed to [GitHub Pages](https://pages.github.com/). On completion of the `cookiecutter` command to create the package, a message of the form below should have been displayed -``` +```sh The 'Documentation' GitHub Actions workflow has been set up to push the built HTML documentation to a branch gh-pages on pushes to main for deploying as a GitHub Pages website. To allow the GitHub Actions bot to push to the gh-pages @@ -201,7 +215,7 @@ The first part of the instructions gives details of how to set the permissions f Once the _Documentation_ workflow has successfully completed, a new branch `gh-pages` should have been created in the repository containing the HTML documentation. The second part of the instructions printed in the message output by the `cookiecutter` command indicate to go to a URL `https://github.com/{github_user}/{project_slug}/settings/pages` to set this branch as the source for a GitHub Pages website for the repository. If you now follow those instructions, setting the `Source` to `Deploy from branch` and `Branch` to `gh-pages`, the a new Actions workflow `pages-build-deployment` will be automatically triggered. Once this workflow has completed, you should be able to view the HTML documentation for the repository at a URL -``` +```sh https://{github_user}.github.io/{project_slug} ``` @@ -215,13 +229,13 @@ There are a variety of virtual environment management tools available for Python A Conda environment for the project can be created by running in a terminal the command -``` +```sh conda create -y -n {project_slug} -c conda-forge python ``` This will create a new environment with name `{project_slug}` (which you should replace with the relevant project slug value for your project), installing the latest version of Python using the package hosted on the [community driven `conda-forge` channel](https://conda-forge.org/). To make this Conda environment the current active environment run -``` +```sh conda activate {project_slug} ``` @@ -229,31 +243,31 @@ again replacing `{project_slug}` with the relevant project slug for your package An alternative to Conda is the [`venv` module](https://docs.python.org/3/library/venv.html) built-in to the Python standard library. This has the advantage of being available in any Python (3.3+) environment, but unlike Conda will not by itself allow you to use a different Python version from the system level install. In contrast to Conda which by default creates all environments in a shared user-level directory (if using Miniconda, by default in a directory `miniconda3/envs` in your user or home directory), the `venv` module requires being passed a path in which to create the directory containing the files associated with the virtual environment. A common pattern is to store the virtual environment files in a directory `.venv` within the root directory of the project repository. This can be achieved by running -``` +```sh python -m venv .venv ``` from the root of the project repository. To activate the new virtual environment, if on Linux or MacOS run from the root of the project repository -``` +```sh source .venv/bin/activate ``` or if on Windows -``` +```sh .venv\Scripts\activate ``` Once you have activated the environment you should make sure the version of the Python package manager `pip` installed in the environment is up to date by running -``` +```sh python -m pip install --upgrade pip ``` Once you have created and activated a virtual environment for the project, you can install the package in [editable mode](https://setuptools.pypa.io/en/latest/userguide/development_mode.html), along with both its required dependencies and optional sets of dependencies for development (`dev`), documentation (`docs`) and testing (`test`) by running -``` +```sh python -m pip install --editable ".[dev,docs,test]" ``` @@ -263,7 +277,7 @@ from the root of the project repository. The package template includes support for running tests using the [`pytest` testing framework](https://docs.pytest.org/). Tests are defined in modules in the `tests` directory. By default a single test module `tests/test_dummy.py` is created with a placeholder test. You can run the test locally using `pytest` directly (you will need to have the virtual environment active in which you installed the package and its dependencies) by running -``` +```sh pytest ``` @@ -271,7 +285,7 @@ from the root of the project repository. The package template also sets up [`tox`](https://tox.wiki), an automation tool which allows easily running tests in isolated environments and under multiple Python versions. You can run the tests across all compatible Python versions currently installed using `tox` by running -``` +```sh tox ``` @@ -281,7 +295,7 @@ from the root of the project repository. It can sometimes be useful when editing docstrings or adding additional pages in the `docs` directory to be able to render the HTML documentation locally. The package is set up to use [MkDocs](https://www.mkdocs.org/) to build the documentation with the package API documentation generated using the [`mkdocstrings` plug-in](https://mkdocstrings.github.io/). These packages will have been installed in to you local development environment providing you installed the package with optional `docs` dependencies as recommended above. To build and serve the documentation locally run -``` +```sh mkdocs serve ``` @@ -289,7 +303,7 @@ from the root of the project repository and navigate to `http://127.0.0.1:8000/` A `tox` environment `docs` to build the documentation is also available. This will be build the documentation in an isolated environment and is also used for building the documentation in the GitHub Actions _Documentation_ workflow so can be useful to run locally when debugging issues with the workflow - it can be executed by running -``` +```sh tox -e docs ``` @@ -299,7 +313,7 @@ from the root of the project repository. The built documentation will be output The package is set-up to use [pre-commit](https://pre-commit.com/), a framework for running [Git hook scripts](https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks) on each commit to the repository. In particular the a `.pre-commit-config.yaml` configuration file is provided which will run a series of linters, checks and formatters on the repository such as [ruff](https://docs.astral.sh/ruff/), [mypy](https://mypy.readthedocs.io/en/stable/) and [prettier](https://prettier.io/) on every commit. These Git hook scripts can installed locally by running -``` +```sh pre-commit install ``` @@ -307,7 +321,7 @@ from the root of the project repository. Once installed the hook scripts will be The `pre-commit` hooks can be run against all files in the repository by running -``` +```sh pre-commit run --all-files ``` diff --git a/{{cookiecutter.project_slug}}/.markdownlint.yaml b/{{cookiecutter.project_slug}}/.markdownlint.yaml new file mode 100644 index 00000000..02907b2b --- /dev/null +++ b/{{cookiecutter.project_slug}}/.markdownlint.yaml @@ -0,0 +1,2 @@ +--- +MD013: false diff --git a/{{cookiecutter.project_slug}}/.pre-commit-config.yaml b/{{cookiecutter.project_slug}}/.pre-commit-config.yaml index 5b491cbb..d0b9f320 100644 --- a/{{cookiecutter.project_slug}}/.pre-commit-config.yaml +++ b/{{cookiecutter.project_slug}}/.pre-commit-config.yaml @@ -4,6 +4,12 @@ repos: hooks: - id: ruff - id: ruff-format + - repo: https://github.com/igorshubovych/markdownlint-cli + rev: v0.42.0 + hooks: + - id: markdownlint-fix + args: + - --dot - repo: https://github.com/Lucas-C/pre-commit-hooks rev: v1.5.5 hooks: diff --git a/{{cookiecutter.project_slug}}/LICENSE.md b/{{cookiecutter.project_slug}}/LICENSE.md index 4901310c..866f8c19 100644 --- a/{{cookiecutter.project_slug}}/LICENSE.md +++ b/{{cookiecutter.project_slug}}/LICENSE.md @@ -1,3 +1,5 @@ + + {%- if cookiecutter.license == "MIT" -%} # MIT License diff --git a/{{cookiecutter.project_slug}}/README.md b/{{cookiecutter.project_slug}}/README.md index 3c7c450f..f6ddf6f5 100644 --- a/{{cookiecutter.project_slug}}/README.md +++ b/{{cookiecutter.project_slug}}/README.md @@ -6,12 +6,6 @@ [![Documentation status][documentation-badge]][documentation-link] [![License][license-badge]](./LICENSE.md) - - [tests-badge]: https://github.com/{{cookiecutter.github_username}}/{{cookiecutter.project_slug}}/actions/workflows/tests.yml/badge.svg [tests-link]: https://github.com/{{cookiecutter.github_username}}/{{cookiecutter.project_slug}}/actions/workflows/tests.yml @@ -19,11 +13,6 @@ [linting-link]: https://github.com/{{cookiecutter.github_username}}/{{cookiecutter.project_slug}}/actions/workflows/linting.yml [documentation-badge]: https://github.com/{{cookiecutter.github_username}}/{{cookiecutter.project_slug}}/actions/workflows/docs.yml/badge.svg [documentation-link]: https://github.com/{{cookiecutter.github_username}}/{{cookiecutter.project_slug}}/actions/workflows/docs.yml -[conda-badge]: https://img.shields.io/conda/vn/conda-forge/{{cookiecutter.project_slug}} -[conda-link]: https://github.com/conda-forge/{{cookiecutter.project_slug}}-feedstock -[pypi-link]: https://pypi.org/project/{{cookiecutter.project_slug}}/ -[pypi-platforms]: https://img.shields.io/pypi/pyversions/{{cookiecutter.project_slug}} -[pypi-version]: https://img.shields.io/pypi/v/{{cookiecutter.project_slug}} {%- if cookiecutter.license == "MIT" %} [license-badge]: https://img.shields.io/badge/License-MIT-yellow.svg {%- elif cookiecutter.license == "BSD-3" %} diff --git a/{{cookiecutter.project_slug}}/docs/LICENSE.md b/{{cookiecutter.project_slug}}/docs/LICENSE.md index 8dbe6e25..75b899af 100644 --- a/{{cookiecutter.project_slug}}/docs/LICENSE.md +++ b/{{cookiecutter.project_slug}}/docs/LICENSE.md @@ -1 +1,3 @@ + + {! include-markdown "../LICENSE.md" !} diff --git a/{{cookiecutter.project_slug}}/docs/index.md b/{{cookiecutter.project_slug}}/docs/index.md index 42911bd5..e8a17ad1 100644 --- a/{{cookiecutter.project_slug}}/docs/index.md +++ b/{{cookiecutter.project_slug}}/docs/index.md @@ -1 +1,3 @@ + + {! include-markdown "../README.md" rewrite-relative-urls=false !}