Table of Contents
- After you have cloned the Git repository, you will need to:
- create the Conda environment lock files for the dependencies
- create a virtual Conda environment for development, where to install the dependencies of the project
- execute the tests
- setup Git LFS if needed
- configure the pre-commit hooks for static code analysis and auto-formatting
- configure the Python IDE (PyCharm)
First, you need to create the Conda environment lock files (*.conda.lock.yml) for the dependencies defined
in pyproject.toml.
Note
As a prerequisite, you need to install some packages in your base Conda environment. To do so,
simply execute devtools\setup-conda-base.bat.
Then, to create the Conda environment lock files, execute devtools\run_conda_lock.bat,
or run from command line:
$ (base) python devtools/run_conda_lock.py
It will create or update .conda.lock.yml files in the environments folder:
one for runtime dependencies, and one for development dependencies (with the -dev suffix),
for each combinations of Python versions and platforms.
The platforms are specified in conda-lock section of the pyproject.toml file:
[tool.conda-lock]
platforms = ['win-64', 'linux-64']The python versions are specified at the beginning of the devtools/run_conda_lock.py file:
_python_versions = ["3.10", "3.11"]The Install_or_Update.bat and the setup-dev.bat will use them to install the environment.
For development, you need a Conda environments. you can install it running the setup-dev.bat or:
$ [path\to\surface-apps]\devtools\setup-dev.bat
This command install a local environment at the base of your repository: .conda-env.
This environment should automatically be recognized by the Conda installation.
To activate this environment, run the following command from the root of the project:
$ conda activate ./.conda-env
Dependencies are listed in pyproject.toml with version constraints.
Versions are then locked using conda-lock as previously described.
Anytime dependencies are added to or removed from the pyproject.toml file,
regenerate the Conda environment lock files, using devtools\run_conda_lock.bat,
or directly from command line:
(base) $ python devtools/run_conda_lock.py
Regenerate the Conda environment lock files as well when you want to fetch newly
available versions of the dependencies (typically patches, still in accordance with
the specifications expressed in pyproject.toml).
First install the dependency using conda:
(path/to/.conda-env) $ conda install my_new_dep
Then update the list of dependencies in pyproject.toml with a suited version constraint
(if for development only, place it under section [tool.poetry.group.dev.dependencies]).
For example, if conda installed version 1.5.2 of my_new_dep,
then add my_new_dep="^1.5.2".
Do not forget to regenerate the Conda environment lock files.
Poetry provides a command line interface to easily add or remove dependencies:
(path/to/.conda-env) $ poetry add another_package --lock
Note the --lock option, that simple creates or updates the lock file, without Poetry installing anything.
poetry would install the package through pip while we want dependencies to be installed through conda
so that they match the version pinned by conda-lock.
One limitation though: Poetry will look for packages in PyPI only and not in the Conda channels. The version selected by Poetry might thus not be aviaible for Conda.
To install Poetry on your computer, refer to the Poetry documentation.
pre-commit is used to automatically run static code analysis upon commit. The list of tools to execute upon commit is configured in the file .pre-commit-config.yaml.
pre-commit can be installed using a Python installation on the system, or one from a Conda environment.
- To install pre-commit using Python (and pip) in your system path:
pip install --user pre-commit- Or to install from an activated Conda environment:
conda install -c conda-forge pre-commitThen, in either way, install the pre-commit hooks as follow (current directory is the project folder):
pre-commit installTo prepare and check the commit messages, you can also use the following commands:
pre-commit install -t prepare-commit-msg -t commit-msgIt configures pre-commit to prepares and checks the commit ensuring it has a JIRA issue ID:
if no ID was provided, it extracts it from the branch name;
if one was provided, it checks it is the same one as in the branch name.
To run pre-commit manually, use the following command:
pre-commit run --all-filesTo run only on changes staged for commit:
pre-commit runIf a tool fails running, it might be caused by an obsolete versions of the tools that pre-commit is trying to execute. Try the following command to update them:
pre-commit autoupdateUpon every commit, all the pre-commit checks run automatically for you, and reformat files when required. Enjoy...
If you prefer to run pre-commit upon push, and not upon every commit, use the following commands:
pre-commit uninstall -t pre-commit
pre-commit install -t pre-pushTest files are placed under the tests folder. Inside this folder and sub-folders,
Python test files are to be named with _test.py as a suffix.
The test function within this files must have a test_ prefix.
If you installed your environment through setup-dev.bat, pytest is already installed.
You can run it from the Conda command (in your project folder):
pytest testsIf you installed your environment through setup-dev.bat, pytest-cov is already installed.
It allows you to visualize the code coverage of your tests.
You can run the tests from the console with coverage:
pytest --cov --cov-report htmlThe html report is generated in the folder htmlcov at the root of the project.
You can then explore the report by opening index.html in a browser.
In pyproject.toml, the section [tool.coverage.report] defines the common options
for the coverage reports. The minimum accepted percentage of code coverage is specified
by the option fail_under.
The section [tool.coverage.html] defines the options specific to the HTML report.
In the case your package requires large files, git-lfs can be used to store those files. Copy it from the git-lfs website, and install it.
Then, in the project folder, run the following command to install git-lfs:
git lfs installIt will update the file .gitattributes with the list of files to track.
Then, add the files and the .gitattributes to the git repository, and commit.
Then, add the files to track with git-lfs:
git lfs track "*.desire_extension"PyCharm, by JetBrains, is a very good IDE for developing with Python.
First, excluded the .conda-env folder from PyCharm.
Do so, in PyCharm, right-click on the .conda-env folder, and Mark Directory as > Excluded.
Then, you can add the Conda environment as a Python interpreter in PyCharm.
In PyCharm settings, open File > Settings, go to Python Interpreter,
and add click add interpreter (at the top left):
Select Conda Environment, Use existing environment,
and select the desired environment from the list (the one in the .conda-env folder):
Then you can check the list of installed packages in the Packages table. You should see
surface-apps and its dependencies. Make sure to turn off the Use Conda Package Manager
option to see also the packages installed through pip:
First, right click on the tests folder and select Mark Directory as > Test Sources Root:
You can now start tests with a right click on the tests folder and
select Run 'pytest in tests', or select the folder and just hit Ctrl+Shift+F10.
PyCharm will nicely present the test results and logs:
You can run the tests with a nice report of the code coverage, thanks to the pytest-cov plugin (already installed in the virtual environment as development dependency as per pyproject.toml).
To set up this option in PyCharm, right click on the tests folder and Modify Run Configuration...,
then add the following option in the Additional Arguments field:
Select pytest in tests, and add the following option in the Additional Arguments field:
--cov --cov-report html
Then, run the tests as usual, and you will get a nice report of the code coverage.
Note
Running tests with coverage disables the debugger, so breakpoints will be ignored.
Here is a suggestion for some plugins you can install in PyCharm.
- Toml, to edit and validate
pyproject.tomlfile. - IdeaVim, for Vim lovers.
- GitHub Copilot, for AI assisted coding.
# TODO: ADD LICENSE TERMS
Copyright (c) 2024 Mira Geoscience Ltd.