diff --git a/Docs/source/developers/local_compile.rst b/Docs/source/developers/how_to_compile_locally.rst similarity index 99% rename from Docs/source/developers/local_compile.rst rename to Docs/source/developers/how_to_compile_locally.rst index 8bfa033a92d..9fc1b397b78 100644 --- a/Docs/source/developers/local_compile.rst +++ b/Docs/source/developers/how_to_compile_locally.rst @@ -1,7 +1,7 @@ .. _developers-local-compile: -Fast, Local Compilation -======================= +How to compile locally and fast +=============================== For simplicity, WarpX :ref:`compilation with CMake ` by default downloads, configures and compiles compatible versions of :ref:`central dependencies ` such as: diff --git a/Docs/source/developers/how_to_guides.rst b/Docs/source/developers/how_to_guides.rst new file mode 100644 index 00000000000..093641e01b0 --- /dev/null +++ b/Docs/source/developers/how_to_guides.rst @@ -0,0 +1,13 @@ +.. _development-howtoguides: + +How-to guides +============= + +.. toctree:: + :maxdepth: 1 + + how_to_profile + how_to_test + how_to_run_clang_tidy + how_to_compile_locally + how_to_write_the_docs diff --git a/Docs/source/developers/profiling.rst b/Docs/source/developers/how_to_profile.rst similarity index 99% rename from Docs/source/developers/profiling.rst rename to Docs/source/developers/how_to_profile.rst index 5acea786920..e756d8362c2 100644 --- a/Docs/source/developers/profiling.rst +++ b/Docs/source/developers/how_to_profile.rst @@ -1,7 +1,7 @@ .. _developers-profiling: -Profiling the Code -================== +How to profile the code +======================= Profiling allows us to find the bottle-necks of the code as it is currently implemented. Bottle-necks are the parts of the code that may delay the simulation, making it more computationally expensive. diff --git a/Docs/source/developers/how_to_run_clang_tidy.rst b/Docs/source/developers/how_to_run_clang_tidy.rst new file mode 100644 index 00000000000..bbcd7b80130 --- /dev/null +++ b/Docs/source/developers/how_to_run_clang_tidy.rst @@ -0,0 +1,51 @@ +.. _developers-run_clang_tidy_locally: + +How to run the clang-tidy linter +================================ + +WarpX's CI tests include several checks performed with the `clang-tidy `__ linter. +The complete list of checks performed is defined in the ``.clang-tidy`` configuration file. + +.. dropdown:: clang-tidy configuration file + :color: light + :icon: info + :animate: fade-in-slide-down + + .. literalinclude:: ../../../.clang-tidy + :language: yaml + +Under `Tools/Linter `__, the script ``runClangTidy.sh`` can be used to run the clang-tidy linter locally. + +.. dropdown:: clang-tidy local run script + :color: light + :icon: info + :animate: fade-in-slide-down + + .. literalinclude:: ../../../Tools/Linter/runClangTidy.sh + :language: bash + +It is a prerequisite that WarpX is compiled following the instructions that you find in our :ref:`Users ` or :ref:`Developers ` sections. + +The script generates a wrapper to ensure that clang-tidy is only applied to WarpX source files and compiles WarpX in 1D, 2D, 3D, and RZ geometry, using such wrapper. + +By default WarpX is compiled in single precision with PSATD solver, QED module, QED table generator and embedded boundary in order to ensure broader coverage with the clang-tidy tool. + +Few optional environment variables can be set to tune the behavior of the script: + +* ``WARPX_TOOLS_LINTER_PARALLEL``: set the number of cores used for compilation; + +* ``CLANG``, ``CLANGXX``, and ``CLANGTIDY``: set the version of the compiler and the linter. + +For continuous integration we currently use clang version 15.0.0 and it is recommended to use this version locally as well. +A newer version may find issues not currently covered by CI tests (checks are opt-in), while older versions may not find all the issues. + +Here's an example of how to run the script after setting the appropriate environment variables: + +.. code-block:: bash + + export WARPX_TOOLS_LINTER_PARALLEL=12 + export CLANG=clang-15 + export CLANGXX=clang++-15 + export CLANGTIDY=clang-tidy-15 + + ./Tools/Linter/runClangTidy.sh diff --git a/Docs/source/developers/testing.rst b/Docs/source/developers/how_to_test.rst similarity index 51% rename from Docs/source/developers/testing.rst rename to Docs/source/developers/how_to_test.rst index 57194b54642..12dc653ac61 100644 --- a/Docs/source/developers/testing.rst +++ b/Docs/source/developers/how_to_test.rst @@ -1,35 +1,40 @@ .. _developers-testing: -Testing the Code -================ +How to test the code +==================== -When proposing a code change, you want to make sure that +When you propose code changes, you want to make sure that -* the code change does not break the existing code; -* the code change gives correct results (numerics, physics, etc.). +* the code changes do not break the behavior of the rest of the code; +* the code changes give correct results (numerics, physics, etc.). -WarpX follows the continuous integration (CI) software development practice, where automated builds and tests are run after merging code changes into the main branch. - -While the code is tested regularly remotely (on the cloud when commits are pushed to an open PR, and every night on local clusters), it can also be useful to run tests on your custom input file. +Following the continuous integration (CI) software development practice, WarpX runs automated builds and tests after a commit is pushed to an open PR as well as after a PR is merged into the main branch. How to run pre-commit tests locally ----------------------------------- -First, when proposing a code change, we perform a couple of automated style and correctness checks. +First, WarpX uses `pre-commit `__ to perform automated style and correctness checks. + +Here is how to install ``pre-commit`` locally: + +#. Install ``pre-commit``: + + .. code-block:: sh + + python -m pip install -U pre-commit -If you install the ``pre-commit`` tool on your local machine via +#. Install the git hook scripts: -.. code-block:: sh + .. code-block:: sh - python -m pip install -U pre-commit - pre-commit install + pre-commit install -the style and correctness checks will run automatically on your local machine, after you commit the change and before you push. +If you install ``pre-commit`` locally, the style and correctness checks will run automatically on your computer, after you commit the code changes and before you push them to the remote repository. -If you do not install the ``pre-commit`` tool on your local machine, these checks will run automatically as part of our CI workflows and a commit containing style and correctness changes might be added automatically to your branch. -In that case, you will need to pull that automated commit before pushing further changes. +If you do not install ``pre-commit`` locally, these checks will run automatically as part of our CI workflows and a commit containing style and correctness changes might be added automatically to your branch after you have pushed your own commit. +In that case, you will need to pull that automated commit before pushing further commits. -See `pre-commit.com `__ and our ``.pre-commit-config.yaml`` file in the repository for more details. +The configuration options for ``pre-commit`` are set in the `pre-commit-config.yaml `__ file. How to configure the automated tests ------------------------------------ @@ -43,11 +48,11 @@ A test that requires a build option that was not configured and built will be sk How to run automated tests locally ---------------------------------- -Once your new feature is ready, there are ways to check that you did not break anything. +Once your code changes are ready, there are ways to check that they do not break the rest of the code. WarpX has automated tests running every time a commit is pushed to an open pull request. The input files and scripts used by the automated tests can be found in the `Examples `__ directory, either under `Physics_applications `__ or `Tests `__. -For easier debugging, it can be convenient to run the tests on your local machine by executing CTest as illustrated in the examples below (where we assume that WarpX was configured and built in the directory ``build``): +For easier debugging, it can be convenient to run the tests on your local computer by executing CTest as illustrated in the examples below (where we assume that WarpX was configured and built in the directory ``build``): * List tests available for the current build options: @@ -101,7 +106,7 @@ For easier debugging, it can be convenient to run the tests on your local machin ctest --test-dir build -R "test_3d_langmuir_multi\..*" Note that filtering with ``-R "test_3d_langmuir_multi"`` would include the additional tests that have the same substring in their name and would not be sufficient to isolate a single test. - Note also that the escaping ``\.`` in the regular expression is necessary in order to take into account the fact that each test is automatically appended with the strings ``.run``, ``.analysis`` and possibly ``.cleanup``. + Note also that the escaping ``\.`` in the regular expression is necessary in order to take into account the fact that each test is automatically appended with the strings ``.run``, ``.analysis``, ``.checksum`` and possibly ``.cleanup``. * Run only tests not labeled with the ``slow`` label: @@ -120,74 +125,56 @@ How to add automated tests An automated test typically consists of the following components: * input file or PICMI input script; + * analysis script; -* checksum file. -To learn more about how to use checksums in automated tests, please see the corresponding section :ref:`Checksums on Tests `. +* checksum file. As mentioned above, the input files and scripts used by the automated tests can be found in the `Examples `__ directory, under either `Physics_applications `__ or `Tests `__. Each test directory must contain a file named ``CMakeLists.txt`` where all tests associated with the input files and scripts in that directory must be listed. -A new test can be added by adding a corresponding entry in ``CMakeLists.txt`` as illustrated in the examples below: - -* Add the **regular test** ``test_1d_laser_acceleration``: +A checksum file is a file that contains reference values obtained by computing a chosen checksum for a set of fields. +More precisely, we compute the sums of the absolute values of the arrays corresponding to each field from the results produced by the automated test and compare these checksums with the reference ones stored in the checksum file of that test, with respect to specific tolerances. +This is expected to be sensitive enough to make the automated test fail if the code changes cause significant differences in the final results, thus catching possible bugs. - .. code-block:: cmake +A new test can be added by calling the function ``add_warpx_test`` in ``CMakeLists.txt``. The function has the following signature: - add_warpx_test( - test_1d_laser_acceleration # name - 1 # dims - 2 # nprocs - inputs_test_1d_laser_acceleration # inputs - "analysis.py diags/diag1000100" # analysis - "analysis_default_regression.py --path diags/diag1000100" # checksum - OFF # dependency - ) +.. code-block:: cmake -* Add the **PICMI test** ``test_2d_laser_acceleration_picmi``: + function(add_warpx_test + name # unique test name: + # test_1d_example, test_2d_example_picmi, etc. + dims # dimensionality: 1, 2, 3, RZ + nprocs # number of processes: 1, 2 + inputs # inputs file or PICMI script: + # inputs_test_1d_example, inputs_test_2d_example_picmi.py, "inputs_test_2d_example_picmi.py arg1 arg2", etc. + analysis # custom test analysis command: + # OFF, "analysis.py", "analysis.py arg1 arg2", etc. + checksum # default regression analysis command: + # OFF, "analysis_default_regression.py --path diags/diag1", etc. + dependency # name of base test that must run first (must match name exactly): + # OFF, test_1d_example_prepare, etc. + ) - .. code-block:: cmake +Here's how to add an automated test: - add_warpx_test( - test_2d_laser_acceleration_picmi # name - 2 # dims - 2 # nprocs - inputs_test_2d_laser_acceleration_picmi.py # inputs - "analysis.py diags/diag1000100" # analysis - "analysis_default_regression.py --path diags/diag1000100" # checksum - OFF # dependency - ) +#. Choose the test directory, either an existing one or a new one. -* Add the **restart test** ``test_3d_laser_acceleration_restart``: +#. Add an input file or PICMI input script. + The name must follow the naming conventions described in the section :ref:`developers-testing-naming` below. - .. code-block:: cmake +#. Add a Python analysis script to analyze the results of the test. - add_warpx_test( - test_3d_laser_acceleration_restart # name - 3 # dims - 2 # nprocs - inputs_test_3d_laser_acceleration_restart # inputs - "analysis_default_restart.py diags/diag1000100" # analysis - "analysis_default_regression.py --path diags/diag1000100" # checksum - test_3d_laser_acceleration # dependency - ) +#. Add the test to the ``CMakeLists.txt`` file (add such file if you are adding the test in a new test directory) using the function ``add_warpx_test`` mentioned above. - Note that the restart has an explicit dependency, namely it can run only provided that the original test, from which the restart checkpoint files will be read, runs first. +#. If the test directory is new, add the directory with the command ``add_subdirectory`` in `Physics_applications/CMakeLists.txt `__ or `Tests/CMakeLists.txt `__, depending on where the test directory is located. -* A more complex example. Add the **PICMI test** ``test_rz_laser_acceleration_picmi``, with custom command-line arguments ``--test`` and ``dir``, openPMD time series output, and custom command line arguments for the checksum comparison: +#. If the test directory is new, make a symbolic link to the default regression analysis script ``analysis_default_regression.py`` from `Examples/analysis_default_regression.py `__, by running ``ln -s ../../analysis_default_regression.py analysis_default_regression.py`` from the test directory. - .. code-block:: cmake +#. Run the test locally with ``ctest``, after setting the environment variable ``CHECKSUM_RESET=ON``, in order to generate automatically the checksum file. - add_warpx_test( - test_rz_laser_acceleration_picmi # name - RZ # dims - 2 # nprocs - "inputs_test_rz_laser_acceleration_picmi.py --test --dir 1" # inputs - "analysis.py diags/diag1/" # analysis - "analysis_default_regression.py --path diags/diag1/ --skip-particles --rtol 1e-7" # checksum - OFF # dependency - ) +Once you have added the test, run the test locally again, after resetting ``CHECKSUM_RESET=OFF``, to check that everything works as expected. The ``analysis`` and ``checksum`` commands passed as arguments to ``add_warpx_test`` can be set to ``OFF`` if the intention is to skip the respective analysis for a given test. @@ -196,14 +183,6 @@ If you need a new Python package dependency for testing, please add it in `Regre Sometimes two or more tests share a large number of input parameters. The shared input parameters can be collected in a "base" input file that can be passed as a runtime parameter in the actual test input files through the parameter ``FILE``. -If the new test is added in a new directory that did not exist before, please add the name of that directory with the command ``add_subdirectory`` in `Physics_applications/CMakeLists.txt `__ or `Tests/CMakeLists.txt `__, depending on where the new test directory is located. - -If not already present, the default regression analysis script ``analysis_default_regression.py`` in the examples above must be linked from `Examples/analysis_default_regression.py `__, by executing once the following command from the test directory: - - .. code-block:: bash - - ln -s ../../analysis_default_regression.py analysis_default_regression.py - Here is the help message of the default regression analysis script, including usage and list of available options and arguments: .. code-block:: bash @@ -216,6 +195,42 @@ Here is the help message of the default regression analysis script, including us --skip-fields skip fields when comparing checksums --skip-particles skip particles when comparing checksums +How to reset checksums locally +------------------------------ + +It is possible to reset a checksum file locally by running the corresponding test with ``ctest`` with the environment variable ``CHECKSUM_RESET=ON``. For example: + + .. code-block:: bash + + CHECKSUM_RESET=ON ctest --test-dir build -R laser_acceleration + +Alternatively, it is also possible to reset multiple checksum files using the output of our Azure pipelines, which can be useful for code changes that result in resetting a large numbers of checksum files. +Here's how to do so: + +#. On the GitHub page of the pull request, find (one of) the pipeline(s) failing due to checksum regressions and click on "Details" (highlighted in blue). + + .. figure:: https://gist.github.com/user-attachments/assets/09db91b9-5711-4250-8b36-c52a6049e38e + +#. In the new page that opens up, click on "View more details on Azure pipelines" (highlighted in blue). + + .. figure:: https://gist.github.com/user-attachments/assets/ab0c9a24-5518-4da7-890f-d79fa1c8de4c + +#. In the new page that opens up, select the group of tests for which you want to reset the checksum files (e.g., ``cartesian_3d``) and click on "View raw log". + + .. figure:: https://gist.github.com/user-attachments/assets/06c1fe27-2c13-4bd3-b6b8-8b8941b37889 + +#. Save the raw log as a text file on your computer. + +#. Go to the directory `Tools/DevUtils `__ and run the Python script `update_benchmarks_from_azure_output.py `__ passing the path of the raw log text file as a command line argument: + + .. code:: bash + + python update_benchmarks_from_azure_output.py path/to/raw_log.txt + + This will update the checksum files for all the tests in the raw log that did not pass the checksum analysis. + +.. _developers-testing-naming: + Naming conventions for automated tests -------------------------------------- @@ -231,40 +246,9 @@ Note that we currently obey the following snake\_case naming conventions for tes #. **Base input files** (that is, files collecting input parameters shared between two or more tests) are typically named ``inputs_base_1d``, ``inputs_base_2d``, ``inputs_base_3d`` or ``inputs_base_rz``, possibly followed by additional strings if need be. -Useful tool for plotfile comparison: ``fcompare`` -------------------------------------------------- - -AMReX provides ``fcompare``, an executable that takes two ``plotfiles`` as input and returns the absolute and relative difference for each field between these two plotfiles. For some changes in the code, it is very convenient to run the same input file with an old and your current version, and ``fcompare`` the plotfiles at the same iteration. To use it: - -.. code-block:: sh - - # Compile the executable - cd /Tools/Plotfile/ # This may change - make -j 8 - # Run the executable to compare old and new versions - /Tools/Plotfile/fcompare.gnu.ex old/plt00200 new/plt00200 - -which should return something like - -.. code-block:: sh - - variable name absolute error relative error - (||A - B||) (||A - B||/||A||) - ---------------------------------------------------------------------------- - level = 0 - jx 1.044455105e+11 1.021651316 - jy 4.08631977e+16 7.734299273 - jz 1.877301764e+14 1.073458933 - Ex 4.196315448e+10 1.253551615 - Ey 3.330698083e+12 6.436470137 - Ez 2.598167798e+10 0.6804387128 - Bx 273.8687473 2.340209782 - By 152.3911863 1.10952567 - Bz 37.43212767 2.1977289 - part_per_cell 15 0.9375 - Ex_fp 4.196315448e+10 1.253551615 - Ey_fp 3.330698083e+12 6.436470137 - Ez_fp 2.598167798e+10 0.6804387128 - Bx_fp 273.8687473 2.340209782 - By_fp 152.3911863 1.10952567 - Bz_fp 37.43212767 2.1977289 +Other resources +--------------- + +With regard to testing the code more generally, not necessarily in the context of continuous integration, AMReX provides a number of useful post-processing tools for plotfiles. +The complete list of tools can be found `here `__. +One tool that traditionally stood out as especially useful for core developers and maintainers is `fcompare `__. diff --git a/Docs/source/developers/documentation.rst b/Docs/source/developers/how_to_write_the_docs.rst similarity index 98% rename from Docs/source/developers/documentation.rst rename to Docs/source/developers/how_to_write_the_docs.rst index 5d604bcf9b3..827b5950d80 100644 --- a/Docs/source/developers/documentation.rst +++ b/Docs/source/developers/how_to_write_the_docs.rst @@ -1,7 +1,7 @@ .. _developers-docs: -Documentation -============= +How to write documentation +========================== Doxygen documentation --------------------- diff --git a/Docs/source/developers/run_clang_tidy_locally.rst b/Docs/source/developers/run_clang_tidy_locally.rst deleted file mode 100644 index 3f600019fe7..00000000000 --- a/Docs/source/developers/run_clang_tidy_locally.rst +++ /dev/null @@ -1,55 +0,0 @@ -.. _developers-run_clang_tidy_locally: - -The clang-tidy linter -===================== - -Clang-tidy CI test ------------------- - -WarpX's CI tests include several checks performed with the -`clang-tidy `__ linter -(currently the version 15 of this tool). The complete list of checks -enforced in CI tests can be found in the ``.clang-tidy`` configuration file. - -.. dropdown:: clang-tidy configuration file - :color: light - :icon: info - :animate: fade-in-slide-down - - .. literalinclude:: ../../../.clang-tidy - :language: yaml - -Run clang-tidy linter locally ------------------------------ - -We provide a script to run clang-tidy locally. The script can be run as follows, -provided that all the requirements to compile WarpX are met (see `building from source `). -The script generates a simple wrapper to ensure that `clang-tidy` is only applied to WarpX source files -and compiles WarpX in 1D,2D,3D, and RZ using such wrapper. By default WarpX is compiled in single precision -with PSATD solver, QED module, QED table generator and Embedded boundary in order to find more -potential issues with the `clang-tidy` tool. - -Few optional environment variables can be set to tune the behavior of the script: - -* ``WARPX_TOOLS_LINTER_PARALLEL``: sets the number of cores to be used for the compilation -* ``CLANG``, ``CLANGXX``, and ``CLANGTIDY`` : set the version of the compiler and of the linter - -Note: clang v15 is currently used in CI tests. It is therefore recommended to use this version. -Otherwise, a newer version may find issues not currently covered by CI tests (checks are opt-in) -while older versions may not find all the issues. - -.. code-block:: bash - - export WARPX_TOOLS_LINTER_PARALLEL=12 - export CLANG=clang-15 - export CLANGXX=clang++-15 - export CLANGTIDY=clang-tidy-15 - ./Tools/Linter/runClangTidy.sh - -.. dropdown:: Script Details - :color: light - :icon: info - :animate: fade-in-slide-down - - .. literalinclude:: ../../../Tools/Linter/runClangTidy.sh - :language: bash diff --git a/Docs/source/developers/workflows.rst b/Docs/source/developers/workflows.rst deleted file mode 100644 index f7c81ae70d8..00000000000 --- a/Docs/source/developers/workflows.rst +++ /dev/null @@ -1,14 +0,0 @@ -.. _development-workflows: - -Workflows -========= - -.. toctree:: - :maxdepth: 1 - - profiling - testing - checksum - run_clang_tidy_locally - local_compile - documentation diff --git a/Docs/source/index.rst b/Docs/source/index.rst index 9668620976a..dfdeb3b9530 100644 --- a/Docs/source/index.rst +++ b/Docs/source/index.rst @@ -126,7 +126,7 @@ Development developers/developers developers/doxygen developers/gnumake - developers/workflows + developers/how_to_guides developers/faq .. good to have in the future: .. developers/repostructure