diff --git a/doc/source/contributor_guide/bug_report_and_enhancement_requests.rst b/doc/source/contributor_guide/bug_report_and_enhancement_requests.rst new file mode 100644 index 00000000..c03738e0 --- /dev/null +++ b/doc/source/contributor_guide/bug_report_and_enhancement_requests.rst @@ -0,0 +1,9 @@ +.. _contributor_guide.bug_report_and_enhancement_requests: + +=================================== +Bug Report And Enhancement Requests +=================================== + +.. TODO Add information to this section once the issue templates have been created on GitHub + +When reporting an issue or request, please fill in an `issue `__ on the GitHub repository directly. diff --git a/doc/source/contributor_guide/development_process_details.rst b/doc/source/contributor_guide/development_process_details.rst new file mode 100644 index 00000000..6a20a47f --- /dev/null +++ b/doc/source/contributor_guide/development_process_details.rst @@ -0,0 +1,1072 @@ +.. _contributor_guide.development_process_details: + +=========================== +Development Process Details +=========================== + +Makefile +-------- + +The `smash` build system revolves around a main makefile. You can find full documentation on how to use the ``make`` command and the makefile +`here `__. + +A number of makefile commands are available to automatically run certain processes, such as compiling the code, installing the python library, +running tests, compiling the docs, and so on. Only the main commands are described in detail in this section. To find out about the various stages +of each command, please look directly at the makefile. + +- ``all`` (default command) + Install the smash library in release mode. It compiles the Fortran and C files, generate the shared libraries to + bind with Python and install the smash library. + +- ``dbg`` + Install the smash library in debug mode (similar to the ``all`` command). + + - Add debug flag, ``-g``, to allow you to go through ``gdb`` (`The GNU Project Debugger `__) + + - Add warning flags, ``-Wall`` and ``-Wextra`` + + - Remove optimization flag ``-O3`` + + - Install the Python library in edit mode. Allows Python files to be modified on the fly without having to re-install the Python library + + .. note:: + Installing smash in debug mode will reduce performance in terms of calculation time but can be extremely practical to avoid writing + ``WRITE(*,*)`` and ``READ(*,*)`` all over the Fortran files to debug. + +- ``dependencies`` + Generate the makefile.dep file. To avoid having to write the Fortran and C files to be compiled in the correct order of dependencies + each time a new file is added or the module dependencies change, this command will automatically read the files, search for module dependencies + and sort them. The makefile.dep is then used in the ``all`` and ``dbg`` commands. + +- ``library-edit`` + Install the Python library in edit-mode. This command is used by default when smash is installed in debug mode (``dbg`` command). + However, when developing only Python files, it may be useful to compile Fortran and C files in release mode but to be able to modify the Python files + on the fly. In this case, you can run the ``all`` command and then ``library-edit``. + +- ``clean`` + Clean the workspace (i.e. deletes everything that was generated to install smash). + +- ``tap`` + Generate adjoint file (``forward_db.f90``). As soon as a Fortran file marked as "to be differentiated" + (i.e. Fortran file starting with ``md`` or ``mwd``) is modified, the adjoint file must be updated and compiled again. + This command acts as an intermediary with a second makefile (in the ``tapenade`` directory) specific to the generation of + the adjoint file by `Tapenade `__. + +- ``tap-cmp`` + Compare the adjoint file inside the sources and the new adjoint file generated by the ``tap`` command. + We do not accept that a contribution be made with an adjoint file that is inconsistent with the sources. + This is why, in the continuous integration service (``CI``), one test is to check that the ``tap-cmp`` command does not return an error. + +- ``doc`` + Generate the documentation. This command acts as an intermediary with a second makefile (in the ``doc`` directory) specific to the generation of + the documenation by `Sphinx `__. + +- ``doc-clean`` + Clean the documentation workspace (i.e. deletes everything that was generated to produce the smash documentation). + +- ``test`` + Run unit tests. This tests are also run in the continuous integration service (``CI``). + +- ``test-coverage`` + Run unit tests with coverage. It will display coverage result in the terminal and generate a html file. + + .. note:: + The html file can be viewed with yout browser + + .. code-block:: none + + firefox smash/tests/htmlcov/index.html + +- ``test-baseline`` + Generate the tests baseline. Some tests require a baseline that allows you to compare current output with expected output. + If any tests have been intentionally modified or added to accommodate new development, the testing baseline must be regenerated + to reflect the changes made. This command will generate a csv file (``diff_baseline.csv``) that displays the changes made to the baseline + and an hdf5 file storing the new baseline (``new_baseline.hdf5``). Once you have ensured that all tests are successfully passed, + rename the ``new_baseline.hdf5`` file to ``baseline.hdf5`` and remove the previous version. + +Fortran guideline +----------------- + +Global convention +***************** + +The aim of this section is to show how to integrate new functions into the Fortran code of smash. + +Style convention +'''''''''''''''' + +Here are the conventions that have been applied on the content of a Fortran file (most of the time ...): + +- Use lowercase for all Fortran constructs (``do``, ``subroutine``, ``module``, ...) + +- For other names use all lowercase and ``snake_case`` as multiple-word identifier format (``optimize``, ``get_parameters``, ``set_states``, ...). + +- Use 4 spaces indentation. + +.. note:: + `fprettify `__ is used to format Fortran file. It can be used as follows: + + .. code-block:: none + + fprettify --indent 4 mwd_parameters.f90 + fprettify --indent 4 *.f90 + +.. _contributor_guide.development_process_details.fortran_guideline.global_convention.file_name_convention: + +File name convention +'''''''''''''''''''' + +If you want to integrate a new Fortran file, a naming convention must be respected in order to make the different automatic installation +processes understand if the file is a module and if it must be wrapped and/or differentiated. + +The structure of a Fortran file name can be written as follows: ``_.f90`` using lowercase and ``snake_case`` +as multiple-word identifier format. + +There are no constraints on ```` here are those on the ````: + +- ``m``: the file is a module (``m_array_creation.f90``) + +- ``mw``: the file is a module and is wrapped (``mw_optimize.f90``) + +- ``md``: the file is a module and is differentiated (``md_constant.f90``) + +- ``mwd``: the file is a module, is wrapped and differentiated (``mwd_setup.f90``) + +.. note:: + We strongly recommand the use of module. Specifically if the file contains sources to be wrapped or differentiated. + +.. _contributor_guide.development_process_details.fortran_guideline.global_convention.floating_point_convention: + +Floating point convention +''''''''''''''''''''''''' + +Most of the real variables are single precision floating-point. In some functions, these variables are casted into double precision floating-point. +Therefore, two constants ``sp`` and ``dp`` are used to precise the floating-point precision, respectively, simple precision and double precision. + +.. code-block:: fortran + + real(sp) :: foo = 2._sp + real(dp) :: bar = 0._dp + + bar = real(foo, dp) + +Compile +******* + +Compile a pre-existing file +''''''''''''''''''''''''''' + +If you are editing a pre-existing file, there are no particular constraints before compiling the code. +Compile with the following command: + +.. code-block:: none + + make + +.. _contributor_guide.development_process_details.fortran_guideline.compile.compile_a_new_file: + +Compile a new file +'''''''''''''''''' + +If you are creating a new file, respecting the naming convention (:ref:`contributor_guide.development_process_details.fortran_guideline.global_convention.file_name_convention`), +you must update the dependencies between modules before compiling. +Update the dependencies and compile with the following command: + +.. code-block:: none + + make dependencies all + +Wrapping +******** + +The Fortran code is wrapped using the `f90wrap `__ library. Here are the different steps to wrap `smash` +code efficiently. We assume here that we are integrating a wrapped module from scratch. Certain steps can be repeated if you are adding to +pre-existing files. + +.. hint:: + Quite a few examples are also available in the f90wrap GitHub directory in the examples folder + (see `here `__ + +.. _contributor_guide.development_process_details.fortran_guideline.wrapping.vector2_case: + +Vector2 case +'''''''''''' + +We are going to create a derived type called ``Vector2DT`` containing two real variables, ``x`` and ``y``, and a set of subroutines/functions +associated with this derived type. + +Create new wrapped files +"""""""""""""""""""""""" + +As explained in the :ref:`contributor_guide.development_process_details.fortran_guideline.global_convention.file_name_convention` section a +Fortran file will be automatically wrapped if it name contains the prefix ``mw`` or ``mwd``. We will consider the following +Fortran files: ``mw_vector2.f90`` and ``mw_vector2_manipulation.f90``. The first file will contain the implementation of the derived type +``Vector2DT`` and the second will contain all the subroutines/functions that manipulate the derived type. It might well have been possible to +do everything in a single file, but it was decided in `smash` to separate them. + +- ``mw_vector2.f90`` (this file can be stored in the folder ``smash/fcore/derived_type``) + +.. code-block:: fortran + + module mw_vector2 + ... + end module mw_vector2 + +- ``mw_vector2_manipulation.f90`` (this file can be stored in the folder ``smash/fcore/routine``) + +.. code-block:: fortran + + module mw_vector2_manipulation + ... + end module mw_vector2_manipulation + +.. note:: + The entire file will be wrapped, so it is advisable to separate the functions to be wrapped from those that are not. + +The files (even empty ones) can be compiled and wrapped (see the :ref:`contributor_guide.development_process_details.fortran_guideline.compile.compile_a_new_file` section) +and imported in Python as follows: + +.. code-block:: python + + >>> import smash.fcore._mw_vector2 + >>> import smash.fcore._mw_vector2_manipulation + +Derived type implementation +""""""""""""""""""""""""""" + +First, we will implement the derived type ``Vector2DT`` in the ``mw_vector2.f90`` file. + +.. note:: + We add the suffix ``DT`` for each derived type because Fortran is case insensitive and will not differentiate between ``vector2`` + and ``Vector2``. + +.. code-block:: fortran + + module mw_vector2 + + use md_constant, only: sp + + implicit none + + type Vector2DT + + real(sp) :: x + real(sp) :: y + + end type Vector2DT + + end module mw_vector2 + +.. note:: + ``sp`` is equal to ``4``, it is simple precision + (see the :ref:`contributor_guide.development_process_details.fortran_guideline.global_convention.floating_point_convention` section) + +A wrapped derived type is interpreted as a Python class. Let's compile, initialize it and view what it contains: + +.. code-block:: + + >>> from smash.fcore._mw_vector2 import Vector2DT + >>> v = Vector2DT() + >>> v + Vector2DT + x: 4.201793856028541e+18 + y: 3.0741685710357837e-41 + >>> v.x + 4.201793856028541e+18 + >>> v.y + 3.0741685710357837e-41 + +We can see that the 2 variables, ``x`` and ``y`` present in the original derived type are accessible in Python as class properties but filled with garbage values because they were not +initialized. There two ways to initialize the values of a derived type: + +- Assign values in the declaration of the derived type variables + +.. code-block:: fortran + + module mw_vector2 + + use md_constant, only: sp + + implicit none + + type Vector2DT + + real(sp) :: x = 0._sp + real(sp) :: y = 0._sp + + end type Vector2DT + + end module mw_vector2 + +- Create a specific initialization subroutine which will be interpreted as a Python class constructor (``__init__`` function). + `f90wrap `__ will automatically detects derived type initialization subroutine + if the subroutine name follows the convention: ``_initialise``. In our case, the subroutine must be called: + ``Vector2DT_initialise``. Let's write the initialization subroutine after adding the ``contains`` statement. + +.. code-block:: fortran + + module mw_vector2 + + use md_constant, only: sp + + implicit none + + type Vector2DT + + real(sp) :: x + real(sp) :: y + + end type Vector2DT + + contains + + subroutine Vector2DT_initialise(this) + + implicit none + + type(Vector2DT), intent(inout) :: this + + this%x = 0._sp + this%y = 0._sp + + end subroutine Vector2DT_initialise + + end module mw_vector2 + +The two methods in this example are equivalent and here is the result in Python: + +.. code-block:: python + + >>> v = Vector2DT() + >>> v + Vector2DT + x: 0.0 + y: 0.0 + +We successfully initialize the derived type with default values. However, the second method, using an initialization function, is more flexible. +We can, for example, not define default values but initialize the derived type with values from Python. Let's rewrite the initialize subroutine and +add arguments. + +.. code-block:: fortran + + module mw_vector2 + + use md_constant, only: sp + + implicit none + + type Vector2DT + + real(sp) :: x + real(sp) :: y + + end type Vector2DT + + contains + + subroutine Vector2DT_initialise(this, x, y) + + implicit none + + type(FooDT), intent(inout) :: this + real(sp), intent(in) :: x + real(sp), intent(in) :: y + + this%x = x + this%y = y + + end subroutine Vector2DT_initialise + + end module mw_vector2 + +We add 2 arguments which correspond to each variable of the derived type to initialize. On the Python side, this is how it translates: + +.. code-block:: python + + >>> v = Vector2DT(0, 0) + >>> v + Vector2DT + x: 0.0 + y: 0.0 + >>> v = Vector2DT(1, 1) + >>> v + Vector2DT + x: 1.0 + y: 1.0 + +It is also possible to modify the values once initialization is complete, since each element of the derived type is a property with a getter and a setter. + +.. code-block:: python + + >>> v = Vector2DT(0, 0) + >>> v + Vector2DT + x: 0.0 + y: 0.0 + >>> v.x = 3 + >>> v.y = 2 + v + Vector2DT + x: 3.0 + y: 2.0 + +Functions implementation +"""""""""""""""""""""""" + +We can now implement a number of subroutines/functions in the ``mw_vector2_manipulation.f90`` file to manipulate this derived type. We need first to import the module where +the ``Vector2DT`` derived type is defined ``mw_vector2`` and them in a ``contains`` statement add the functions. + +.. code-block:: fortran + + module mw_vector2_manipulation + + use md_constant, only: sp + use mw_vector2, only: Vector2DT + + implicit none + + contains + + function vector2_add_value(v, add) result(res) + + type(Vector2DT), intent(in) :: v + real(sp), intent(in) :: add + + type(Vector2DT) :: res + + res%x = v%x + add + res%y = v%y + add + + end function vector2_add_value + + function vector2_dot_product(v1, v2) result(res) + + type(Vector2DT), intent(in) :: v1 + type(Vector2DT), intent(in) :: v2 + + real(sp) :: res + + res = v1%x*v2%x + v1%y*v2%y + + end function vector2_dot_product + + end module mw_vector2_manipulation + +We have added two functions, one to add a value to each element of the ``Vector2DT`` and the other one to compute +the dot product between two ``Vector2DT`` , so let's see how this translates into Python: + +.. code-block:: python + + >>> from smash.fcore._mw_vector2 import Vector2DT + >>> from smash.fcore._mw_vector2_manipulation import vector2_add_value, vector2_dot_product + + >>> v = Vector2DT(0, 0) + >>> vector2_add_value(v, 5) + Vector2DT + x: 5.0 + y: 5.0 + + >>> v1 = Vector2DT(1, 1) + >>> v2 = Vector2DT(2, 3) + >>> vector2_dot_product(v1, v2) + 5.0 + +This completes the first example of Fortran wrapping in `smash`. The next examples will be less detailed but will aim to expose a wider range of functionality, +variable types, allocation management, string management, etc. + +Matrix2 case +'''''''''''' + +We are going to create a derived type called ``Matrix2DT`` containing one allocatable real variable of 2 dimensions, ``vle``, two integer variables +representing the number of rows and columns of the matrix, ``n`` and ``m``, respectively and a set of subroutines/functions associated with +this derived type. Similar to the :ref:`contributor_guide.development_process_details.fortran_guideline.wrapping.vector2_case` section, two files +are created, ``mw_matrix2.f90`` and ``mw_matrix2_manipulation.f90``. The aim of this case is to illustrate how arrays can are handled. + +- ``mw_matrix2.f90`` (this file can be stored in the folder ``smash/fcore/derived_type``) + +.. code-block:: fortran + + module mw_matrix2 + + use md_constant, only: sp + + implicit none + + type Matrix2DT + + integer :: n + integer :: m + real(sp), dimension(:, :), allocatable :: vle + + end type Matrix2DT + + contains + + subroutine Matrix2DT_initialise(this, n, m, vle0) + + implicit none + + type(Matrix2DT), intent(inout) :: this + integer, intent(in) :: n, m + real(sp), intent(in) :: vle0 + + this%n = n + this%m = m + allocate (this%vle(this%n, this%m)) + this%vle(:, :) = vle0 + + end subroutine Matrix2DT_initialise + + end module mw_matrix2 + +- ``mw_matrix2_manipulation.f90`` (this file can be stored in the folder ``smash/fcore/routine``) + +.. code-block:: fortran + + module mw_matrix2_manipulation + + use md_constant, only: sp + use mw_matrix2, only: Matrix2DT, Matrix2DT_initialise + + implicit none + + contains + + function matrix2_add_value(mat, add) result(res) + + implicit none + + type(Matrix2DT), intent(inout) :: mat + real(sp), intent(in) :: add + + type(Matrix2DT) :: res + + call Matrix2DT_initialise(res, mat%n, mat%m, 0._sp) + + res%vle(:, :) = mat%vle(:, :) + add + + end function matrix2_add_value + + function matrix2_transpose(mat) result(res) + + implicit none + + type(Matrix2DT), intent(in) :: mat + + type(Matrix2DT) :: res + integer :: i, j + + call Matrix2DT_initialise(res, mat%m, mat%n, 0._sp) + + ! Could also use directly the Fortran intrinsic function TRANSPOSE + ! res%vle = TRANSPOSE(mat%vle) + do i = 1, mat%m + do j = 1, mat%n + res%vle(i, j) = mat%vle(j, i) + end do + end do + + end function matrix2_transpose + + end module mw_matrix2_manipulation + +This translates into Python: + +.. code-block:: python + + >>> from smash.fcore._mw_matrix2 import Matrix2DT + + >>> mat = Matrix2DT(2, 3, 0) + >>> mat + Matrix2DT + m: 3 + n: 2 + vle: array([[0., 0., 0.], + [0., 0., 0.]], dtype=float32) + >>> type(mat.vle) + + +Fortran arrays are casted to `numpy.ndarray` when accessed in Python. So all the methods associated with a `numpy.ndarray` can be used. + +.. code-block:: python + + >>> from smash.fcore._mw_matrix2 import Matrix2DT + >>> from smash.fcore._mw_matrix2_manipulation import matrix2_add_value, matrix2_transpose + + >>> mat = Matrix2DT(2, 3, 0) + >>> mat + Matrix2DT + m: 3 + n: 2 + vle: array([[0., 0., 0.], + [0., 0., 0.]], dtype=float32) + + >>> mat.vle.shape + (2, 3) + >>> mat.vle.dtype + dtype('float32' + + >>> mat.vle[0, :] = 2 + >>> mat + Matrix2DT + m: 3 + n: 2 + vle: array([[2., 2., 2.], + [0., 0., 0.]], dtype=float32) + + >>> matrix2_add_value(mat, 4) + Matrix2DT + m: 3 + n: 2 + vle: array([[6., 6., 6.], + [4., 4., 4.]], dtype=float32) + + >>> matrix2_transpose(mat) + Matrix2DT + m: 2 + n: 3 + vle: array([[0., 2.], + [0., 2.], + [0., 2.]], dtype=float32) + +Matrix2Array case +''''''''''''''''' + +We are going to create a derived type called ``Matrix2ArrayDT`` containing one allocatable ``Matrix2DT`` type variable of 1 dimension. +The aim of this case is to illustrate how derived type arrays are handled. We will keep the previous files created for ``Matrix2DT`` +(i.e. ``mw_matrix2.f90`` and ``mw_matrix2_manipulation.f90``). + +.. code-block:: fortran + + module mw_matrix2 + + use md_constant, only: sp + + implicit none + + type Matrix2DT + + integer :: n + integer :: m + real(sp), dimension(:, :), allocatable :: vle + + end type Matrix2DT + + type Matrix2ArrayDT + + integer :: n + type(Matrix2DT), dimension(:), allocatable :: mat + + end type Matrix2ArrayDT + + contains + + subroutine Matrix2DT_initialise(this, n, m, vle0) + + implicit none + + type(Matrix2DT), intent(inout) :: this + integer, intent(in) :: n, m + real(sp), intent(in) :: vle0 + + this%n = n + this%m = m + allocate (this%vle(this%n, this%m)) + this%vle(:, :) = vle0 + + end subroutine Matrix2DT_initialise + + subroutine Matrix2ArrayDT_initialise(this, n, n_arr, m_arr, vle0_arr) + + implicit none + + type(Matrix2ArrayDT), intent(inout) :: this + integer, intent(in) :: n + integer, dimension(n), intent(in) :: n_arr, m_arr + real(sp), dimension(n), intent(in) :: vle0_arr + + integer :: i + + this%n = n + + allocate (this%mat(this%n)) + + do i = 1, this%n + + call Matrix2DT_initialise(this%mat(i), n_arr(i), m_arr(i), vle0_arr(i)) + + end do + + end subroutine Matrix2ArrayDT_initialise + + end module mw_matrix2 + +Here we create a derived type ``Matrix2ArrayDT`` which contains an array of ``Matrix2DT``. To initialize this derived type, we pass the number of +``Matrix2DT`` that we want to allocate ``n``, the number of rows and columns for each allocated matrix ``n_arr`` and ``m_arr``, respectively and +an initial value for each ``vle0_arr``. This translates into Python: + +.. code-block:: python + + >>> import numpy as np + >>> from smash.fcore._mw_matrix2 import Matrix2ArrayDT + + >>> n = 2 + >>> n_arr = np.array([2, 3], dtype=np.int32) + >>> m_arr = np.array([4, 1], dtype=np.int32) + >>> vle_arr = np.array([1, 5], dtype=np.float32) + + >>> mat_arr = Matrix2ArrayDT(n, n_arr, m_arr, vle_arr) + >>> mat_arr + Matrix2ArrayDT + mat: ['Matrix2DT', 'Matrix2DT'] + n: 2 + +It allows us to create an array of ``Matrix2DT`` that can have different shapes. Here ``(2, 4)`` and ``(3, 1)``. We can iterate over as follows: + +.. code-block:: python + + >>> import numpy as np + >>> from smash.fcore._mw_matrix2 import Matrix2ArrayDT + + >>> n = 2 + >>> n_arr = np.array([2, 3], dtype=np.int32) + >>> m_arr = np.array([4, 1], dtype=np.int32) + >>> vle_arr = np.array([1, 5], dtype=np.float32) + + >>> mat_arr = Matrix2ArrayDT(n, n_arr, m_arr, vle_arr) + >>> mat_arr + Matrix2ArrayDT + mat: ['Matrix2DT', 'Matrix2DT'] + n: 2 + + >>> for m in mat_arr.mat.items(): + >>> m + Matrix2DT + m: 4 + n: 2 + vle: array([[1., 1., 1., 1.], + [1., 1., 1., 1.]], dtype=float32) + Matrix2DT + m: 1 + n: 3 + vle: array([[5.], + [5.], + [5.]], dtype=float32) + + >>> mat_arr.mat[0] + Matrix2DT + m: 4 + n: 2 + vle: array([[1., 1., 1., 1.], + [1., 1., 1., 1.]], dtype=float32) + + >>> mat_arr.mat[1] + Matrix2DT + m: 1 + n: 3 + vle: array([[5.], + [5.], + [5.]], dtype=float32) + +Character/String case +''''''''''''''''''''' + +We are going to create a derived type called ``CharacterDT`` containing a character ``c`` and character array ``c_arr`` in order to get +into the details of this specific edge case of the wrapping and how we handle it in `smash`. +Let's create a ``mw_character.f90`` file. + +.. code-block:: fortran + + module mw_character + + use md_constant, only: sp, lchar + + implicit none + + type CharacterDT + + character(lchar) :: c = "foo" + character(lchar), dimension(2) :: c_arr = "bar" + + end type CharacterDT + + end module mw_character + +This translates into Python: + +.. code-block:: python + + >>> from smash.fcore._mw_character import CharacterDT + >>> char = CharacterDT() + >>> char + CharacterDT + c: b'foo' + c_arr: array([[ 98, 98], + [ 97, 97], + [114, 114], + ... + [ 32, 32], + [ 32, 32]], dtype=uint8) + + >>> type(char.c) + + >>> type(char.c_arr), char.c_arr.dtype + , dtype('uint8') + +As you can see, when wrapped to Python, a Fortran character is interpreted as ``bytes`` and character array as a `numpy.ndarray` of dtype ``uint8`` +(unsigned 8 bits integer). To get something interpretable, we can cast ``bytes`` to ``str`` with the ``decode`` method and decode each ``ASCII`` +value in the character array. + +.. code-block:: python + + >>> from smash.fcore._mw_character import CharacterDT + >>> char = CharacterDT() + >>> char + CharacterDT + c: b'foo' + c_arr: array([[ 98, 98], + [ 97, 97], + [114, 114], + ... + [ 32, 32], + [ 32, 32]], dtype=uint8) + + >>> char.c.decode() + 'foo' + + # Cast to bytes + >>> char.c_arr.tobytes(order="F") + b'bar + bar + ' + # Decode with utf-8 encoding + >>> char.c_arr.tobytes(order="F").decode() + 'bar + bar + ' + # Split by whitespaces + >>> char.c_arr.tobytes(order="F").decode().split() + ['bar', 'bar'] + +We have managed to interpret these values, but it's not particularly conveniente. Moreover, how can we change the values in Python ? + +.. code-block:: python + + >>> from smash.fcore._mw_character import CharacterDT + >>> char = CharacterDT() + >>> char + CharacterDT + c: b'foo' + c_arr: array([[ 98, 98], + [ 97, 97], + [114, 114], + ... + [ 32, 32], + [ 32, 32]], dtype=uint8) + + >>> char.c = "baz" + >>> char.c + "baz" + >>> char.c_arr = ["buz", "buz"] + ValueError: invalid literal for int() with base 10: 'buz' + >>> char.c_arr = np.array(["buz", "buz"]) + ValueError: invalid literal for int() with base 10: 'buz' + +It's ok for a character but not for the character array. To get around this, some self made Fortran directives can be inserted at the definition +of the variables in the derived type. ``!$F90W char`` for character and ``!$F90W char-array`` for character array. + +.. code-block:: fortran + + module mw_character + + use md_constant, only: sp, lchar + + implicit none + + type CharacterDT + + character(lchar) :: c = "foo" !$F90W char + character(lchar), dimension(2) :: c_arr = "bar" !$F90W char-array + + end type CharacterDT + + end module mw_character + +This allows us to manipulate this derived type in Python in a more practical way: + +.. code-block:: python + + >>> import numpy as np + >>> from smash.fcore._mw_character import CharacterDT + >>> char = CharacterDT() + >>> char + CharacterDT + c: 'foo' + c_arr: array(['bar', 'bar'], dtype='>> char.c = "baz" + >>> char.c_arr = np.array(["buz", "buz"]) + >>> char + CharacterDT + c: 'baz' + c_arr: array(['buz', 'buz'], dtype='>> import numpy as np + >>> from smash.fcore._mw_array_index import ArrayIndexDT + >>> from smash.fcore._mw_array_index_manipulation import array_index_at_ind + + >>> ai = ArrayIndexDT() + >>> ai + ArrayIndexDT + ind: 1 + r_arr: array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], dtype=float32) + + >>> ai.r_arr = np.arange(0, ai.r_arr.size) + >>> ai + ArrayIndexDT + ind: 1 + r_arr: array([0., 1., 2., 3., 4., 5., 6., 7., 8., 9.], dtype=float32) + +Now we can for exemple store the indice of the maximum value of the array in ``ai.ind`` and try to access the maximum value back with this indice +from Python and Fortran + +.. code-block:: python + + >>> ai.ind = np.argmax(ai.r_arr) + >>> ai + ArrayIndexDT + ind: 9 + r_arr: array([0., 1., 2., 3., 4., 5., 6., 7., 8., 9.], dtype=float32) + + # Access from Python + >>> ai.r_arr[ai.ind] + 9.0 + + # Access from Fortran + >>> array_index_at_ind(ai) + 8.0 + +As you can see, the value are different and it's because the arrays are not indexed in the same way. The value of ``ai.ind`` is set to ``9`` which +is correct in Python but should be ``10`` in Fortran. As a result, we'd need to manipulate the index depending on whether we calculated it in +Fortran or Python, which isn't practical and is prone to out-of-range accesses. To get around this, a self made Fortran directive ``!$F90W index`` +can be used to substract 1 from the value of a variable storing indices when passing from Fortran to Python or add 1 the other way around. +Let's do the same thing with the new directive. + +.. note:: + + If the variable storing the indices is an array, the directive is ``!$F90W index-array`` instead of ``!$F90W index``. + +.. code-block:: fortran + + module mw_array_index + + use md_constant, only: sp + + implicit none + + type ArrayIndexDT + + integer :: ind = 1 !$F90W index + real(sp), dimension(10) :: r_arr = 0._sp + + end type ArrayIndexDT + + end module mw_array_index + +.. code-block:: python + + >>> import numpy as np + >>> from smash.fcore._mw_array_index import ArrayIndexDT + >>> from smash.fcore._mw_array_index_manipulation import array_index_at_ind + + >>> ai = ArrayIndexDT() + >>> ai + ArrayIndexDT + ind: 0 + r_arr: array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], dtype=float32) + + >>> ai.r_arr = np.arange(0, ai.r_arr.size) + >>> ai + ArrayIndexDT + ind: 0 + r_arr: array([0., 1., 2., 3., 4., 5., 6., 7., 8., 9.], dtype=float32) + + >>> ai.ind = np.argmax(ai.r_arr) + >>> ai + ArrayIndexDT + ind: 9 + r_arr: array([0., 1., 2., 3., 4., 5., 6., 7., 8., 9.], dtype=float32) + + # Access from Python + >>> ai.r_arr[ai.ind] + 9.0 + + # Access from Fortran + >>> array_index_at_ind(ai) + 9.0 + +Python guideline +---------------- + +Test +---- + +Documentation +------------- \ No newline at end of file diff --git a/doc/source/contributor_guide/development_process_summary.rst b/doc/source/contributor_guide/development_process_summary.rst new file mode 100644 index 00000000..66284ac0 --- /dev/null +++ b/doc/source/contributor_guide/development_process_summary.rst @@ -0,0 +1,109 @@ +.. _contributor_guide.development_process_summary: + +=========================== +Development Process Summary +=========================== + +Fork +---- + +Go to https://github.com/DassHydro/smash/fork and create your own ``Fork`` of the project. Uncheck the box "Copy the main branch only". + +Clone +----- + +Clone the Git repository: + +.. code-block:: none + + git clone https://github.com/username/smash.git + +If you prefer working on the Git repository without authentication required, you can create a personal access token by following these +`detailed instructions `__. +Once you have generated a token, clone the Git repository using the following command: + +.. code-block:: none + + git clone https://username:token@github.com/username/smash.git + +Replace ``username`` with your GitHub username and ``token`` with the personal access token you generated. + +Install developer environment +----------------------------- + +Go to the smash repository: + +.. code-block:: none + + cd smash + +It is recommended that you install the smash development environment using `Anaconda `__: + +.. code-block:: none + + conda env create -f environment-dev.yml + +Activate the conda environment: + +.. code-block:: none + + conda activate smash-dev + +Develop contribution +-------------------- + +Switch to the latest development version of smash (``maintenance/x.y.z``): + +.. code-block:: none + + (smash-dev) git checkout maintenance/x.y.z + +Create a new branch on which you will implement your development: + +.. code-block:: none + + (smash-dev) git checkout -b your-development-branch + +Submit +------ + +Before submitting your changes, make sure that the tests pass (although they will also be executed automatically once the submission has been made): + +.. code-block:: none + + (smash-dev) make test + +If changes have also been made to the documentation, also run the command to check the compilation of the documentation: + +.. code-block:: none + + (smash-dev) make doc + +If all the tests pass, commit your changes: + +.. code-block:: none + + (smash-dev) git add . + (smash-dev) git commit + +Write a clear message to help reviewers understand what has been done in this contribution and finally, push your changes back to your fork: + +.. code-block:: none + + (smash-dev) git push --set-upstream origin your-development-branch + +You will be asked for your username and password (unless you have generated a personal token access). Then, go to GitHub. +The new branch will show up with a green ``Pull Request`` button. Make sure the title and message are clear. Then click the button to submit it. + +Review process +-------------- + +Reviewers (the other developers and interested community members) will write inline and/or general comments on your Pull Request (``PR``) to help you improve its implementation, +documentation and style. + +To update your ``PR``, make your changes on your local repository, commit, run tests, and only if they succeed push to your fork. +As soon as those changes are pushed up (to the same branch as before) the ``PR`` will update automatically. +If you have no idea how to fix the test failures, you may push your changes anyway and ask for help in a ``PR`` comment. + +Various continuous integration (``CI``) services are triggered after each ``PR`` update to build the code, run unit tests, compare adjoint code and check documentation compilation. +The ``CI`` tests must pass before your ``PR`` can be merged. If ``CI`` fails, you can find out why by clicking on the “failed” icon (red cross) and inspecting the build and test log. diff --git a/doc/source/contributor_guide/index.rst b/doc/source/contributor_guide/index.rst new file mode 100644 index 00000000..f05d6174 --- /dev/null +++ b/doc/source/contributor_guide/index.rst @@ -0,0 +1,15 @@ +.. _contributor_guide: + + +.. based on https://numpy.org/doc/stable/dev/index.html and https://pandas.pydata.org/docs/development/index.html#development + +================= +Contributor Guide +================= + +.. toctree:: + :maxdepth: 2 + + bug_report_and_enhancement_requests + development_process_summary + development_process_details \ No newline at end of file diff --git a/doc/source/contributors/index.rst b/doc/source/contributors/index.rst deleted file mode 100644 index a492a408..00000000 --- a/doc/source/contributors/index.rst +++ /dev/null @@ -1,27 +0,0 @@ -.. _contributors: - -============ -Contributors -============ - -**Contact:** *pierre-andre.garambois@inrae.fr* - -- François Colleoni -- Ngo Nghi Truyen Huynh -- Pierre-André Garambois -- Maxime Jay-Allemand -- Lilian Villenave -- Mouad Ettalbi -- Benjamin Renard -- Didier Organde -- Igor Gejadze -- Pierre Javelle -- Patrick Arnaud -- Julie Demargne -- Catherine Fouchier -- Nathalie Folton -- Killian Pujol -- Juliette Godet -- Sarah Vigoureux -- Abubakar Haruna -- Matthias Paluszkiewicz \ No newline at end of file diff --git a/doc/source/developers_guide/index.rst b/doc/source/developers_guide/index.rst deleted file mode 100644 index 3602802e..00000000 --- a/doc/source/developers_guide/index.rst +++ /dev/null @@ -1,220 +0,0 @@ -.. _developers_guide: - -================ -Developers Guide -================ - -Are you a new `smash` contributor? Here are the instructions to complete your development for smash. - -Installation -************ - -Clone the Git repository: - -.. code-block:: none - - git clone https://github.com/DassHydro/smash.git - -If you prefer working on the Git repository without authentication required, you can create a personal access token by following these `detailed instructions `__. -Once you have generated a token, clone the Git repository using the following command: - -.. code-block:: none - - git clone https://username:token@github.com/DassHydro/smash.git - -Replace ``username`` with your GitHub username and ``token`` with the personal access token you generated. - -Go to the smash repository: - -.. code-block:: none - - cd smash - -It is recommended that you install the smash development enviroment using `Anaconda `__: - -.. code-block:: none - - conda env create -f environment-dev.yml - -Activate the conda environment: - -.. code-block:: none - - conda activate smash-dev - -Compile the code and install the Python library: - -.. code-block:: none - - (smash-dev) make - -Make a unit test to verify the installation: - -.. code-block:: none - - (smash-dev) make test - -Coding workflow -*************** - -Once you have completed the installation, you can now start your development for smash. - -Switch to the latest development version ----------------------------------------- - -Switch to the latest development version of smash (``maintenance/x.y.z``): - -.. code-block:: none - - (smash-dev) git checkout maintenance/x.y.z - -Create a new branch on which you will implement your development: - -.. code-block:: none - - (smash-dev) git checkout -b your-development-branch - -Fully recompile the code with the latest development version: - -.. code-block:: none - - (smash-dev) make clean - (smash-dev) make - -Fortran coding style --------------------- - -Some convention on the content of Fortran files: - -- Write in lower case (even if the code is case insensitive, just for consistency). -- Use 4 spaces indentation. -- Use snake_case as multiple-word identifier format (i.e. **subroutine get_foo()**). -- Strongly prefer the use of module. This allows to use the ``use`` statement and to trace subroutine calls within the files (i.e. **use m_foo, only: get_foo**) -- Do not create global variables which are not well handled by the wrapper. -- Use single precision floating-point ``sp`` variable in ``md_constant.f90`` (i.e. **real(sp) :: foo**) - -If you want to integrate a new Fortran file, a naming convention must be respected in order to make the different automatic installation processes understand -if the file is a module and if it must be wrapped and/or differentiated. - -The structure of a Fortran file name can be written as follows: ``_.f90``. - -There are no constraints on ```` here are those on the ````: - -- ``m``: the file is a module (i.e. ``m_array_creation.f90``) -- ``mw``: the file is a module and is wrapped (i.e. ``mw_optimize.f90``) -- ``md``: the file is a module and is differentiated (i.e. ``md_constant.f90``) -- ``mwd``: the file is a module, is wrapped and differentiated (i.e. ``mwd_setup.f90``) - -Other development commands --------------------------- - -Here are some other commands that may be necessary for your development: - -- Compile the adjoint and tangent linear model. This step is required if you are developing on the Fortran interface, particularly for the development on differentiated routines. - -.. code-block:: none - - (smash-dev) make tap - -- Compile the code in debug mode. This allows you to compile the Fortran interface with warning flags (i.e. ``-Wall``, ``-Wextra``, etc.). - -.. code-block:: none - - (smash-dev) make debug - -- If your changes are only made on the Python interface, you don't need to recompile all the code each time, but only the Python interface: - -.. code-block:: none - - (smash-dev) make library - -.. note:: - - Whenever a modification is made to any Python file, it is necessary to recompile the Python interface by executing the ``make library`` command. - To avoid having to perform this step manually each time a change is made, the code can be automatically updated using the following command: - - .. code-block:: none - - (smash-dev) make library-edit - - -Peer review preparation -*********************** - -Upon completion of your coding development, it is imperative to ensure that all existing tests have been passed: - -.. code-block:: none - - (smash-dev) make test - -Unless any tests have been intentionally modified or added to accommodate new development. -In such cases, the testing baseline must be regenerated to reflect the changes made: - -.. code-block:: none - - (smash-dev) make test-baseline - -It is advisable to verify all changes on the baseline by referencing the ``diff_baseline.csv`` file in the ``smash/tests/`` directory. -Once you have ensured that all tests are successfully passed, rename the ``smash/tests/new_baseline.hdf5`` file to ``smash/tests/baseline.hdf5`` and remove the previous version. - -We also recommend that you add your contribution to the release notes of the current development version and the documentation in the ``doc/source/`` directory. -If you intend to create new reStructuredText (``.rst``) files for the documentation in ``doc/source/``, we suggest using the following command to generate the ``.rst`` file with auto-defined label: - -.. code-block:: none - - (smash-dev) cd doc/source/ - (smash-dev) python gen_rst.py path-to-your-rst-file - -After returning to the Git repository, compile the documentation to apply your changes: - -.. code-block:: none - - (smash-dev) make doc - -The initial compilation may take a while, but subsequent compilations will only require the time it takes to compile the modified files. - -.. note:: - - If you encounter any issues when compiling the documentation, try cleaning the ``doc/`` directory and then recompiling the documentation. - This can help eliminate any potential conflicts and bugs that may be causing the issue. - - .. code-block:: none - - (smash-dev) make doc-clean - (smash-dev) make doc - -Make sure that the Git repository is cleaned and the Python files are formatted before submitting your work: - -.. code-block:: none - - (smash-dev) make clean - (smash-dev) make doc-clean - (smash-dev) cd smash - (smash-dev) black *.py - -Submission and review process -***************************** - -After completing the previous steps and committing your new branch, push it to Git using the following command: - -.. code-block:: none - - git push --set-upstream origin your-development-branch - -Next, create a new pull request to the current development branch ``maintenance/x.y.z``. -Ensure that your commits have passed the CI/CD pipelines; otherwise, you will be required to fix them before the review process begins. -So, it is highly recommended to run the pipelines locally before opening a pull request: - -.. code-block:: none - - (smash-dev) make clean - (smash-dev) make doc-clean - (smash-dev) make tap - (smash-dev) make - (smash-dev) make test - (smash-dev) make doc - -Reviewers, including other developers or relevant people, will be assigned to review your work. -At this stage, friendly discussions may occur to enhance your implementation and maintain consistency in coding style. -The duration of this procedure may vary depending on the nature of your contribution, such as introducing new features, addressing major bug fixes, or replacing functionalities. -Once your pull request is approved by at least one reviewer, your branch will be merged into the latest development version. diff --git a/doc/source/index.rst b/doc/source/index.rst index 19ddf281..1d591652 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -92,7 +92,6 @@ smash documentation api_reference/index math_num_documentation/index release/index - contributors/index - developers_guide/index + contributor_guide/index license/index bibliography/index