Write fluxes #199

asjeb · 2024-06-06T10:17:13Z

merge write_fluxes into main
writes fluxes :
["pn", "en", "pr", "perc", "lexc", "prr", "prd", "qr", "qd"], # % gr4
["pn", "en", "pr", "perc", "lexc", "prr", "prd", "qr", "qd"], # % gr5
["ei", "pn", "en", "pr", "perc", "prr", "qr"], # % grd
["ei", "pn", "en", "pr", "perc", "prr", "prd", "qr", "qd"], # % loieau
["pn", "en", "qr", "qb"], # % vic3l

writes states :
["hi", "hp", "ht"], # % gr4
["hi", "hp", "ht"], # % gr5
["hp", "ht"], # % grd
["ha", "hc"], # % loieau
["hcl", "husl", "hmsl", "hbsl"], # % vic3l

writes stats :
mean, var, minimum, maximum, median
when run_forward is worked with activated return_options

pag13

Nice job!
This is only comments from a first reading of the code, without testing in details on a case.

pag13 · 2024-06-06T14:34:09Z

smash/_constant.py

+            ["pn", "en", "pr", "perc", "lexc", "prr", "prd", "qr", "qd"],  # % gr5
+            ["ei", "pn", "en", "pr", "perc", "prr", "qr"],  # % grd
+            ["ei", "pn", "en", "pr", "perc", "prr", "prd", "qr", "qd"],  # % loieau
+            ["pn", "en", "qr", "qb"],  # % vic3l


for vic, add vertical drainage fluxes between each production layer (d_{umsl}, medium and bottom layers also..)

(not critical, this can be fixed later)

All drainage fluxes are on active cells...

pag13 · 2024-06-06T14:53:34Z

smash/core/simulation/run/run.py

    SIMULATION_RETURN_OPTIONS_TIME_STEP_KEYS,
+    STRUCTURE_RR_STATES,


if "STRUCTURE" dependence mentioned (which I agree) in this "keyword" for RR_STATES, then idem for internal fluxes? INTARNAL_FLUXES => STRUCTURE_INTERNAL_FLUXES ?

pag13 · 2024-06-06T15:26:59Z

smash/fcore/common/md_stats.f90

+        real(sp) :: m
+
+        fx = returns%stats%internal_fluxes(:, :, idx)
+        !$AD start-exclude


seen. needed here if call to the function is excluded? entire stats module has to be differentiated?

smash/fcore/derived_type/mwd_returns.f90

pag13 · 2024-06-07T06:52:17Z

smash/fcore/derived_type/mwd_setup.f90

@@ -137,6 +137,7 @@ module mwd_setup
        integer :: nrrs = -99
        integer :: nsep_mu = -99
        integer :: nsep_sigma = -99
+        integer :: nfx = -99


what is nfx? clarify in routine header (cf. nqz!)

pag13 · 2024-06-07T06:55:23Z

smash/fcore/forward/md_simulation.f90

@@ -366,7 +379,17 @@ subroutine simulation_checkpoint(setup, mesh, input_data, parameters, output, op
                rr_parameters_inc = rr_parameters_inc + 1

            end select
-
+            !$AD start-exclude


ok. to be investigated later.

pag13 · 2024-06-07T06:56:31Z

smash/fcore/operator/md_gr_operator.f90

@@ -202,6 +204,30 @@ subroutine gr4_time_step(setup, mesh, input_data, options, time_step, ac_mlt, ac
                ! Transform from mm/dt to m3/s
                ac_qt(k) = ac_qt(k)*1e-3_sp*mesh%dx(row, col)*mesh%dy(row, col)/setup%dt

+                !$AD start-exclude


ok. to be investigated later.

pag13 · 2024-06-07T07:00:39Z

smash/fcore/operator/md_gr_operator.f90

+                if (returns%pn_flag) returns%pn(row, col, time_step) = pn
+                if (returns%en_flag) returns%en(row, col, time_step) = en
+                if (returns%pr_flag) returns%pr(row, col, time_step) = pr
+                if (returns%perc_flag) returns%perc(row, col, time_step) = perc
+                if (returns%lexc_flag) returns%lexc(row, col, time_step) = l
+                if (returns%prr_flag) returns%prr(row, col, time_step) = prr
+                if (returns%prd_flag) returns%prd(row, col, time_step) = prd
+                if (returns%qr_flag) returns%qr(row, col, time_step) = qr
+                if (returns%qd_flag) returns%qd(row, col, time_step) = qd


ok for this first version. flux maps getter implemented?
Wouldn't this be better to use sthg like returns%internal_fluxes_maps(:,:,:,:)
with 1..N fluxes which names is defined for each model structure (as for params or for stats%internal_fluxes above) in _constants.py?

pag13 · 2024-06-07T07:02:02Z

smash/fcore/operator/md_vic3l_operator.f90

+                if (returns%stats_flag) then
+                    returns%stats%internal_fluxes(row, col, 1) = pn
+                    returns%stats%internal_fluxes(row, col, 2) = en
+                    returns%stats%internal_fluxes(row, col, 3) = qr
+                    returns%stats%internal_fluxes(row, col, 4) = qb
+                end if
+                if (returns%pn_flag) returns%pn(row, col, time_step) = pn
+                if (returns%en_flag) returns%en(row, col, time_step) = en
+                if (returns%qr_flag) returns%qr(row, col, time_step) = qr
+                if (returns%qb_flag) returns%qb(row, col, time_step) = qb
+                !$AD end-exclude


vertical fluxes between production store layers are missing

inoelloc · 2024-06-06T16:01:31Z

f90wrap/py_mod_names

A supprimer

inoelloc · 2024-06-06T16:01:41Z

makefile.dep

A supprimer

inoelloc · 2024-06-06T16:09:27Z

smash/_constant.py

Je trouve ca un peu complexe de demander à l'utilisateur de cocher chacun des flux un par un. Je regrouperais bien tous les flux dans une seule variable: internal_fluxes. L'utilisateur récupère tout mais pas grave c'est consistent avec les états.
Pour stats, je trouve pas ca hyper clair comme mot clef. Je mettrais plutot internal_fluxes_stats
On pourrait aussi récupérer ces valeurs en sortie d'une optimisation (c'est à dire, écrire les flux sur le dernier run forward de l'optimisation) ? Ca nous oblige à relancer un run direct en fin d'optimisation si on veut récupérer ces flux

inoelloc · 2024-06-06T16:12:21Z

smash/_constant.py

+INTERNAL_FLUXES = dict(
+    zip(
+        HYDROLOGICAL_MODULE,
+        [
+            ["pn", "en", "pr", "perc", "lexc", "prr", "prd", "qr", "qd"],  # % gr4
+            ["pn", "en", "pr", "perc", "lexc", "prr", "prd", "qr", "qd"],  # % gr5
+            ["ei", "pn", "en", "pr", "perc", "prr", "qr"],  # % grd
+            ["ei", "pn", "en", "pr", "perc", "prr", "prd", "qr", "qd"],  # % loieau
+            ["pn", "en", "qr", "qb"],  # % vic3l
+        ],
+    )
+)
+


La par contre, on doit déclarer le nom de chaque flux pas de pb. Cependant, on restreint les flux au module hydrologique. On pourrait avoir des flux internes dans le module neige et le module de routage. Il faudrait revoir cette constante pour quelle englobe les 3 modules et donc avoir des flux internes par structure, un peu comme les parameters et états. (voir get_structure_rr_parameters, etc)

inoelloc · 2024-06-06T16:13:35Z

smash/_constant.py

+        ],
+    )
+)
+
 SIMULATION_RETURN_OPTIONS_TIME_STEP_KEYS = ["rr_states", "q_domain"]


J'aimerais bien mettre les flux internes au moins les grilles dans les variables qui peuvent être récupérées qu'a certains pas de temps. Pour les statisques sur les flux internes on peut renvoyer toute la chronique dans l'absolu

inoelloc · 2024-06-06T17:04:14Z

smash/fcore/operator/md_gr_operator.f90

+                !$AD start-exclude
+                !internal fluxes
+                if (returns%stats_flag) then
+                    returns%stats%internal_fluxes(row, col, 1) = ei
+                    returns%stats%internal_fluxes(row, col, 2) = pn
+                    returns%stats%internal_fluxes(row, col, 3) = en
+                    returns%stats%internal_fluxes(row, col, 4) = pr
+                    returns%stats%internal_fluxes(row, col, 5) = perc
+                    returns%stats%internal_fluxes(row, col, 6) = prr
+                    returns%stats%internal_fluxes(row, col, 7) = prd
+                    returns%stats%internal_fluxes(row, col, 8) = qr
+                    returns%stats%internal_fluxes(row, col, 9) = qd
+                end if
+                if (returns%ei_flag) returns%ei(row, col, time_step) = ei
+                if (returns%pn_flag) returns%pn(row, col, time_step) = pn
+                if (returns%en_flag) returns%en(row, col, time_step) = en
+                if (returns%pr_flag) returns%pr(row, col, time_step) = pr
+                if (returns%perc_flag) returns%perc(row, col, time_step) = perc
+                if (returns%prr_flag) returns%prr(row, col, time_step) = prr
+                if (returns%prd_flag) returns%prd(row, col, time_step) = prd
+                if (returns%qr_flag) returns%qr(row, col, time_step) = qr
+                if (returns%qd_flag) returns%qd(row, col, time_step) = qd
+                !$AD end-exclude


A revoir si on change les flags de return options

inoelloc · 2024-06-06T17:04:21Z

smash/fcore/operator/md_vic3l_operator.f90

+                !$AD start-exclude
+                !internal fluxes
+                if (returns%stats_flag) then
+                    returns%stats%internal_fluxes(row, col, 1) = pn
+                    returns%stats%internal_fluxes(row, col, 2) = en
+                    returns%stats%internal_fluxes(row, col, 3) = qr
+                    returns%stats%internal_fluxes(row, col, 4) = qb
+                end if
+                if (returns%pn_flag) returns%pn(row, col, time_step) = pn
+                if (returns%en_flag) returns%en(row, col, time_step) = en
+                if (returns%qr_flag) returns%qr(row, col, time_step) = qr
+                if (returns%qb_flag) returns%qb(row, col, time_step) = qb
+                !$AD end-exclude


A revoir si on change les flags de return options

inoelloc · 2024-06-06T17:04:44Z

smash/fcore/routine/mwd_parameters_manipulation.f90

@@ -74,6 +74,7 @@ module mwd_parameters_manipulation
    use mwd_options !% only: OptionsDT
    use mwd_returns !% only: ReturnsDT
    use mwd_control !% only: ControlDT_initialise, ControlDT_finalise
+    use mwd_stats !% only: StatsDT


A enlever surement

inoelloc · 2024-06-06T17:05:07Z

smash/core/simulation/run/run.py

+    if wrap_returns.stats_flag:
+        wrap_returns.stats.fluxes_keys = INTERNAL_FLUXES[model.setup.hydrological_module]
+        wrap_returns.stats.rr_states_keys = STRUCTURE_RR_STATES[model.setup.structure]
+


A revoir peut etre je sais pas

le déplacer apres wrap_forward_run apres if ret:

inoelloc · 2024-06-07T08:11:55Z

smash/fcore/forward/md_simulation.f90

+            if (returns%stats_flag) then
+                do i = 1, setup%nfx
+                    call compute_fluxes_stats(mesh, t, i, returns)
+                end do
+
+                do i = 1, setup%nrrs
+                    call compute_states_stats(mesh, output, t, i, returns)
+                end do
+            end if
+            !$AD end-exclude


Peut etre mettre ca dans store_time_step

pag13

Est que les nombreux commentaires et suggestions de FC ont été addressés.

Quid du mwd_stat ?

Quid de la resolution de pb de la diff auto ?

à investiguer ensemble et finaliser asap svp.

asjeb · 2024-09-18T12:24:59Z

Maintenant que l'on a les flux internes est ce que une classe stat python ne serait pas tout aussi adapté ? Ou bien qq chose à récupérer sur les gradients ?

pag13 · 2024-09-18T12:51:51Z

Maintenant que l'on a les flux internes est ce que une classe stat python ne serait pas tout aussi adapté ? Ou bien qq chose à récupérer sur les gradients ?

Coucou, je ne suis pas sûr de bien comprendre, peux tu clarifier stp ?
Statistiques (sur dim spatiale et ou temporelle) de states/flux en Python - quid vs calcul de stat sur tableaux fortran vs memoire ?
Une fonction Python pour tracer des cartes de gradient serait très utile, Truyen a réalsié des scripts pr cela et publi WRR.

Apolline El-Baz added 27 commits March 1, 2024 14:11

writing lateral discharge by passing option in forward

b04f8ad

options discriminated by GR(4)

8485fdc

access fluxes

7910d90

add attribut mesk_gauge

487880c

stats for pn

2aba721

add function compute_stat(one flux)

ccd5f41

add class stats and internal_fluxes returns option

a45d177

compute_stats all fluxes structures

1026348

move internal_fluxes inside stats class

6d7a75d

add obj stats

648248f

add a getter

009ebd4

process lexc and median

fe07b71

add stats on the states

187052f

add a getter for the fluxes

8764da9

add keys for the states inside Stats class

fdcb106

write fluxes one by on all times

b5ab0fb

Merge branch 'main' into write_fluxes

c2c878b

fixed stats

32eceb0

change functions signatures of sim/GR

2834ce8

fix files format

8571dbc

modified forward_db

3ba2d22

Fix format

ed99b8f

type checking of stats_getter

a0b8114

Merge branch 'maintenance/1.1.x' into write_fluxes

a5ac29e

excluded compute_stats from adjoint computation

109ac6e

arange module/routine

a5706e9

forward tap changing

b579be6

asjeb requested review from pag13 and inoelloc June 6, 2024 10:17

inoelloc added the enhancement New feature or request label Jun 6, 2024

inoelloc added this to the Release v1.1.0 milestone Jun 6, 2024

inoelloc linked an issue Jun 6, 2024 that may be closed by this pull request

ENH: Save internal fluxes #180

Closed

pag13 reviewed Jun 7, 2024

View reviewed changes

inoelloc reviewed Jun 18, 2024

View reviewed changes

pag13 reviewed Sep 7, 2024

View reviewed changes

asjeb closed this Nov 19, 2024

nghi-truyen deleted the write_fluxes branch December 10, 2024 15:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Write fluxes #199

Write fluxes #199

asjeb commented Jun 6, 2024

pag13 left a comment

pag13 Jun 6, 2024

pag13 Jun 6, 2024

asjeb Jun 24, 2024

pag13 Jun 6, 2024

pag13 Jun 6, 2024

pag13 Jun 7, 2024

pag13 Jun 7, 2024

pag13 Jun 7, 2024

pag13 Jun 7, 2024

pag13 Jun 7, 2024

inoelloc Jun 6, 2024

inoelloc Jun 6, 2024

inoelloc Jun 6, 2024

inoelloc Jun 6, 2024

inoelloc Jun 6, 2024

inoelloc Jun 6, 2024

inoelloc Jun 6, 2024

inoelloc Jun 6, 2024

inoelloc Jun 6, 2024

inoelloc Jun 10, 2024

inoelloc Jun 7, 2024

pag13 left a comment

asjeb commented Sep 18, 2024

pag13 commented Sep 18, 2024

		SIMULATION_RETURN_OPTIONS_TIME_STEP_KEYS,
		STRUCTURE_RR_STATES,

Write fluxes #199

Write fluxes #199

Conversation

asjeb commented Jun 6, 2024

pag13 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pag13 left a comment

Choose a reason for hiding this comment

asjeb commented Sep 18, 2024

pag13 commented Sep 18, 2024