Refactor FloPy Simulation, Model, Package #37
Replies: 8 comments 14 replies
-
|
a few more:
|
Beta Was this translation helpful? Give feedback.
-
|
after some discussions I think we can outline a plan for checks:
|
Beta Was this translation helpful? Give feedback.
-
|
the existing grid modules are stable and probably outside the scope of this work, but we should devote some attention to synchronizing them with dis and other packages |
Beta Was this translation helpful? Give feedback.
-
|
maybe we should consider deprecating |
Beta Was this translation helpful? Give feedback.
-
|
Coming back to MFStructure. Our internal usages mainly look at |
Beta Was this translation helpful? Give feedback.
-
Excellent! |
Beta Was this translation helpful? Give feedback.
-
|
Another: we have long discussed converting DFNs to TOML, YAML or some other standard data interchange format. Maybe also opportunities to shrink/revise the DFN specification |
Beta Was this translation helpful? Give feedback.
-
|
some tentative steps here (summarizing from discussion)
Ability to load DFNs and emit TOML. Also to structure the input specification, moving from flat to nested representation. Where currently a DFN file is a list of variables each of which can be part of another variable, an input definition is really a tree with composite variables (lists, records, unions) as internal nodes and scalars/arrays as leaves. This is the internal structure used for code generation and runtime introspection. Moving to structured definition files makes it unnecessary to inflate composite types from a flat list in Python. Nested TOML files still appear flat so they will look quite familiar.
We discussed using FloPy as a staging ground for the new TOML format. The code generation mechanism can temporarily add an internal step to convert DFNs to TOML before loading them. This gives us a place to kick the tires until we are comfortable taking it elsewhere
Migrate the DFNs to TOML in the MF6 repo. This will take a bit of work as DFNs are parsed to generate docs and source code, but the structured format will hopefully have similar benefits here as in FloPy |
Beta Was this translation helpful? Give feedback.
-
|
we should version the file format too: v1 is DFN, v2+ as TOML |
Beta Was this translation helpful? Give feedback.
-
|
a short-term action item: check which DFN variable attributes are used by flopy, and which by mf6... are any redundant or unnecessary? |
Beta Was this translation helpful? Give feedback.
-
|
once we have trialed the new toml definition file format in flopy for some time, we feel it has stabilized, and we are ready to adopt it back in mf6, we could consider moving the toml conversion script and any dfn-related tooling to modflow-devtools so both flopy and mf6 can use it |
Beta Was this translation helpful? Give feedback.
-
|
Another from discussion with @langevin-usgs and @jdhughes-usgs: review input file path management. Any component, many blocks, variables, and even parts of variables (e.g. array layers) can be associated with input files (see MF6IO appendix for more details on |
Beta Was this translation helpful? Give feedback.
-
|
Design-wise, file path management seems pretty straight forward. A simulation contains simulation data, and the simulation data contains a file management object. During load, write, and run, this classDiagram
class MFSimulationBase {
simulation_data: MFSimulationData
set_sim_path(path:StrPath)
}
class MFSimulationData {
mfpath: MFFileMgmt
}
class MFFileMgmt {
simulation: MFSimulationBase
existing_file_dict: dict[MFFilePath]
+ copy_files()
+ get_model_path()
+ get_sim_path()
~ get_updated_path()
~ strip_model_relative_path()
~ set_sim_path(path:StrPath)
~ unique_file_name(file_name, lookup)$
~ set_last_accessed_path()
~ set_last_accessed_model_path()
~ resolve_path()
}
note for MFFileMgmt "simulation reference can go, because there\nis a deprecation warning in set_sim_path\nthat it should become an internal function.\nThat would remove the circular dependency.\nIs copy_files really intended for public use?"
MFSimulationBase *--> "*" MFModel
MFSimulationBase *--> "1" MFSimulationData
MFSimulationData *--> "1" MFFileMgmt
MFSimulationBase <-- MFFileMgmt : calls set_sim_path()
MFModel ..> MFFileMgmt: calls static unique_file_name()
MFFileMgmt *--> "*" MFFilePath
Looking at this drawing, I think it would be best to move MFFileMgmt to an internal file (_file_manager.py), showing the intention directly to the user. Anything that would need to be made public for users should be a function in MFSimulationBase instead. Candidates are |
Beta Was this translation helpful? Give feedback.
-
|
I was playing a bit with "Move static methods to modules". I figured that the # _model_factory.py with an underscore so it's internal
_models_by_type: dict[str, type["MFModel"]] = {}
def register_model_type(model_type: str, model_class: type["MFModel"]) -> None:
_models_by_type[model_type] = model_class
def get_model_class(model_type: str) -> type["MFModel"] | None:
_models_by_type.get(model_type)I don't think it needs to be anything more, and it can be internal this way. We can do a similar setup for the parts of package, because that does exactly the same. |
Beta Was this translation helpful? Give feedback.
-
|
we can do an incremental deployment once we have a full initial prototype: first internally (flopy tests, mf6 tests, mf6 example models) before considering a public release |
Beta Was this translation helpful? Give feedback.
-
|
eventually we want to rewrite tutorials/examples where each is more focused/finer-grained/shorter |
Beta Was this translation helpful? Give feedback.
-
|
mf6 tutorials maybe a good place to start, topics including
linked PRs |
Beta Was this translation helpful? Give feedback.
-
|
I was looking a bit at |
Beta Was this translation helpful? Give feedback.
-
|
Some items on the topic of parity between flopy3 and flopy4 input files... in general we want to shoot for functionally identical, with possible cosmetic differences. Floating point precision:
Other possible criteria to consider:
|
Beta Was this translation helpful? Give feedback.
-
|
related: whether to support wrapped arrays.. is this needed, since mf6 no longer has a line length restriction? |
Beta Was this translation helpful? Give feedback.
-
|
related: currently flopy allows comments at the end of lines, with no comment character — consider requiring end of line comments to start with e.g. "#"? |
Beta Was this translation helpful? Give feedback.
-
We will need to make quite some adjustments in the FloPy code base. We will need to refactor the current code base and split out certain functionality into a more modular approach, before we can actually implement some of the new ideas that we have. This discussion aims to make a breakdown of the work that is still to be done. From there we can start estimating the work.
Work with minimal impact on the API
Refactor
PackageContainerto composition over inheritance ✅I have been playing with the idea of doing a refactor and keeping as much in place as possible. I agree that
PackageContainerandPackageInterfaceare daunting classes, but we can separate the functionality in a more modular fashion to simplify our intentions.What we see in the current flopy design, is the overuse of inheritance where composition would most likely be sufficient. We see that both simulation, model, and package are inheriting from
PackageContainer, while having aPackageContaineras a field would be sufficient. Inheritance should be used when there is also functionality that expects any kind ofPackageContainer, no matter what the subclass is. I have not found such code. In this case inheritance was used to share functionality among different types of classes. Which can also be achieved with composition, as thePackageContainerdoesn't need anything from its subclass and there are no abstract methods inPackageContainer.During modflowpy/flopy#2324, I came to the point that it was very well possible to do this refactor, but I have doubts on which functions can actually be removed, because they are supposed to be for internal use only. There are options to either just remove them, keep them, or
@deprecatethem.Move functions out of
PackageInterfacethat can be done modularWe could also look at other types of functions that could move out, like the
exportfunctionality. It would make more sense to do this in a similar fashion tocattrswith a module and single-dispatch functions. The functions that can go out ofPackageInterfaceare:plot(),export(), andcheck().It is possible to do this with minimal user impact by keeping the
plot()function, but@deprecatethem in favor of the modular version. Theplot()function inPackageInterfacewould simply point to the modular version.Add type hints to critical public classes
I do expect that refactoring is a tough job at the moment, because there are no type hints in the code. VS Code cannot follow the methods and properties when renaming or moving and doesn't give any error messages when functions no longer exist. I recommend adding type hints in the classes we want to refactor, and let mypy run over it.
This can be done with minimal impact on backwards compatibility, as we are just adding type hints, and not removing anything for the user.
Move static methods to modules
Some functions are now decorated as
@staticmethod, but they don't really apply to their class and could just as well be a global function in a module. E.g.Package.add_to_dtype()doesn't seem to use anything from thePackageclass and could just as well be a separate function.Similar to making functions like
plot()more modular, the same accounts for these@staticmethods. We could keep the old ones for now and@deprecatethem. Then simply point to the new modular version.Let generate_classes.py use
jinja✅The code that is now generated with print statements and
write()functions. It would be so much easier to use jinja, so it is clearer to read the initial file that is being used for generation. The jinja package handles template generation better thanwrite()with raw strings.Generate dfn classes on user install
See #35.
This idea would have minimal impact on the user, as it is being performed during get-modflow. So no changes in the API.
Work with likely more impact on the user
Use XArray / Dask to speed up performance
See #49.
Use
attrsfor property generationThere is quite some boilerplate code for variable getters and setters. The
attrspackage could help with this and simplify the code.Introducing attr itself and simply replacing the things that we have would not have any impact on the user's code, but if we decide to move / remove certain variables, it will.
Beta Was this translation helpful? Give feedback.
All reactions