Example which requires external data

[ivan-pi]

I am assembling an fpm package with an example program that needs to load data from a file.

The file tree is something like:

├── examples
│   ├── dataset.dat
│   └── example.f90
├── fpm.toml
└── src
    └── lib.f90

The file "example.f90" contains the command open(newunit=unit,file='dataset.dat',action="write"), which obviously fails when I run fpm run example. What would be good package practice in this case? Should I modify the example to read the line command line input for the file?

Will in this case a command like fpm run example ./example/dataset.dat work?

[awvwgk]

I have a similar issue with the current design of fpm test (see #179) for my TOML-Fortran project, but no answer on how to deal with this problem.

First, one issue is that you don't know the current working directory, for fpm it is currently always the project root, but for most other build systems it is usually the build directory, which might be somewhere. For fpm itself the working directory might also change at some point, so it is dangerous to rely on this.

So far, the design of fpm, cargo, stack, ... seem to not consider passing command line arguments to an executable or invoking them with a runner by default, it is left as an exercise to the user.

[ivan-pi]

So it means the best I can do right now is to have my app ./build/<compiler>_<mode>/example emit a message and ask the user to run it manually with the complete or relative path to the data...

With CMake I used to add a copy command as follows:

file(COPY ${CMAKE_CURRENT_SOURCE_DIR}/../graphs/fdm.graph
     DESTINATION ${CMAKE_BINARY_DIR}/bin/)

[awvwgk]

Maybe you have a better proposal for a way to specify default arguments / runners in the package manifest. I would be happy to move this forward if we agree on a specification.

[ivan-pi]

I will take a closer look at your issue.

If I were to follow the Cargo model, the easiest way to achieve what I need is to have a build script (Makefile or Fortran program) which would do the following:

program main

  use fpm_tools, only: join_path, copy
  implicit none
  character(len=255) :: outdir
  integer :: stat

  ! Build scripts communicate with Cargo by printing to stdout. 
  ! Cargo will interpret each line that starts with cargo: as an 
  ! instruction that will influence compilation of the package. 
  ! All other lines are ignored.

  print *, "fpm:rerun-if-changed=examples/dataset.dat"

  ! Build scripts may save any output files in the directory 
  ! specified in the OUT_DIR environment variable. Scripts should 
  ! not modify any files outside of that directory.

  call get_environment_variable("OUT_DIR",outdir,status=stat)
  if (stat /= 0)
    error stop "Error retrieving environment variable"
  end if

  ! The build script’s current directory is the source 
  ! directory of the build script package.
  
  call copy(source="examples/dataset.dat",directory=outdir)

end program

If I understand correctly the "OUT_DIR" env variable remains defined when invoking Cargo, so I can then use it in my example code to find the correct location of my dataset file.

[everythingfunctional]

I would suggest that hard-coding filenames and paths is generally bad practice in almost any scenario. The most common solution (and IMHO preferable) is to pass filenames/paths as command line arguments.

As you noticed, we don't currently have a method of setting the default runner or command line options for a given executable (or test or example), but this would probably be the "right" way to do it.

However, this doesn't answer the follow up question of, what is the current working directory when an executable is run? Right now it's the project root, but as @awvwgk pointed out, we haven't set that in stone, and so that probably shouldn't be relied on. Probably fpm should have some "variables" defined that can be used for such things. Off the top of my head I would suggest the following, but there are probably more:

• FPM_PROJECT_DIR - Absolute path to the root of the project
• FPM_EXECUTABLE_DIR - Absolute path to the directory of the executable about to be run

[ivan-pi]

I believe your strategy is similar in spirit to what Cargo does: https://doc.rust-lang.org/cargo/reference/environment-variables.html#environment-variables-cargo-sets-for-build-scripts

In their case they have OUT_DIR as the environment variable which is to be accesed by the build script. The FPM_PROJECT_DIR in their case is called CARGO_MANIFEST_DIR:

• CARGO_MANIFEST_DIR — The directory containing the manifest for the package being built (the package containing the build script). Also note that this is the value of the current working directory of the build script when it starts.
• OUT_DIR — the folder in which all output should be placed. This folder is inside the build directory for the package being built, and it is unique for the package in question.

[certik]

This is a very common case that we need to have a solution for and document how to do it.

When I implemented the Rust prototype for fpm, then cargo test runs the test from the root directory. I suggest we do the same and let users know they can rely on that.

So if you change your code to:

open(newunit=unit,file='examples/dataset.dat',action="write")

It should start working.

[ivan-pi]

If we can't get variables at runtime, and we can't do it in a portable way at compile time, what even remains? Develop all executables/tests in a way that they are completely agnostic of the build system and environment.

Any filepath information required by a program or test should be retrieved using get_command / get_command_argument and get_environment_variable. If I am using fpm to run my programs I should make no assumptions about location in a command such as fpm run my_prog <path-to-file> and supply the absolute path for any command-line arguments.

[ivan-pi]

I suppose these issues did not exist in the punch card era. If you did not have the the data present in physical form as a deck of punch cards or magnetic tape, it was obvious your program can't run.

Some time ago I was fiddling with some old NASA quadrature routines (see PDF here); here is an excerpt from the main program (I added the comments starting with c):

      PROGRAM SAMPLE(INPUT=1, OUTPUT=1, PUNCH=1
     +              ,TAPE5=INPUT,TAPE6=OUTPUT,TAPE7=PUNCH)
      COMMON//X(2000),W(2000)
      DOUBLE X,W
      COMMON/BOFN/B(100)/COFN/C(100)
      DOUBLE B, C
      DOUBLE S(10),PD(10),PS(10),Q(10)
      DOUBLE ALPHA,UA,H,HP,HS,ADD,FAC
      DOUBLE DLOG,PNFUN
      READ 101, IFLAG
      IF(IFLAG.NE.0) GO TO 12
*  IF IFLAG.NE.0 THE RECURSION COEFFICIENTS ARE READ IN.

*  PART 1 - RECURSION COEFFICIENT CALCULATION
c
c  the coefficients are calculated using some user input
c   about what type of polynomial and how many quadrature points
c
c   the interesting part is the coefficients are punched (!) back
c   onto cards
c
      PUNCH 301, ALPHA,LX,NC,NMAX
      DO 11 N=1,NMAX
   11 PUNCH 302, N,B(N),C(N)
      GO TO 14

*  PART 2 - QUADRATURE ABSCISSA AND WEIGHT CALCULATION

   12 READ 301, ALPHA,LX,NC,NMAX
      DO 13 N=1,NMAX
   13 READ 302, N,B(N),C(N)
*  THIS PART OF THE PROGRAM COMPUTES THE ABSCISSAS AND WEIGHTS.
*  IT IS WEIGHT FUNCTION INDEPENDENT
   14 READ 101, N1,N2,N3
      IF(N2.LE.0) STOP 2
      IF(N3.LE.0) L3=I
      PUNCH 303, ALPHA,N1,N2,N3
      DO 15 N=N1,N2,N3
      PUNCH 304, N
      CALL NGAUSS(N,X,W)
      DO 15 M=1,N
   15 PUNCH 302, M,X(M),W(M)
      IF(ALPHA.EQ.1..AND.NMAX.GE.16) CALL COMPARE
      STOP 1
c
c  (format statements)
c
      END         PROGRAM SAMPLE

The program would actually punch the polynomial recursion coefficients onto cards. The next time you ran the program you would simply place the punched coefficients on top of your card deck and set IFLAG to a number different then zero to read them in.

[everythingfunctional]

IMO, one should use some combination of get_command_argument, get_environment_variable, and (in limited circumstances) read(*,*) to determine runtime "configurations" in a way that makes the most sense for the given application. One should not rely on assumptions about file locations, especially with regards to build system specifics. Doing so leads to extra complications and requirements placed on the user, and difficulties with portability.

For information that should be determined at compile time, we should devise a portable way of determining that information at compile time. An idea I had was perhaps to devise a "standard" module that could be supplied by fpm containing some standard compile time info like version, package name, etc. No preprocessor needed then, although then the package truly is reliant on fpm.

[awvwgk]

Totally agree with Brad here, environment variables and command line arguments are probably the best solution here.

Now how do we allow to set those with fpm in the package manifest to pass those arguments or environment variables to our test/example program?

[everythingfunctional]

My suggestion is some additional entries in the executable/example/test section for specifying default runner, command line arguments, and environment variables for a given executable. There should be some "variables" that fpm would replace before passing these along as well. (i.e. $FPM_BUILD_DIR)

[ivan-pi]

The accepted answer in rust is to base paths against the manifest directory: https://stackoverflow.com/questions/30003921/how-can-i-locate-resources-for-testing-with-cargo

I couldn't find an answer to what were the design decisions behind Cargo. They seem to have some open issues around things such as "Provide access to project paths for tests".

One case which might conflict with running the tests from the root directory is if the root-directory is read only. An example would be an fpm package located in a shared folder on a HPC system. In that case I should still be able to navigate to the fpm package and build it in a target directory under my own user path, e.g. fpm build --target-dir ~/fpm_shenanigans/. This would seem to exclude running the tests in the root directory.

[certik]

I still think running from the root (manifest) directory is the way to go. That's what Cargo does and it seems to work.

This would be contrary to most existing high-level build systems and will break the project upon export to CMake and meson, which we planning to do at some point.

Yes, we definitely plan cmake and other backends. I don't see right now why cmake couldn't do the same. If ctest is to be used to execute tests, then the cmake backend has to generate a proper stub (bash or fortran executable) that will switch the directories appropriately.

Read only file system

I think tests should still be executed from the root even on read only file system. By root I mean the (read only) directory with the manifest. Tests should only read from the directory (your use case above). If a test has to create some output, it should do it in a temporary directory.

[urbanjost]

I like the idea for those who have to have a pre-defined path or who might have output files generated that if a directory called run/ exists that the run command runs from that directory. This could be supplemented with an environment variable $FPM_RUNDIR.

Entries in the manifest file or a --rundir option on the run command would add a lot of flexibility but complications as well. It would be emphasized the location is strictly meant for scratch use and to remove the ambiguity surrounding a specific location for running (there would be a natural tendency to put input files there, which has pros and cons -- that would probably be good for a specific run, but would be a bad place to put them permanently, I think).

Currently we are not depending on the CHDIR() extension (extremely common in Fortran implementations) or on POSIX interfaces but this could be easily implemented currently, as fpm has to always start (currently) in the package root, so just prefixing the command executed with "cd run;" would work for ULS systems. I am not sure all systems support compound commands separated by a semi-colon but the equivalent should be possible on any current system and that should be easy. If we assume gfortran is used to build fpm (or PGI or Intel compilers ...) we could use CHDIR() in the meantime as well.

Perhaps instead of run/ tmp/ would be a good name. On second thought, I like run/ as it is clearly associated with running programs.
To make it more obvious to users there is a question as to whether fpm new would do anything by default to create a run/ directory or not.

but in general I do think if you use the fpm command to run something you should be certain fpm itself is running in the root directory of the project. I think that should be guaranteed.

[urbanjost]

A related use case:

no input files, but output files in the current directory

I thought I would describe a related use-case that is quite far along. In this case a bunch of example programs are intentionally very simple and just create an output file in the current directory (a graphics file -- gif, svg, PostScript, ...). When the examples are run as-is in fpm it results in a bunch of junk in the project root. So this is a case where I want the examples to be simple (no environment variables, no reading the command line, no prompting for an output directory ..) and running them from fpm is a bit messy. I can install the examples and run them directly somewhere else, which is what I currently do; but that has a few minor issues too.

So to simulate this you can set up something like

 tree
├── example
│   ├── demo10.f90
│   └── demo20.f90
├── fpm.toml
├── README.md
└── src

where each example program does a "write(LUN,*)'output for',LUN" where demo10 writes to LUN=10, etc..

So if I run that now I end up with something like "fort.10" and "fort.20" in the project root (names are compiler-specific and I argue non-standard but all compilers allow it, probably for historical reasons). That is ugly ( * and requires I have write access to the project home,
but I think that is a problem with using fpm for something it was not (currently?) designed for).

nit 1

I have to have a (empty) src/ or app/ directory for this to build by the way, which I do not like._ I do not think an example-only directory is unreasonable,especially for trying out remote dependencies. That would only require a line or two to allow, especially since
an empty src/ directory gets rid of the problem.

nit 2

install does not have an option to install example programs so I can install the executables somewhere else with a run command with a --runner option, or make a copy of the package and built it from scratch myself or make fancier examples that handle running from fpm better. Not always all trivial steps. So for this case (no input files; a lot of output files in current directory) using fpm has complications. But in this case I want install to not just be able to install the binaries, but the source of the examples; and currently no --list option or packaging option exists to do that. Since all the examples are in example/ it is not much of a big deal to copy them to
a user-accessibe area in this case, but I have wondered if any other source package manager allowed for such options.

So for this case having a run/ directory for the developer helps manage the mess while developing. So that is the related part and how
to simulate it.

the real case

In the real case, I am really jdeveloping a library that is installed as a "normal" library in an HPC environment, and the user is unaware of fpm. The examples and documentation are copied to a standard area accessed by an environment module, and the user is told to copy the examples to his own working area and how to access and build the examples, and the documentation is installed and has become part of his "standard" environment. The examples are primarily for use by a developer using the library, not just for running as-is in fpm so this works well (enough). fpm works well for development, and creates a package that can easily be placed into a git repository for easy access by other developers of the library, but fpmis not really yet integrated into the process used by developers using the library as-is. So except for the mess in the root directory, and that one of the test cases runs the examples to verify their output, fpm is working well enough for this so-far, but being able to specify a run directory would be nice. It is running so nicely that
I removed a bunch of Cmake and Make-related files, and recently almost eliminated the TOML file. It just has metadata and a [link]
section. The code is composed of C, Fortran and uses external system libraries. Nice.

fpm does not work well so far as also providing an installed library product to be used in a shared HPC environment, but I think integrating with a packaging system for binary releases is a better direction there. I do not picture end-users of a program using the fpm package to run an application, for example. For environments where the result is primarily for use in the local HPC environment install is adequate for the most part.

So it would be helpful to the developer to have a guaranteed/controlled point of execution and some environment variables
for controlling output and input.

So far fpm lacks other features useful for a tiered development. If I put a package together that is a library for other developers to use and others then want to use it in their project as a dependency fpm works well as a source packager for them to build it into their project from source (which is great) but I still have to go "outside" of fpm or rummage through the build/ directory to access package documentation and/or examples. So far (if it is even in the package). But then the resulting library contains the dependencies
which is often not what I want as a final result, although useful for development.

As part of a solution for that I can pull my dependencies separately and look though them that way. I have been finding (using a slightly customized fpm) that for some packages instead of using them as remote dependencies I pull them separately and use them as local dependencies so I can modify them and look through their documenation better. I can then also built them and then just use them as [load] options instead of
as source dependencies, which works around some issues. So I think in some cases where I do not necessarily want everything built together it would be nice if dependencies were kept seperated more distinctly and used more like normal libraries relatively visibly;
like being pulled into subdirectories in the project that were more visible to the user than build/ and the main project would build and
just link to them in some fashion as an optional build method.

So fpm is working great for some cases. I love being able to go onto a machine with nothing on it but git and gfortran at home
and being able to build a bootstrap fpm just using gfortran and starting a project using five or six external packages as black boxes in a few minutes. Fantastic!

But if I do not know the other packages residency and how to use them, not having a searchable repository with a standard way to access documentation (perhaps just a required link or flat text file a registered package would potentially be required to provide) I find it hard to use anything but my own packages, but that is a topic being discussed a bit already.

So I thought I would share a production use of fpm, especially since one of the issues is where examples run at; and a few of the experiences so far.

nit 3

Maybe not a general issue, but I use a custom preprocessor for documentation purposes. Since I only edit these ".ff" files I have to
run the preprocessor to generate the affected .f90 files. It is a trivial command to enter, but make(1) was set up to handle that automatically, which I now have to remember to do manually. It would be nice to have some ability to add rules for custom suffixes
to call a preprocessor like Make provides with a few lines.

Regarding this particular package, I wish I could share some of the high-level graphics (in Fortran) that use the base library but that is not possible. It is hard to believe that Fortran + a little machine code once dominated the world of graphics. In the wider world it is hard to find a Fortran-based graphics package now-adays that is available as Open Source.