Lua scripting

[ghost]

Now that fortran-lua by @interkosmos has support for fpm, interkosmos/fortran-lua53@8298146, fpm could expose parts of fpm_model_t to Lua script, for customization of the build process.

For example, I would use this feature in

-- This is fpm.lua in the package root directory

function OnBuild()
   -- pseudo-lua code

   if fpm.compiler.name == "gfortran" and fpm.compiler.version > 9 then
      fpm.compiler.add_flags("-fallow-argument-mismatch")
   end

   -- OK, continue build process
   return true
end

where

• OnBuild is a predefined function to be called by fpm after fpm_model_t is built, and before the actual build starts.
• fpm.compiler.name and fpm.compiler.version are read-only properties
• fpm.compiler.add_flags is a function called from Lua into fpm. Fpm can accept or reject writes to the model.
• return true - tells fpm everything is alright. return false to stop the build process.

[ivan-pi]

I was thinking about something similar lately while reflecting upon #611. The TOML spec supports multi-line strings (see https://toml.io/en/v1.0.0#string), meaning that you could in principle include a script directly in the manifest.

Instead of Lua, I was thinking of a pseudo-Fortran language, a variation of the "simple programming language" (spl) introduced in the book Fortran Tools for VAX/VMS and MS-DOS. The authors of that book wrote a compiler/interpreter for their pseudo-Fortran language in Fortran. Programs are executed by a language virtual machine (again written in Fortran).

The main difference of spl to Fortran is the language has reserved keywords (simplifies parsing), and only a subset of types is supported including real, integer, logical, and character (there is no "kind" specifier). There are a few other quirks on how I/O and strings are handled.

I started working my way through the online book Crafting Interpreters to hopefully build the skill set needed to pursue such a project, but there's still a long journey ahead.

Given the maturity of Lua, and the fact it can be built/installed on any platform with a C compiler, fpm + Lua form a good combination. The biggest disadvantage I see for now is that not many Fortran programmers are familiar with Lua.

[awvwgk]

This is kind of a fundamental question in the design of fpm, whether we want allow a turing complete language as a build script / package manifest (CMake, scons, waf, xmake, ...) or whether we want to restrict ourselves by using a limited DSL (meson) or a configuration language (cargo, stack). Turns out in practice even tools like cargo have a way to circumvent the self-imposed limitations of the package manifest language, either by adding logical selectors as keys or loading a build script written in a turing complete language.

I personally like the way meson is going by offering a non-turing complete language for the build files, this reduces the complexity that you might have to deal with as user significantly, however you don't need a turing complete language to pull big stunts in your build files ;). Still, having guardrails that stop you from getting carried away helps a lot.

We are seeing this issue with fpm now, TOML alone is just too limited to express the complexity of some Fortran builds. I personally think that a build script in some other programming language (Fortran, Lua, Python, ...) is not the right step to take. However, adding some simple logical selectors in our package manifest to deal with different operating systems and compilers might be desirable, for this purpose we could use a Fortran-like expression parser to evaluate some logical conditions.

[ivan-pi]

In principle with Lua, you could have the scripts run in a sandbox, e.g. https://stackoverflow.com/questions/1224708/how-can-i-create-a-secure-lua-sandbox

If I understand correctly when you load a function you can control the environment the function runs in, restricting access to language features which are potentially unsafe.

But I see now this is a bigger design decision. I don't know enough about meson to say what are the advantages of the non-scriptable approach. What I do see with the scriptable build tools (CMake, scons, waf, xmake) is they quickly become a mess to deal with.

It would be good to understand what are the use cases where the manifest is too restricting. You've mentioned two, platform- and compiler-specific settings, optional library dependencies and preprocessors are two more areas.

[zoziha]

It seems that there is a cargo-make (a cargo plug-in) here, which can make some conditions for judging, maybe we can refer to:
https://github.com/sagiegurari/cargo-make#conditions

cargo-make: Rust task runner and build tool.

[tasks.test-condition]
condition = { platforms = ["windows", "linux"], channels = ["beta", "nightly"] }
script = '''
echo "condition was met"
'''

Although it looks a bit cumbersome.

[ghost]

OK. I understand your concerns and I agree this is an important design decision. I chose Lua because the language is simple, mature and designed to be embedded in other programs; we can easily remove parts of Lua standard library we don't need. Regarding the alternatives, I'm not familiar with SPL or DSL, and can't say if they are good or not. If they are good, it is OK for me. Anyway I am trying fpm with Lua and I think it is very nice. Here's a quick demo.

fpm-lua.mp4

And this fpm.lua

print("Hi guys, I'm a lua script ...")

print("fpm.version: ",   fpm.version)
print("fpm.fc: ",        fpm.fc)
print("fpm.fcversion: ", fpm.fcversion)
print("fpm.fflags: ",    fpm.fflags)

if fpm.fc == "gfortran" then
    if string.find(fpm.fflags, "-Wall") then
        print("-Wall enabled!")
    end
elseif fpm.fc == "lfortran" then
    print("Good choice")    
end

if math.pi > 3.14 then
    print(math.pi)
else
    -- lol
end

print("-----------------------------")

[awvwgk]

I started working my way through the online book Crafting Interpreters to hopefully build the skill set needed to pursue such a project, but there's still a long journey ahead.

Thanks for the book tip, I put together an implementation of the tree-walking interpreter in Fortran (https://github.com/awvwgk/flox). My main takeaway from this exercise is that while writing an expression evaluation in Fortran is quite straight-forward, the implementation can be a bit lengthy.

One application would be linking against platform specific libraries like ld, which is not needed on Windows (dynamic loading features would be required for most embedded scripting languages). Having the option to specify this in the manifest would be important, but we have to agree on a way to represent this information. Some suggestions:

[build]
link = [
  "${os /= 'windows' ? 'ld'}$",  # requires parsing the string for special tokens
  {if="os /= 'windows'", then="ld"},  # use table to create conditional
]

We could exploit TOML tables as replacement for a full grammar here and limit the parsing only to the expression for the condition, which would be Fortran-like and with predefined parameters (os, fc, arch, ...).

[eli-schwartz]

But I see now this is a bigger design decision. I don't know enough about meson to say what are the advantages of the non-scriptable approach. What I do see with the scriptable build tools (CMake, scons, waf, xmake) is they quickly become a mess to deal with.

The limitations on the mess you can get into is a major advantage of its own. Over at meson, we have a FAQ entry on the topic: https://mesonbuild.com/FAQ.html#why-is-meson-not-just-a-python-module-so-i-could-code-my-build-setup-in-python
The link in that FAQ entry makes a compelling argument IMHO. :D

Meson policy is, roughly speaking, that if users feel there is a need to do something very clever with their builds, they should consider instead hammering out an API design for a meson module (meson does not permit thirdparty modules, that would constitute "coding your build setup in python") with a clean interface for performing that need in a declarative way.

I personally like the way meson is going by offering a non-turing complete language for the build files, this reduces the complexity that you might have to deal with as user significantly, however you don't need a turing complete language to pull big stunts in your build files ;). Still, having guardrails that stop you from getting carried away helps a lot.

And of course the ultimate fallback is "write a python script, declare it as a custom_target() in meson, and have it run, do its thing, and produce a file as output". You can even use that to glue multiple build systems together -- people do this, for example, to run cargo and produce an executable or library which is then used elsewhere in the meson.build
But no matter what you do along these lines, the meson.build parts will tend to be simple enough to understand (and that is what is in charge of declaring the build edges in Ninja).

[eli-schwartz]

I think that as a key-value file format which is essentially "a glorified ini file", you won't be able to be very expressive in TOML and you are doomed to any form of conditionals you consider implementing, being... non-optimal. It's probably fine for static metadata though.

[ivan-pi]

Thanks for the book tip, I put together an implementation of the tree-walking interpreter in Fortran (https://github.com/awvwgk/flox). My main takeaway from this exercise is that while writing an expression evaluation in Fortran is quite straight-forward, the implementation can be a bit lengthy.

Glad you found it interesting. I think the bytecode interpreter might be a better fit to Fortran? It's really cool that you used the Rust-style diagnostic messages in your Fortran lox interpreter 🚀. I was also kind of surprised I couldn't find any fypp files in your repository. As a side note, did you identify any missing string functions that would ease writing a interpreter in Fortran? (I ask because Kernighan writes that's how they came up with many tools in the C standard library.) A blog post with your takeaways and discussion of some the design choices would be of great interest. The Crafting Interpreters book cracks a few jokes about Fortran. I think it would go down well with the author if you inform him about flox. 👍

Thanks @eli-schwartz for joining the discussion. By key-value format, you are referring

 {if="os /= 'windows'", then="ld"}  # use table to create conditional

to this part? Personally, I find it a bit peculiar that two table entries are needed for a single branch. Consider the example from @jvdp1 in #611 (comment) which requires two branches. Would it translate to the following?

link = [
  {if="fc==gfortran", then="gomp",elseif="fc==ifort",then="iomp"}
]

(Built-in handling of OpenMP has it's own issue: #161)

[awvwgk]

Thanks for the insights, @eli-schwartz.

Meson policy is, roughly speaking, that if users feel there is a need to do something very clever with their builds, they should consider instead hammering out an API design for a meson module (meson does not permit thirdparty modules, that would constitute "coding your build setup in python") with a clean interface for performing that need in a declarative way.

Introducing a similar policy in fpm might be the most reasonable way forward. Encoding the logic in dependencies provided by fpm would free us from introducing any conditional constructs in the manifest syntax.

[eli-schwartz]

@ivan-pi in that snippet it seems you have two different values for the toml configuration key link.then and that would probably either choke the parser or cause one to overwrite the other.

That's what I mean by a key-value format.

It's possible you could still encode conditional branches into a key-value config file by using e.g. link.if1, link.then1, link.elseif2, link.then2, link.elseif3, link.then3.

link = [
  {if1="fc==gfortran", then1="gomp",elseif2="fc==ifort",then2="iomp"}
]

And iterate over keys and translate them into a traditional if/then/elseif/then/elseif/then structure by trimming the ordering/uniqueness integer and parsing it into a proper branching tree.

Alternatively you could give up on expressing it using the toml key-value format and just have a link key with a string value of the form:

if fc == "gfortran"; then return "gomp";; elseif fc == "ifort"; return "iomp"; endif

Combining two languages (one config, one scripting) in one file.

If you teach the build system about this instead, it can autodetect everything it needs from e.g. the $FC environment variables (this is what meson does) and use toolchain files to let people specify their environment. (See: machine files.)

e.g.

[profiles.gnu-fanatic]
fc = "gfortran"
link = "gomp"
[profiles.intel-did-nothing-wrong]
fc = "ifort"
link = "iomp"

And then you can invoke fpm build --profile .... possibly loading these from a separate file even.