SPEC: create standard key for communicating dynamic dependencies to vendors/packagers

[xdg]

From discussion with @xsawyerx, we have a gap in META when dynamic config is true (or missing) that vendors or packagers have no way to know what might be included in any standard way. They have to run the .PL file or read it in order to discover it.

We could have a meta field that provides module names as keys and data structures with version and rationale that explains the conditions for a dynamic dependency (in a human-readable way).

Eg.

dynamic_prereqs => {
    runtime => {
        requires => {
            "Win32::Foo::Bar" => {
                version => "1.23",
                condition => "Required on Win32",
            },
            "Wibble::XS" => {
                version => "0",
                condition => "needs a compiler",
                use => "speed"
            },
            "Devel::ReallyAnnoying" => {
                version => "0.17",
                condition => "perls before 5.14",
                use => "avoid RT#666",
            },
            "Data::ReallyOptional::Foo" => {
                version => "0",
                condition => "during runtime, if installed",
                use => "prettier output",
            },
        }
    }
}

This would help vendors decide on what environment is necessary to run the .PL file to get certain dynamic dependencies configured. It would also allow metadata sites like MetaCPAN to provide some information about what non-canonical requirement might possibly be required at configuration time.

[karenetheridge]

If it's meant to be only human-readable, and not machine-readable, then couldn't comments in Makefile.PL serve the same purpose? That is -- authors should already be properly annotating their dynamic prereqs.

[xdg]

Comments in Makefile.PL don't only serve this purpose -- there may be many unrelated comments. This proposal makes it easy to find and to write tools that can display it. Whether there needs to be a lot of structure for a given module or just a single comment field is a bikeshed matter that can be settled later.

[Leont]

This sounds like a good idea, but it may be helpful to be slightly more machine-readable (even if we don't strictly guarantee it)

[dolmen]

I approve the idea of this feature.

But concerning the design, I think it would be worth trying to keep the structures similar to other existing structures in the spec:

• keep prereqs as (module => version) pairs
• I see some similarity with optional_features

So here is an alternate proposal:

  dynamic_prereqs => {
    "Win32 only" => {
      condition => "required on Win32",
      prereqs => {
        runtime => { requires => { "Win32::Foo::Bar" => "1.23" } },
      },
    },
    "Faster with Wibble::XS" => {
      condition => "needs a compiler",
      prereqs => {
        runtime => { requires => { "Wibble::XS" => "0" } },
      },
    },
    "perls before 5.14, avoid RT#666" => {
      condition => "perl before 5.14",
      prereqs => {
        runtime => { requires => { "Devel::ReallyAnnoying" => "0.17" } },
      },
    },
    "Prettier output, optional" => {
      condition => "used at runtime if available",
      prereqs => {
        # Notice I've changed the requirement type to 'recommends'
        runtime => { recommends => { "Data::ReallyOptional::Foo" => "0" } },
      },
    },

[kentfredric]

Should probably define how these "imagined to be handled" in the development lifecycle.

For instance, "optional_features" is otherwise the same as this, but there's the question of "when does tooling handle this, before, or after ./configure?"

"Can configure change this and emit the result in MYMETA.*?"

etc.

Given all the examples are /runtime requires/ this suggests the latter is happening.

So theoretical examples with configure_requires might be nice.

[xdg]

@xsawyerx had the example that an admin wants to know before running .PL and wants something more direct than having to read/grok a possibly poorly documented *.PL file. (Often for custom non-toolchain packaging e.g. chef/RPMs etc.) He should explain this idea further, I think, as I just wrote it up so it wouldn't get lost.

[xsawyerx]

It is sub-optimal (for the lack of a better term) to require to run code in order to know which modules will be required. Let's take MetaCPAN for example:

MetaCPAN would like to list all the requirements. Modules have a way to list requirements. Unfortunately, modules need to account for some dynamically-decided requirements (and possibly other configuration options), so they specify dynamic_config. However, that requires running the code. Those configuration options (and dynamically-decided requirements) might (and are meant for) different operating systems. That would mean MetaCPAN reports whatever it will install, not what a user might install. ("a user" and "might" were bold separately in this sentence.)

If MetaCPAN knew about this, MetaCPAN could list them under "The following might be required, depending on your situation". A proper modal could include the information about when and why they could be required.

[xsawyerx]

Now, taking a vendor tooling perspective, it is preferable (for the lack of a stronger term) to not run third-party code in order to understand what the requirements are. It is also preferable to read these from a machine-readable file rather than sifting through a Makefile.PL/Build.PL, which someone used to make this module a complete vendor solution.

A good example of how far the last sentence can go is the Makefile.PL of a distribution that decided to make it a complete vendor solution and installer, Net::FullAuto: 6897 lines (6006 of code), 254K size.

In C world we know that the contract between the developer and the user is the help menu of the configure/Configure program. The only difference is that they do not have a meta file. We do. We might as well use it.

[kentfredric]

@xsawyerx can you state why this is better than using the optional_features hash, or otherwise extending the definition of optional features?

The only real distinction I'm seeing here is optional features are "user decides" where dynamic_config is "makefile.PL + env decides".

I think that might be a a narrow enough distinction to consider unifying the data structures somehow.

maybe an "available_if" => "condition" in "optional_features"

[xdg]

optional_features is not right for this. From the spec:

Consumers must not include optional features as prerequisites without explicit instruction from users (whether via interactive prompting, a function parameter or a configuration value, etc. ).

At least some of what @xsawyerx is talking about are the automatic configure-time dependencies that people enable via Makefile.PL probes.

Then there are things like "runs faster if these XS modules are installed", which could be considered as optional_features, but are often done (as in JSON::MaybeXS) as configure-time probes for a compiler. If the probe succeeds, the XS version is added to prereqs. That approach does not fit into optional_features.

[Leont]

Any system that is powerful enough to be useful will essentially be a DSL:

dynamic_prereqs => {
  "Win32 only" => {
    condition => "os=MSWin32,
    prereqs => {
      runtime => { requires => { "Win32::Foo::Bar" => "1.23" } },
    },
  },
  "Faster with Wibble::XS" => {
    condition => "compiler=true,
    prereqs => {
      runtime => { requires => { "Wibble::XS" => "0" } },
    },
  },
  "perls before 5.14, avoid RT#666" => {
    condition => "perl<5.14",
      prereqs => {
        runtime => { requires => { "Devel::ReallyAnnoying" => "0.17" } },
      },
    },
  "Prettier output, optional" => {
    condition => "installed(Data::ReallyOptional::Foo) >= 0",
    prereqs => {
      # Notice I've changed the requirement type to 'recommends'
      runtime => { recommends => { "Data::ReallyOptional::Foo" => "0" } },
  },
},

[xdg]

Goal is to avoid a Turing complete. Aim is "human readable info in well
defined place".
On Jun 27, 2016 1:41 PM, "Leon Timmermans" [email protected] wrote:

Any system that is powerful enough to be useful will essentially be a DSL:

dynamic_prereqs => {
"Win32 only" => {
condition => "os=MSWin32,
prereqs => {
runtime => { requires => { "Win32::Foo::Bar" => "1.23" } },
},
},
"Faster with Wibble::XS" => {
condition => "compiler=true,
prereqs => {
runtime => { requires => { "Wibble::XS" => "0" } },
},
},
"perls before 5.14, avoid RT#666" => {
condition => "perl<5.14",
prereqs => {
runtime => { requires => { "Devel::ReallyAnnoying" => "0.17" } },
},
},
"Prettier output, optional" => {
condition => "installed(Data::ReallyOptional::Foo) >= 0",
prereqs => {
# Notice I've changed the requirement type to 'recommends'
runtime => { recommends => { "Data::ReallyOptional::Foo" => "0" } },
},
},

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#112 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AHRaapVBIpY_3Tr2LWZHI0irkVcq_t2Qks5qQAs3gaJpZM4I9RFP
.

[karenetheridge]

It feels like half the people in this thread are assuming this spec is for something that will be machine-parsable. I thought it was only intended to be human-readable.. but I will bet you a dollar that someone will see this data and write a tool to try to parse it...

[xsawyerx]

@kentfredric optional_features is a different thing. It defines what features a user could pick. dynamic_config is what a system has decided it should installed, based on what a user has available or on the system on which the user runs Makefile.PL.

You use optional_features for "Pick what you like" and dynamic_config for "This is what I decided you should have, based on what you run or what you have".

Some of the modules in dynamic_config are clearly not optional, but rather depend on what you must have. For example, having a dynamic configuration that adds a Win32 module (because you run Windows and that's the appropriate module), it might as well add it as required. It isn't "optional" for you to pick "no". It's required for any Windows user.

In short, very different things.

[kentfredric]

Yeah. They become more different really specifically because of the "non-turing complete" goal.

Mentally and logically, there will be cases where there are "optional features" that the user cant choose because system requirements prohibits it.

However, there's no present way to express that, other than munging optional features during configure ( and that doesn't work, because the design of OPT features is its supposed to happen before configure, eugh, or something like that )

Granted having non-turing conditions on the optional features can't help that situation either.

Was hoping to euthanise two birds with one needle.

[xsawyerx]

I don't think the differences have to do with turing-complete conditions. The optional features are meant for the user to decide and dynamic config is meant for the computer to decide.

That makes optional features at least supposed to handle only optional features, decided by the user (and sure, possibly a default value here), while dynamic config is meant for supported (on a technical/platform/etc.) configuration - the ol' "You have Windows, you need this module". It's not something a user picks or decides. It's not "optional".

The reason they are confusing is because they both fall under "You might get these installed", but for different reasons and purposes, and by different agents.

[Leont]

Goal is to avoid a Turing complete. Aim is "human readable info in well
defined place".

Meta files have always been meant for machines, and I never really considered JSON human-friendly.

It that's what you want, why not add a PACKAGING with an explanation file or some such?

[kentfredric]

It that's what you want, why not add a PACKAGING with an explanation file or some such?

That's sadly of limited utility here, because one of the motivations here is "Because the actual deps are declared in a structure, metacpan can display them ... somehow".

This seems to be part of our ongoing struggle how META.json is used during the install process and as a non-installation informational-transfer structure.

I do agree I would rather we not continue to proliferate this sin. But there's not a lot of ways out atm.

[dolmen]

Goal is to avoid a Turing complete. Aim is "human readable info in well defined place".

Meta files have always been meant for machines, and I never really considered JSON human-friendly.

JSON human-friendliness is out of scope here. The intent was there is part of meta files meant purely for machines and part of meta (such as optional_features entries list) meant to be read by machines but presented to a human. optional_features has been designed to try to standardize existing practices of interactive questions at configure time which already required a human.

I've never been enchanted by optional_features because so far the install tools don't store anywhere what has been selected, and this makes install reproducibility and automatic upgrade (preserving previously selected features) impossible.
But as prereqs in META v2 doesn't allow to request some optional_features to be enabled, this is a an inconsistent solution anyway, so restricted to very few uses cases. There is [very few optional_features uses on CPAN](http://grep.cpan.me/?q=file%3AMETA.json optional_features"\s*%3A\s*{%24). And it has especially not gained traction on the CPAN client tools authors side. But this is of course in part a chicken-egg problem.

But to agree somewhat with @Leont, I think that "human readable info in a well defined place" only gives frustrations to humans wanting that information in a 100% machine readable way.

Despites not being an ideal solution dynamic_prereqs as at least partially machine readable is a step forward. We must take care to leave space in the spec for further improvement.

[kentfredric]

The intent was there is part of meta files meant purely for machines and part of meta (such as optional_features entries list) meant to be read by machines but presented to a human.

The problem here is that the "condition" here is not designed to be computable, it is only informational, its just a free-form text field.

Making "condition" computable means we have to implement an entire mini-language to define the use of conditions.

Hence, the actual conditions are to be ignored by automated processes, and are only a suggestion for humans to read to understand what the attached payload does.

Quote:

we have a gap in META when dynamic config is true (or missing) that vendors or packagers have no way to know what might be included in any standard way.

This would help vendors decide on what environment is necessary to run the .PL file to get certain dynamic dependencies configured.

It would also allow metadata sites like MetaCPAN to provide some information about what non-canonical requirement might possibly be required at configuration time.

Emphasis added.

This data does not have a "machine" as its end user like the rest of the CPAN tools do. Even optional features have "a machine" as the end user, because the machine evaluates the structures and presents choices based on those choices to the user.

There is however no desire for this feature to be used in an automated fashion by installers in the initial design request.

frustrations to humans wanting that information in a 100% machine readable way.

• if the definition of the language is not designed to be machine read, then it won't ever reach 90%
• if the definition of the language is designed to be machine read, we're going to have to implement a turing complete mini-language, which then gets us back to the original problem where vendors have to read code to work out what's going on.

Hence, the more pragmatic approach is:

"Here's some deps, and here's a description that humans can read to determine what they might do, but run the code itself to determine what in fact must happen".

Having 2 different code systems to determine the same answer is surely going to turn into self abuse.

I can at best see a middle ground where the "dynamic_prereqs" stash contains a "Token" key of some kind that identifies "this block of stuff is turned on by code", and then the *.PL files can then refer to that same token and implement the actual logic to enable that prereqs set.

[kentfredric]

I'm just going to dump a non-JSON alternative solution to poison your mind with, because personally, inventing a new turing complete language when everyone is going to have Perl at the ready anyway ( and will need it anyway for an implementation of this new language ):

Inspired by CPANFile syntax hybridised with syntax from @karenetheridge's DZP:DynamicPrereqs.

providing "Additional Win32 Support" => sub {
      condition "Needs win32" => sub { $^O =~ qr/Win32/ };
      requires "Win32::Foo::Bar" => "1.23";
};
providing "Acceleration via Wibble::XS" => sub {
      condition "Needs C Compiler" => \&has_compiler;
      requires "Wibble::XS";
};
providing "perls before 5.14, avoid RT#666" => sub {
     condition "Needs Perl  5.14 or earlier" => sub { $] <= 5.14 };
     requires "Devel::ReallyAnnoying" => "0.17";
};

providing "Prettier output, optional" => sub  {
       condition "Already has thing" => sub { has_module("Data::ReallyOptional::Foo", ">=0"); };
       on "test" => sub {
           recommends "Data::ReallyOptional::Foo";
       };
};

This structure can be statically extracted to produce a non-executable textual description that a Vendor can read without needing to know any deep magic, doesn't require them to learn a new language.

And the structure itself can also be directly executable by installer toolchains. ( because the "condition" block simply associates a reference to the sub, which can then be executed as needed )

The non-executable descriptive parts of this can be statically extracted and stuffed in a JSON file somewhere as "information only" usages similar to how cpanfile currently serves as a staging ground for META.json.

[xsawyerx]

I think this completely misses the point here. A vendor shouldn't have to run code to know what the prereqs are. We have a meta file to convey dependencies. If we have condition-based dependencies, we should at least make those visible. It really is that simple.

[kentfredric]

If we have condition-based dependencies, we should at least make those visible. It really is that simple.

One part of the equation is having a framework to expose that data.

The other part of that equation is having a mechanism to communicate that data to the META.json.

And ideally, one does so without duplicating effort.

So: requirements:

1. A static, vendor readable set of data that doesn't need executing to understand.
2. An ability to communicate that data somehow to the vendor
3. An ability to implement logic that actually implements what we told vendors we were doing at install time.

Obviously its too hard to derive 3 from 1 without horror shows.

And its an expected problem to have 3 constantly in sync with 1 if they're implemented independently.

And obviously, native code structures don't convert 3 to 1 on their own.

Hence, the proposed secondary structure that can be used to produce targets both 1 and 3.

[xdg]

I declare this thread "bikeshedded". At this point, we know what the ideas are. Should we ever have a v3 spec RFC, the specifics can be debated then.