[DRAFT] scope required may be another version

[hboutemy]

in OBOM use case, component listed at runtime is very precise

in SBOM use case, in many languages or package/dependency managers, when the dependency tree is calculated at build time, components listed at build time may be replaced at runtime by another version "of the same component": this is particularly true with libraries.

This ability to change the version between build time and runtime makes risk evaluation of libraries not accurate.

This PR proposes to explicit this question of version: perhaps the best solution will be create another scope instead, to differentiate this strict version vs loose version

[ppkarwasz]

I like this proposal: it makes explicit the fact that the version of the dependency at runtime might be different.

The case, when the version of a dependency is fixed occurs IMHO only when the dependency is included (statically compiled, shaded, etc.) in the artifact. In all the other situations, you should be able to replace the dependency with a patched version.

As discussed in CycloneDX/cyclonedx-maven-plugin#472 there is already a way to express inclusion through $.components.components, so an additional scope might not be necessary. The composition mechanism is more powerful than a scope, since it gives a tree representation of which component is included in which one. A scope would just express the inclusion of a component in the main component.

Unfortunately, in practice, I haven't seen SBOMs use the composition mechanism, so we might:

• Either encourage developers to implement it in their tools. This change might encounter a lot of resistance in some ecosystems: for a Docker image it is "obvious" that all the dependencies are included.
• Or we could add a new scope (e.g. included) and deprecate composition.

What do you think?

[jkowalleck]

@hboutemy , please start your change request with writing a ticket discussing what the problem is, and why it is relevant.

We then can discuss all possible solutions on this very ticket.

(A PR w/o a proper issue means somebody skipped the most relevant part: give people a proper understanding of the "problem" and give them the freedom to think about possible solutions freely. Starting with a solution always narrows the view of everybody, and in almost all cases the proposed solution is not the best one.)

[jkowalleck]

this would make a MUST a MAY.

Spec-wise, i dislike the change. it is a breaking change from OBOM's point of viev:
In an OBOM a required component has a specific version.
with this change, spec-wise, this would be no longer the case.

[jkowalleck]

@hboutemy , the problem you describe, that this PR is intended to solve,
it seams like it is much related (almost identical) to use-cases in ticket #321
It also discusses alternative solutions, and shows other related cases.

What do you think?

[hboutemy]

@jkowalleck to be honest, I knew this was a bigger issue than "just a PR", but I could not figure out precisely where/how to report in a concise way (and I'm not used to working on specs...)

I don't expect this PR to be merged, as it would seriously change the CycloneDX 1.6 spec: I consider it more as an issue with one very concrete but incomplete approach

#321 looks related (just for fun, first comment on this issue is please provide a PR as a PoV = what I did here, with the exact intent :) )

this would add another use case to #321: "as a library supplier with a dynamic language and associated dependency manager (Java with Maven, Gradle or any other, C# with NuGet and DLLs, ...)", I ship a library that requires a component as a dependency: I know the version I will build my library with, but I don't know the runtime version of the dependency that will be selected by my users"

I don't get if #321 is only about what Id' call environmental prerequisites (OS, language runtime, ...), or about dependencies defined in build in languages where dependency management has an impact when consuming

the consequence is that it may require different solutions, I don't yet have an opinion, but I'm interested to participate in the discussion

[jkowalleck]

I don't expect this PR to be merged [...]

#321 looks related (just for fun, first comment on this issue is please provide a PR as a PoV = what I did here, with the exact intent :) )

If you continued reading, you would have seen #326.

🤡
Just for fun, can I close your PR then, since it does not solve #321? :)
Anyway, I see your PR as a "draft" since you expressed there was no initial intention for merging in the first place.

this would add another use case to #321 [...]

👍
Could you comment this on the respective ticket, so it can be part of the discussion?

[...] I don't yet have an opinion, but I'm interested to participate in the discussion

👋
You are more than welcome to write comments for #321 and pushing the discussion forward.

[algomaster99]

Since @hboutemy and I have talked about this briefly, I would like to clarify that this PR scope required may be another version does not fit any of the use case in #321

The problem is well summarised in the PR description

when the dependency tree is calculated at build time, components listed at build time may be replaced at runtime by another version "of the same component": this is particularly true with libraries.

This happens because there can be dependency conflicts while building a package as there could be two versions of the same dependency. Which version of dependency is selected strongly depends upon the dependency resolution algorithm of the build tool. See this post by my research group and I also summarise it for you here:

1. Maven: the dependency that comes first in BFS of the dependency tree is selected.
2. Go: selects the latest version of SemVer.
3. NPM: includes all the dependent under nested node_modules
4. Pip: will take the latest version according to SemVer if the version pattern allows.

Now why I think this topic deserves its own issue is because there is no use-case in #321 that fits this topic:

These external dependencies need to be provided by the later runtime

Dependency is not external. It is part of the dependency tree. Only the version is unknown.

which runtime-dependencies need to be provided by the user of my library

Maven has the concept of Uber jar where the dependency is copied into the jar and shipped along. In that case, we want to know the precise version of dependency. If these dependencies are not bundled into one Uber jar, then this use case fits.

Rest of the use-cases do not pertain clearly. However, if it is not clear, I am happy to clarify.

[stevespringett]

This happens because there can be dependency conflicts while building a package as there could be two versions of the same dependency. Which version of dependency is selected strongly depends upon the dependency resolution algorithm of the build tool.

This is a well-known challenge with package managers. However, I fail to see why this is a specification issue. In CycloneDX, a BOM asserts the identity of components. Changing that has a cascading effect on other things. SBOM creation should not be viewed as a one-time event, but rather a process. This is documented on page 32 of the Authoritative Guide to SBOM and CISA has working groups that are doing similar things. Expecting the output from a build, which relies on a package manager, to have a completely accurate SBOM, is not realistic. What's more realistic is to utilize a process of enrichment and verification which takes the SBOM as input and makes any corrections to it throughout the process, resulting in an accurate component inventory (including the versions of components that are delivered). The enrichment and verification process itself can even be documented in the resulting SBOM for independent verification.

[jkowalleck]

re: #578 (comment)

Since @hboutemy and I have talked about this briefly, I would like to clarify that this PR scope required may be another version does not fit any of the use case in #321

The problem is well summarised in the PR description

when the dependency tree is calculated at build time, components listed at build time may be replaced at runtime by another version "of the same component": this is particularly true with libraries.

What you describe is literally the first use case listed in #321.

Furthermore, #321 describes in the section "Discussion" why scope is probably not the right choice for a spec-change.

Please read #321 again. short version: a non-shipped dependency that the downstream-users need to provide -- exactly what you described in your comment and what this very PR discusses, too.

However, if it is not clear, I am happy to clarify.

@algomaster99 , please clarify.

[algomaster99]

This is documented on page 32 of the Authoritative Guide to SBOM and CISA has working groups that are doing similar things. Expecting the output from a build, which relies on a package manager, to have a completely accurate SBOM, is not realistic.

Thanks for the clarification, @stevespringett !

What's more realistic is to utilize a process of enrichment and verification which takes the SBOM as input and makes any corrections to it throughout the process,

So I believe this issue would be more suited for cyclonedx-maven-plugin?

[algomaster99]

What you describe is literally the first use case listed in #321.

I disagree. Conflicting dependencies are not "external/extraneous" dependencies. For me, an external dependency would be a Jar file that is loaded at runtime from its URL. This is encapsulated by the last two use-cases.

please clarify.

The other use cases talk about the runtime version and hardware which are both irrelevant. Anyway, @stevespringett 's answer is a clear answer for me that this issue is not a specification problem and something that them SBOM producers like cyclonedx-maven-plugin should take care of.

[hboutemy]

SBOM creation should not be viewed as a one-time event, but rather a process.

I understand on SBOM of an app, or anything that embeds dependencies.
Here, the issue is for SBOM of libraries, that never embed dependencies in their whole lifecycle = what many OSS project produce, then try to create SBOMs for as good citizens for their consumers, and are hit later by people consuming their SBOMs and seeing old versions of components listed, that are de-facto unused because replaced by newer versions at runtime.

What you describe is literally the first use case listed in #321.

not really: first use case explains about exceptional and advanced case, for example when license of the dependency does not permit embedding
Here, it is the vast majority of dependencies of libraries (in many languages): not embedded.
We would need to add an explicit additional use case, the solution could match (the "extraneous" dependency is automatically provided by build tool / dependency manager eventually with another version when library is consumed).

One option with CycloneDX <= 1.6 is for SBOMs of libraries in these languages to not list dependencies at all, as they are not embedded (not shipped) as part of library: I fear this will cause frustration

[ppkarwasz]

Here, the issue is for SBOM of libraries, that never embed dependencies in their whole lifecycle = what many OSS project produce, then try to create SBOMs for as good citizens for their consumers, and are hit later on people consuming their SBOMs and seing old versions of components listed, that are de-facto unused because replaced by newer versions at runtime.

To be precise, libraries almost never embed dependencies, so SBOM consumers might be more lenient if the component type is library. Security problems might only arise if they embed dependencies. If we want to be strict by default, there should be an eco-system independent way to tell consumers that the version of the library (and all its dependencies) is only a suggestion or that the version of the dependency must exactly match the one provided.

The technical reason, why a consumer can not use a different version of the library is not really important.

[algomaster99]

If we want to be strict by default, there should be an eco-system independent way to tell consumers that the version of the library (and all its dependencies) is only a suggestion

This is exactly what this PR vouches for :)

[hboutemy]

we had a very in-depth discussion with @jkowalleck to analyze our differences: hard to relate here all the good learning we found, would be too big and diverse

but @jkowalleck proposed me one surprising option that IMHO solves our problem: declare components without version

my first reaction was: it would not be CycloneDX nor purl compliant! (honestly, in my mind, I silently added "stupid" first :) )
The fact is that version is really optional: see https://cyclonedx.org/docs/1.6/json/#components and https://github.com/package-url/purl-spec#purl

I never saw a CycloneDX document with a component without version, but it is definitively what we should have done from the very fist time to represent dependencies of libraries, that are not embedded, not shipped with the library, but to be resolved by consumer, with eventually (often?) a different effective resolved version
This implies we'll need also to clearly identify when it's not a library that is built, but an app that ships dependencies (or a library with some embedded deps): for these ones, version is necessary.
=> implementing will require serious work
But with that strategy in mind, now it's "just" a good implementation to do
And also good communication, because I think the first times CycloneDX documents listing components without versions will be reviewed, we'll have strong surprised reactions, or even negative :)

[stevespringett]

That sounds like a great solution. Yes, version is optional. We did this because version is optional in package.json. In fact, name is also optional, but we had to draw the line somewhere, and we decided to prohibit anonymous components.

But yes, version is also optional in purl so that we can represent versionless components.

I would hope that the majority of components from a Maven build would have a version and its only a few corner cases that would not.

To mitigate the "strong reaction" upon first seeing this, you may want to consider using CycloneDX Annotations, as a way to automatically provide commentary as to why a component is versionless. For the annotation, the subject would be the bom-ref of the component, the annotator would be the tool (Maven plugin) making the annotation, and then just supply the timestamp and text description of why its versionless. The "strong reaction" may turn into a pleasantly surprised one. 😄

[hboutemy]

I would hope that the majority of components from a Maven build would have a version and its only a few corner cases that would not.

You'll be surprised:

• for libraries, this is the opposite
• but for apps (war, ear, executable jar, ...), assemblies, shaded, there is a version.

The key reasoning is "shipped or not shipped in the product":

• when "shipped in the product", everything is strictly controlled at build time, when assembling the shipped product. In Java + Maven, this is the case for war, ear, binary assembly, "fat jar" from different forms including executable jar, shaded jar (with a scope of copied content varying given configuration), ...
• when "not shipped in the product", like library, provided dependencies from Maven plugin or Jenkins plugin, OSGi dependencies, ..., the build resolves a version to compile, but does not copy the result nor really record "dependency tree": runtime dependency will be resolved later by a consume-time resolver (Maven again, when library is consumed, Jenkins server or OSGi container for other cases, ...)

To mitigate the "strong reaction" upon first seeing this, you may want to consider using CycloneDX Annotations

thanks for the good idea: yes, definitively we'll have to do this

[hboutemy]

notice: the "shipped vs non-shipped in the product" reasoning + version--less existing in npm makes me feel that

1. this is not Maven specific but impacts other build tools including npm: some detailed analysis by experts will have to happen, like the one I did for Maven to distinguish the 2 cases, because this does not impact 100% of each build tool use cases
2. I'm sure that it will also impact Feature: documenting external/extraneous dependencies #321 : if you document version-range, it implicitely means it's "not shipped with product", then probably should have been described version-less

[jkowalleck]

notice: the "shipped vs non-shipped in the product" reasoning + version--less existing in npm makes me feel that

1. this is not Maven specific but impacts other build tools including npm: some detailed analysis by experts will have to happen, like the one I did for Maven to distinguish the 2 cases, because this does not impact 100% of each build tool use cases

I think after #321 becomes a part of CDX, tool makes will find a good use case to apply the new "extraneous" flag and version-ranges - for exactly this reason.

We might add to the authoritative guide, that a BOM for non-bundled/non-shipped MUST NOT have concrete version, but MAY have a version-range(which may match exactly one version), and MUST be flagged as "extraneous".
The same goes to BOMs with the $.metadata.lifecycles[].phase=design.
(PS: added this as followup to #321)

anyway, here is an example how such a task would look like for node-tools

• [FEATURE] library mode cyclonedx-node-yarn#252
• [FEATURE] library mode cyclonedx-node-npm#1259

2. I'm sure that it will also impact Feature: documenting external/extraneous dependencies #321 : if you document version-range, it implicitely means it's "not shipped with product", then probably should have been described version-less

thanks. :-)
this is already part of #321:

• add constraint, that a component's optional version MUST be omitted, if component's is "external"
• add constraint, that a component's optional version-range MUST be omitted, if component's is not "external"

PS: you are not alone.
see the section "motivation" of #321

[ppkarwasz]

but @jkowalleck proposed me one surprising option that IMHO solves our problem: declare components without version

That is a great idea, thanks both of you! 💯

I never saw a CycloneDX document with a component without version, but it is definitively what we should have done from the very fist time to represent dependencies of libraries, that are not embedded, not shipped with the library, but to be resolved by consumer, with eventually (often?) a different effective resolved version

We do use versionless components in our CycloneDX VDR, but that is an entirely a different application.

And also good communication, because I think the first times CycloneDX documents listing components without versions will be reviewed, we'll have strong surprised reactions, or even negative :)

I think we should stress in the specification that version means exact version required by the component, something like the = constraint for Debian packages, but not like 2.3.4 in Maven. Probably people will be surprised that 2.3.4 in Maven means any version (although 2.3.4 is recommended), unlike [2.3.4], which excludes the possibility of using other versions.

[hboutemy]

I think we should stress in the specification that version means exact version required by the component

no need to stress the spec: honestly the spec is immediate, version means immediate version

it's the build tool approach taken by any build tool that has to resolve transitive dependencies and conflicts that makes things more complex and unexpected for people not having really thought precisely about it: it has to be more flexible, or you can't assemble anything

[hboutemy]

CycloneDX/cyclonedx-maven-plugin#589 created for CycloneDX Maven plugin

(I let other build tools create their own issue to implement equivalent logic)