The eclipse-sdk-prereqs.target contains content with requirements that can't be resolved

[merks]

I've extracted all the direct-from-maven dependencies in eclipse-sdk-prereqs.target into a separate *.target file and am using tycho-p2-repository-plugin with includeAllDependencies configured to true to assemble a repository.

https://tycho.eclipseprojects.io/doc/latest/tycho-p2/tycho-p2-repository-plugin/assemble-repository-mojo.html#includeAllDependencies

I'm doing this a test that the requirements of all the bundles actually resolve to ensure that nothing important is missing. Also, I've specified includeDependencyDepth="none" to ensure that all the artifacts are explicitly listed with specific versions (that I can analyze for available updates).

This reveal the following anomalies:

Missing requirement: org.osgi.service.http 1.2.2.202109301733 requires 'java.package; javax.servlet [2.1.0,3.0.0)' but it could not be found
Missing requirement: org.osgi.service.http.whiteboard 1.1.1.202109301733 requires 'osgi.contract; (&(osgi.contract=JavaServlet)(version=3.1.0))' but it could not be found
Missing requirement: org.objectweb.asm.util 9.4.0 requires 'java.package; org.objectweb.asm.tree.analysis [9.4.0,10.0.0)' but it could not be found
Missing requirement: org.slf4j.api 2.0.6 requires 'osgi.extender; (&(osgi.extender=osgi.serviceloader.processor)(version>=1.0.0)(!(version>=2.0.0)))' but it could not be found

Not surprisingly, none of the bundles with these missing requirements actually appear in a 4.27 I-build's p2 update site.

For org.osgi.service.http and org.osgi.service.http.whiteboard I'm not sure if there are plans to replace what's in org.eclipse.osgi.services but at this point it's rather pointless to include then when it's not possible to use them.
@HannesWell @laeubi Perhaps you know the strategy here and the reason for the current inclusion of these two things in the target platform?

For org.objectweb.asm.util we should either remove asm-util, given it's not used, or add asm-analysis because it requires that.
@mickaelistria should we remove it or add what it requires?

For org.slf4j.api version 2.0.6 there is this ongoing issue:

#682

[akurtakov]

My opinion - if not used, should be removed. At the very least we shouldn't make licensecheck verify more libraries than actually needed.

[laeubi]

My opinion - if not used, should be removed. At the very least we shouldn't make licensecheck verify more libraries than actually needed.

Nothing from the target is analyzed by the license check as far as I know.

I'm doing this a test that the requirements of all the bundles actually resolve to ensure that nothing important is missing.

Beside that I'm not sure what the goal of this is, managing all dependencies "by hand" will just increase maintenance efforts and blow up the target, resolving inside an OSGi context is independent from target context so there is no such thing as "missing requirements" from the point of view of maven target locations.

[HannesWell]

Missing requirement: org.osgi.service.http 1.2.2.202109301733 requires 'java.package; javax.servlet [2.1.0,3.0.0)' but it could not be found
Missing requirement: org.osgi.service.http.whiteboard 1.1.1.202109301733 requires 'osgi.contract; (&(osgi.contract=JavaServlet)(version=3.1.0))' but it could not be found
Missing requirement: org.objectweb.asm.util 9.4.0 requires 'java.package; org.objectweb.asm.tree.analysis [9.4.0,10.0.0)' but it could not be found

These will likely will be resolved when #239 lands, which is part of the bigger effort in eclipse-equinox/equinox#18. When the former lands I should also be able to complete the latter, by removing the last sources of the http packages in o.e.osgi.services and deprecating that Plugin.

Missing requirement: org.slf4j.api 2.0.6 requires 'osgi.extender; (&(osgi.extender=osgi.serviceloader.processor)(version>=1.0.0)(!(version>=2.0.0)))' but it could not be found

As you said that is effectively handled by #682. But to make it really usable we should first complete #588.
If I can complete https://jira.qos.ch/browse/SLF4J-576 soon that migration should be completable as soon as a new slf4j version is released.

[merks]

At this point, everything resolves if I augment the TP with these (based mostly on helpful hints from @HannesWell)

        <dependency>
          <groupId>org.ow2.asm</groupId>
          <artifactId>asm-analysis</artifactId>
          <version>9.4</version>
          <type>jar</type>
        </dependency>
        <dependency>
          <groupId>org.apache.aries.spifly</groupId>
          <artifactId>org.apache.aries.spifly.dynamic.bundle</artifactId>
          <version>1.3.5</version>
          <type>jar</type>
        </dependency>
        <dependency>
          <groupId>org.slf4j</groupId>
          <artifactId>slf4j-nop</artifactId>
          <version>2.0.6</version>
          <type>jar</type>
        </dependency>
        <dependency>
          <groupId>org.apache.felix</groupId>
          <artifactId>org.apache.felix.http.servlet-api</artifactId>
          <version>1.2.0</version>
          <type>jar</type>
        </dependency>

[laeubi]

based mostly on helpful hints from

Describe the problem than a preset solution would allow to give more "helpful" hints ... from "others". The target obviously "resolved" before so we are probably talking about different things.

Really strange to see how platform is able to even transform the maven-targets thing that was meant to simplify things into something complex that needs management, manual work, additional releng effort and even different tools...

[merks]

It seems my compliment to @HannesWell is somehow interpreted as a slight to others. That was not the intent.

I think the problem is statement is pretty clearly in the title. But, more long winded, the target platform contains bundles with requirements that cannot be resolved by other things in the target platform nor by other things defined in the Platform source code. Such content cannot actually be used produce something that p2 can install. This is simply a sanity check that the target platform is logically complete.

It feels to me somewhat offensive to suggest that the "platform" (whomever or whatever that is) has taken something simple and made it complex, presumably unnecessarily so. In the end, SimRel consists of 60+ project, all of which need to coordinate which versions of which common libraries are being used such that we don't end up with 3 batiks and 3 guavas. The platform is modifying its subset of ~85 directed-from-maven bundles on a semi-daily basis, and new releases of those bundles appear on an ad hoc schedule. Sometimes those new versions need license review. There is no magic wand that makes such a complex problem simple. So I'm really not sure where you are finding fault...

[laeubi]

the target platform contains bundles with requirements that cannot be resolved by other things in the target platform

This is not actually a requirement. A target only offers items that might be used to compile (not required to contain 'everything') or build other items (might or not require complete items)

by other things defined in the Platform source code

This can change anytime.

something that p2 can install

Not everything in the target is meant to be installed by P2... beside that it depends on what is installed together in the EPP what things are satisfied from the current profile already.

and new releases of those bundles appear on an ad hoc schedule.

As long as you don't modify the target version or those do not include bad version range nothing will change

So I'm really not sure where you are finding fault...

As mentioned above:

1. flatten maven dependencies is not what one wants, the goal is to reduce the number of items to manage/upgrade
2. having "all maven" to form a fully self-contained set of OSGi resolvable items is not a requirement, nor is if "wrong"
3. Adding random stuff (e.g. slf4j-nop .. nop = not operational(!) -> from the docs: "Binding/provider for NOP, an implementation that silently discards all logging messages." ) is clearly not what one wants, there is a discussion already if/how platform should include a default binding for slf4j but including just a random one to satisfy any "checks" will not solve the issue at all, see for example Impossible to enable logging for Apache HTTP Client 5 (no slf4j implementation available) #1226

[merks]

This seems argumentative primarily for the sake of hypothetical argument and that strikes me as counter productive. It's likely pointless to point out (again) that the org.osgi.service.http* bundles are meant to be used, eventually, and then their requirements will need to be satisfied. Similarly slf4j-API version 2.0.6 is meant to be used much sooner rather than later, and again its requirements will need to be satisfied. Moreover, slf4j-nop version 1.7.36 is already in the target platform. It might well that no one wants that, but it's there nevertheless and we would not be able to create installations if it wasn't there.

So yes, hypothetically not everything in the target platform needs to have requirements that resolve, but in all these specific cases the bundles are 1) not needed at all, so why specify then and why maintain them, or 2) they are actually installed or are intended to be installed by p2, so it is wrong for their requirements not to resolve and it will need to be fixed sooner or later. The flattened set of bundles is what the target platform specifies already. I did not make it that way, so no point in finding fault here in this thread.

[laeubi]

This seems argumentative primarily for the sake of hypothetical argument

You initial question was

Perhaps you know the strategy here and the reason for the current inclusion

I just answered that. So if the actually question was "what do I need to include so I can install all and every bundle from the target into an empty p2 profile only using maven dependencies" then I clearly didn't understand it and my answer was hypothetical because my hypothesis was wrong from the beginning that the goal was to understand why we have items that are not strictly installable only by the definition of the eclipse-sdk-prereqs.target file.

The flattened set of bundles is what the target platform specifies already.

Then I also don't get what this in your initial post means:

Also, I've specified includeDependencyDepth="none" to ensure that all the artifacts are explicitly listed with specific versions (that I can analyze for available updates).

includeDependencyDepth and includeDependencyScopes are the maven ways to specify something similar to planner and slicer and if it is not used, requirements can not be automatically derived and thus requires explicit listing and managing of (manually derived) dependencies.

[merks]

I asked about the "strategy" specifically with respect to what I stated in that paragraph, i.e., "I'm not sure if there are plans to replace what's in org.eclipse.osgi.services but at this point it's rather pointless to include then when it's not possible to use them." That question was answered and addressed by this pending change:

#239

Yes, I fully understand what includeDependencyDepth means. This is how it's used by the platform:

I could have omitted mentioning it at all, since I'm doing exactly what the platform is doing already.

[laeubi]

I'm not sure if there are plans to replace what's in org.eclipse.osgi.services but at this point it's rather pointless to include then when it's not possible to use them.

That is not pointless, because of how the current sdk build works, it is not really possible to consume this other ways and thus we cannot test it or make progress. Beside as mentioned, this is just API, that means I can compile against that, but at runtime i can bind to either of those and it is the choice of the package what to include, I never want it to be installed it should be provided actually.

So the strategy is, we try out how far we can get and probably one time we would install the osgi provided jars instead of org.eclipse.osgi.services ... but that's nothing to configure in the target, that has to happen at the feature or product level and then we will see if something missing because the build then will fail ... or we then even probably would just consume it from somewhere else than the preq-target or add.

[merks]

So you present theories, while the problem is actually getting fixed:

#239

[laeubi]

So you present theories,

The only "theory" I see here is that the items are pointless ... while referencing an issue that shows how these items are used to identify issues and making progress.

while the problem is actually getting fixed

So can we close this?

[merks]

No.

[HannesWell]

It seems my compliment to @HannesWell is somehow interpreted as a slight to others. That was not the intent.

Thanks for that. :)

Without having followed exactly who said what, AFAIK it is possible to have units in the target that actually not resolved, but that is Not noticed because Tycho only resolves whats needed by the projects in the reactor.

In the end, SimRel consists of 60+ project, all of which need to coordinate which versions of which common libraries are being used such that we don't end up with 3 batiks and 3 guavas. The platform is modifying its subset of ~85 directed-from-maven bundles on a semi-daily basis, and new releases of those bundles appear on an ad hoc schedule. Sometimes those new versions need license review.

Do you plan to create a 'dependencies' p2-repo based on maven-targets that other Projects can consume?
I think that would simplify the SimRel.

[laeubi]

Do you plan to create a 'dependencies' p2-repo based on maven-targets that other Projects can consume?
I think that would simplify the SimRel.

Should we probably name it orbit? Thats exactly that part I mean about "turning into something more complex"... we are using maven-deps to not needing consume them by orbit to then transform it into something like orbit ...

It is and was always valid to have multiple versions of the same dependency, and this works without a problem (I have a product that has different guava + batik versions), the issues only arise because no one literally cares using proper version ranges and instead relies on platform to fixup these for them with extra ordinary burden on the releng process...

[merks]

@HannesWell Yes, a dependencies repo is what I'm prototyping; it's already working well locally, automatically composing the target platforms from multiple SimRel projects. I too think it will simplify SimRel and make it easier to coordinate versions across dozens of project.

@laeubi "Should we probably name it orbit?" is just kind of offensive as I'm sure you realize when you say it. Please stop doing this. It's simply not nice. I am not asking you nor the platform to do anything for me. Quite to the contrary, I'm doing many things for the platform, among other projects. I'm helping keep versions up-to-date (and automating that), helping things be more consistent (for the benefit of SimRel), and helping improve Maven publishing (for the benefit of Tycho and many others), and so on. It's often a thankless task, but with this you are crossing the thankless line and I really don't appreciate that.

It is not always valid to have multiple versions. It's perhaps usually valid, but it sometimes leads to wiring problems, when the dependencies peculate up into the API, particularly for Guava:

org.osgi.framework.BundleException: Could not resolve module: org.eclipse.emf.parsley [867]
  Unresolved requirement: Require-Bundle: org.eclipse.emf.parsley.runtime; visibility:="reexport"
    -> Bundle-SymbolicName: org.eclipse.emf.parsley.runtime; bundle-version="1.14.0.v20221128-1543"
       org.eclipse.emf.parsley.runtime [880]
         No resolution report for the bundle.  Unresolved requirement: Require-Bundle: org.eclipse.emf.parsley.common; visibility:="reexport"
    -> Bundle-SymbolicName: org.eclipse.emf.parsley.common; bundle-version="1.14.0.v20221129-1056"; singleton:="true"
       org.eclipse.emf.parsley.common [870]
         No resolution report for the bundle.  Bundle was not resolved because of a uses constraint violation.
  org.apache.felix.resolver.reason.ReasonException: Uses constraint violation. Unable to resolve resource org.eclipse.emf.parsley [osgi.identity; osgi.identity="org.eclipse.emf.parsley"; type="osgi.bundle"; version:Version="1.14.0.v20221128-1543"; singleton:="true"] because it is exposed to package 'com.google.common.base' from resources com.google.guava [osgi.identity; osgi.identity="com.google.guava"; type="osgi.bundle"; version:Version="31.1.0.jre"] and com.google.guava [osgi.identity; osgi.identity="com.google.guava"; type="osgi.bundle"; version:Version="30.1.0.v20221112-0806"] via two dependency chains.

Chain 1:
  org.eclipse.emf.parsley [osgi.identity; osgi.identity="org.eclipse.emf.parsley"; type="osgi.bundle"; version:Version="1.14.0.v20221128-1543"; singleton:="true"]
    require: (osgi.wiring.bundle=org.eclipse.emf.parsley.runtime)
     |
    provide: osgi.wiring.bundle: org.eclipse.emf.parsley.runtime
  com.google.guava [osgi.identity; osgi.identity="com.google.guava"; type="osgi.bundle"; version:Version="31.1.0.jre"]

Chain 2:
  org.eclipse.emf.parsley [osgi.identity; osgi.identity="org.eclipse.emf.parsley"; type="osgi.bundle"; version:Version="1.14.0.v20221128-1543"; singleton:="true"]
    require: (osgi.wiring.bundle=org.eclipse.emf.parsley.runtime)
     |
    provide: osgi.wiring.bundle; bundle-version:Version="1.14.0.v20221128-1543"; osgi.wiring.bundle="org.eclipse.emf.parsley.runtime"
  com.google.inject [osgi.identity; osgi.identity="com.google.inject"; type="osgi.bundle"; version:Version="5.0.1.v20221112-0806"]
    import: (&(osgi.wiring.package=com.google.common.collect)(&(version>=30.1.0)(!(version>=31.0.0))))
     |
    export: osgi.wiring.package: com.google.common.collect; uses:=com.google.common.base
    export: osgi.wiring.package=com.google.common.base
  com.google.guava [osgi.identity; osgi.identity="com.google.guava"; type="osgi.bundle"; version:Version="30.1.0.v20221112-0806"]
	at org.eclipse.osgi.container.Module.start(Module.java:463)
	at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel$2.run(ModuleContainer.java:1852)
	at org.eclipse.osgi.internal.framework.EquinoxContainerAdaptor$1$1.execute(EquinoxContainerAdaptor.java:136)
	at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1845)
	at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1786)
	at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.doContainerStartLevel(ModuleContainer.java:1750)
	at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1672)
	at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1)
	at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:234)
	at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:345)

I'm told multiple asm versions can be problematic as well, but I don't understand that problem In any case, even when it is valid, and doesn't cause problem, it's just not desirable. So hopefully I'll be forgiven for trying to address this problem for SimRel as a whole and we won't continue to spiral off the deep end here...

[laeubi]

"Should we probably name it orbit?" is just kind of offensive as I'm sure you realize when you say it.
It's simply not nice.

Why is it offensive to speak out the, what effectively is done here? Or how would you describe a "a dependencies repo ... to coordinate versions across dozens of project" otherwise?

Why is is more offensive then constantly claiming that my concerns are "theoretical" or not "pragmatic"? That's also not "nice" weather or not it was not meant that way. Its also not "nice" to effectively contradict the efforts of @akurtakov and @mickaelistria to bring in plain-maven-dependencies in contrast to some wherever-else-managed central repro ...

Anyways I wish you all the best luck and success to whatever will the outcome be as you obviously don't like to share your ideas beforehand as with the jetty-p2 repo that again was a way to turn back the wheel... you just should not assume everyone is happy with that and every criticism is meant as an offense.

[akurtakov]

Is there anything left here or this one should be closed?

[merks]

Some of the problems are resolved or have been worked around at my end now.