Unable to use SmallRye config in native image

[thai-op]

I'm trying to use SmallRye ConfigMapping in a native image build (not Quarkus-based) and the native-image is unable to load the config mapping class due this this issue:

java.lang.UnsupportedOperationException: Defining new classes at runtime is not supported
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.util.VMError.unimplemented(VMError.java:136)
	at [email protected]/java.lang.invoke.MethodHandles$Lookup.defineClass(MethodHandles.java:46)
	at io.smallrye.common.classloader.ClassDefiner$1.run(ClassDefiner.java:19)
	at io.smallrye.common.classloader.ClassDefiner$1.run(ClassDefiner.java:14)

Then when I looked into the specific ClassDefiner implementation#419, I found this comment.

**
     * Do not remove this method or inline it. It is keep separate on purpose, so it is easier to substitute it with
     * the GraalVM API for native image compilation.
     *
     * We cannot keep dynamic references to LOOKUP, so this method may be replaced. This is not a problem, since for
     * native image we can generate the mapping class bytes in the binary so we don't need to dynamically load them.
     */

    private static Class<?> defineClass(final Class<?> parent, final String className, final byte[] classBytes) {
        return ClassDefiner.defineClass(LOOKUP, parent, className, classBytes);
    }

Then I followed the breadcrumb and found that Quarkus indeed replaces that method in order to run in native image. So my question is: is SmallRye config framework not a native image friendly? And could we somehow bake the logics used by Quarkus here so that folks can use SmallRye as a standalone library in a native image?

Thanks

[radcortez]

Hi @thai-op, thanks for using SmallRye Config standalone :)

Unfortunately, that is the case :(

Mapping implementations for standalone JVM mode are generated on the fly and loaded in the appropriate Classloader. For Quarkus, implementations are generated during compilation, which is a requisite for the native image, but also benefits the Quarkus JVM mode.

Until now, we never got a use case to support native as standalone, so we piggybacked on Quarkus. It shouldn't be too difficult to do it. Pretty much all the code is there, but we would need to attach a plugin to generate the implementations.

Any chance that you could help with this work? At the moment, I have a few other priorities so it may take me a while to get to this. Thanks!

[thai-op]

Well, I spent a day trying to make it work using similar code to Quarkus but have yet to make it working. It's a work project so I don't want to spend too much time debugging / fiddling when the alternative is just "get rid of it and use plain properties config file".

My current problem is that SmallRye seems to expect an instance of the class to be created during compilation time. For example, after the native image build, the library is looking for a target=<CLASS_NAME><HASH_CODE> to be found in the native class path. That seems correct since we can't use reflection so the class has to be built from the interface and materialized somewhere.

Where/how do I get the native image compilation to take care of that step?

[radcortez]

My current problem is that SmallRye seems to expect an instance of the class to be created during compilation time. For example, after the native image build, the library is looking for a target=<CLASS_NAME><HASH_CODE> to be found in the native class path. That seems correct since we can't use reflection so the class has to be built from the interface and materialized somewhere.

Correct. Implementations class names are generated using a combination of the interface name plus a hash of the FQN to avoid clashes. What you observe is SmallRye Config trying to load the implementation class.

Where/how do I get the native image compilation to take care of that step?

Something like this should work (it is what we do in Quarkus):

List<ConfigMappingMetadata> configMappingsMetadata = ConfigMappingLoader.getConfigMappingsMetadata(Mapping.class);
for (ConfigMappingMetadata metadata : configMappingsMetadata) {
    String fullQualifiedName = metadata.getClassName();
    // parse path and name
    new FileOutputStream(className + ".class").write(metadata.getClassBytes());
}

That method will provide you all generated classes for a hierarchy, so you need to call that for each class annotated with @ConfigMapping. The getClassName returns you the fully qualified name, so you need to create the directories for and parse the class name. Finally, just write the bytes in the class file. This is direct bytecode for each class.

I will provide a plugin for this, but it will take me some time to get into this, since I'm with other priorities at the moment :(

[thai-op]

Ah, now that makes sense. That's the missing piece. So basically I'll need to manually write all *.class files during the build phase into the current ./target/classes which helps the native image to find those classes during its runtime.

This would be perfect for an annotation + annotation processors. Thanks for the pointers!

[radcortez]

Ah, now that makes sense. That's the missing piece. So basically I'll need to manually write all *.class files during the build phase into the current ./target/classes which helps the native image to find those classes during its runtime.

Correct.

This would be perfect for an annotation + annotation processors. Thanks for the pointers!

It depends... the API requires the compiled mappings classes to generate the implementations. An annotation processor would only work if these are available. It would need to have the mappings in a separate module to be compiled and then generated in the required module by the processor.

An alternative is to make a plugin that takes the compiled sources and then generate the implementations after compilation (in process-classes for instance).

I already had a few thoughts about moving the code to an annotation processor, which would enable us to also generate documentation for the config, but that is a huge task since the APIs are so different.

[thai-op]

An alternative is to make a plugin that takes the compiled sources and then generate the implementations after compilation (in process-classes for instance).

A plugin as in a Maven / Gradle plugin? That would work too. Although I think annotation processors are a little better now since it is more well-integrated (in the IDE for example), and we are already using a bunch of annotations. The idea of generating documentation for the config sounds helpful as well.

but that is a huge task since the APIs are so different

Yeah, I don't know enough about writing a plugin vs. annotation processors. But I think it could be worth the effort.

[radcortez]

A plugin as in a Maven / Gradle plugin? That would work too.

Yes.

Although I think annotation processors are a little better now since it is more well-integrated (in the IDE for example), and we are already using a bunch of annotations. The idea of generating documentation for the config sounds helpful as well.

The problem with the annotation processor is that you work with sources and not classes, so you can't use reflection to introspect the mappings. You need to use the processor API to retrieve what you need. A possible trick would be to have a separate module to compile the mappings and then have it as a dependency on the processor to use the current code in config.

I'll try to come up with something.