"ERROR StatusLogger Unrecognized format specifier [d]" after i update the log4j version from 2.20.0 to 2.21.0

[mpwong852]

Before I only use below "dependencies"

[INFO] +- org.apache.logging.log4j:log4j-core:jar:2.20.0:compile
[INFO] \- org.apache.logging.log4j:log4j-1.2-api:jar:2.20.0:compile
[INFO]    \- org.apache.logging.log4j:log4j-api:jar:2.20.0:compile

It doesn't show me "ERROR StatusLogger Unrecognized format specifier [d]"

After I update to "2.21.0", it start show me "ERROR StatusLogger Unrecognized format specifier [d]"

ERROR StatusLogger Unrecognized format specifier [d]
ERROR StatusLogger Unrecognized conversion specifier [d] starting at position 16 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [thread]
ERROR StatusLogger Unrecognized conversion specifier [thread] starting at position 25 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [level]
ERROR StatusLogger Unrecognized conversion specifier [level] starting at position 35 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [logger]
ERROR StatusLogger Unrecognized conversion specifier [logger] starting at position 47 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [msg]
ERROR StatusLogger Unrecognized conversion specifier [msg] starting at position 54 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [n]
ERROR StatusLogger Unrecognized conversion specifier [n] starting at position 56 in conversion pattern.

To fix this, I need to use plugin to merges Log4j2Plugins.dat plugin caches from all the jars containing Log4j 2.x Core components.

ref : https://logging.apache.org/log4j/transform/latest/#log4j-plugin-cache-transformer

I have already apply this solution on 2.21.0 to solve my problems.

The question is why I need to use plugin to fix this issue ? What is the change in 2.21.0 causing Log4j2Plugins.dat have problems ? Any solution for me not using that plugin ?

[mpwong852]

One more thing i want to ask, is below change is fixing the issue that I mention here ? If yes then I will wait for that change.

https://issues.apache.org/jira/browse/LOG4J2-2621

[ppkarwasz]

In version 3.x the problem you are currently experiencing will only slightly change: instead of properly merging META-INF/org/apache/logging/log4j/core/config/plugins/Log4j2Plugins.dat files using an ad-hoc Maven Shade Resource Transformer, you'll need to properly merge META-INF/services/* files using a more standard ServicesResourceTransformer.

[ppkarwasz]

The question is why I need to use plugin to fix this issue ? What is the change in 2.21.0 causing Log4j2Plugins.dat have problems ? Any solution for me not using that plugin ?

TL;DR: See How do I create a single-JAR application containing Log4j Core?

The Maven Shade Plugin creates a single-JAR application by copying all the files from the dependency JARs and putting them into a single JAR. Each time two or more JARs have a file with the same name, only one is kept and the other files are discarded.
This usually breaks the application in some way.

In order to deal with file name conflicts for each conflict you must configure an appropriate Resource Transformer to merge multiple files into one. The Log4j Plugin Cache Transformer is the appropriate resource transformer for META-INF/org/apache/logging/log4j/core/config/plugins/Log4j2Plugins.dat files.

In the next release I added a FAQ entry on how to create single-JAR applications. You can see a preview version on our staging site.

Why did your application work with version 2.20.0? My guess is that the Maven Shade Plugin shades the JARs in some random, but deterministic order. When you used version 2.20.0, Maven Shade kept the Log4j2Plugins.dat file from log4j-core and dropped the one from log4j-1.2-api. Now it drops the Log4j2Plugins.dat file from log4j-core and keeps the one from log4j-1.2-api.

In both situation your application was somehow crippled, but since the plugins in log4j-1.2-api mostly concentrate on the support of the Log4j 1 configuration file format, you didn't notice that they were missing.

[mpwong852]

The question is why I need to use plugin to fix this issue ? What is the change in 2.21.0 causing Log4j2Plugins.dat have problems ? Any solution for me not using that plugin ?

TL;DR: See How do I create a single-JAR application containing Log4j Core?

The Maven Shade Plugin creates a single-JAR application by copying all the files from the dependency JARs and putting them into a single JAR. Each time two or more JARs have a file with the same name, only one is kept and the other files are discarded. This usually breaks the application in some way.

In order to deal with file name conflicts for each conflict you must configure an appropriate Resource Transformer to merge multiple files into one. The Log4j Plugin Cache Transformer is the appropriate resource transformer for META-INF/org/apache/logging/log4j/core/config/plugins/Log4j2Plugins.dat files.

In the next release I added a FAQ entry on how to create single-JAR applications. You can see a preview version on our staging site.

Why did your application work with version 2.20.0? My guess is that the Maven Shade Plugin shades the JARs in some random, but deterministic order. When you used version 2.20.0, Maven Shade kept the Log4j2Plugins.dat file from log4j-core and dropped the one from log4j-1.2-api. Now it drops the Log4j2Plugins.dat file from log4j-core and keeps the one from log4j-1.2-api.

In both situation your application was somehow crippled, but since the plugins in log4j-1.2-api mostly concentrate on the support of the Log4j 1 configuration file format, you didn't notice that they were missing.

Hi I have a new finding, it seems there are issue between the "log4j" & "maven-assembly-plugin".

link: https://github.com/mpwong852/log4j-issue

When I update to "maven-assembly-plugin" 3.7.0 + log4j 2.23.1, it will show me uncorrect log.
I need to downgrade "maven-assembly-plugin" or log4j version to fix this issue.

Would you help me to have a look to see that issue is caused by "log4j" or "maven-assembly-plugin" ?

[ppkarwasz]

Would you help me to have a look to see that issue is caused by "log4j" or "maven-assembly-plugin" ?

With maven-assembly-plugin it is not possible to make a functional application containing multiple Log4j modules. See its documentation:

Note that jar-with-dependencies provides only basic support for uber-jars. For more control, use the Maven Shade Plugin.

You need to replace it with another Maven plugin.

Easy way

Use the Spring "Boot Maven Plugin. You don't need to know anything about your dependencies, since the JARs will be simply embedded into your application's JAR.

Click to see an example of Spring Boot Maven Plugin configuration

<plugin>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-maven-plugin</artifactId>
  <version>3.3.2</version>
  <executions>
    <execution>
      <goals>
        <goal>repackage</goal>
      </goals>
      <configuration>
        <mainClass>org.example.Main</mainClass>
      </configuration>
    </execution>
  </executions>
</plugin>

Hard way

If you use Maven Shade Plugin you get a warning for each overlapping resource:

[WARNING] Discovered module-info.class. Shading will break its strong encapsulation.
[WARNING] Discovered module-info.class. Shading will break its strong encapsulation.
[WARNING] Discovered module-info.class. Shading will break its strong encapsulation.
[WARNING] log4j-1.2-api-2.23.1.jar, log4j-api-2.23.1.jar, log4j-core-2.23.1.jar define 3 overlapping resources: 
[WARNING]   - META-INF/DEPENDENCIES
[WARNING]   - META-INF/LICENSE
[WARNING]   - META-INF/NOTICE
[WARNING] log4j-1.2-api-2.23.1.jar, log4j-api-2.23.1.jar, log4j-core-2.23.1.jar, log4j-issue-test.jar define 1 overlapping resource: 
[WARNING]   - META-INF/MANIFEST.MF
[WARNING] log4j-1.2-api-2.23.1.jar, log4j-core-2.23.1.jar define 1 overlapping resource: 
[WARNING]   - META-INF/org/apache/logging/log4j/core/config/plugins/Log4j2Plugins.dat
[WARNING] maven-shade-plugin has detected that some files are
[WARNING] present in two or more JARs. When this happens, only one
[WARNING] single version of the file is copied to the uber jar.
[WARNING] Usually this is not harmful and you can skip these warnings,
[WARNING] otherwise try to manually exclude artifacts based on
[WARNING] mvn dependency:tree -Ddetail=true and the above output.
[WARNING] See https://maven.apache.org/plugins/maven-shade-plugin/

For each of those overlapping files, you need to learn what they do and how to merge them. Below you'll find an example configuration that solves most conflicts for applications that include Log4j artifacts.

I am not a lawyer, so I don't know how to deal with the embedded licenses and notices.

Click of a complete example of Maven Shade Plugin configuration

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-shade-plugin</artifactId>
  <version>3.6.0</version>
  <dependencies>
    <dependency>
      <groupId>org.apache.logging.log4j</groupId>
      <artifactId>log4j-transform-maven-shade-plugin-extensions</artifactId>
      <version>0.1.0</version>
    </dependency>
  </dependencies>
  <executions>
    <execution>
      <id>shade-jar-with-dependencies</id>
      <goals>
        <goal>shade</goal>
      </goals>
      <configuration>
        <filters>
          <filter>
            <artifact>*:*</artifact>
            <excludes>
              <exclude>module-info.class</exclude>
              <exclude>META-INF/DEPENDENCIES</exclude>
              <exclude>META-INF/MANIFEST.MF</exclude>
            </excludes>
          </filter>
        </filters>
        <transformers>
          <transformer implementation="org.apache.logging.log4j.maven.plugins.shade.transformer.Log4j2PluginCacheFileTransformer"/>
          <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
          <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
            <mainClass>org.example.Main</mainClass>
            <manifestEntries>
              <Multi-Release>true</Multi-Release>
            </manifestEntries>
          </transformer>
        </transformers>
      </configuration>
    </execution>
  </executions>
</plugin>