How do I bind an AsyncLogger and an Appender into a separate LoggerContext for isolation?

[CrazyHZM]

My project was upgraded from log4j1 to log4j2.
Prerequisite: log4j1 and log4j2 use asynchronous logging, and log4j2 uses AsyncLogger.
In log4j1, each Logger and Appender can be isolated; each Appender has a different thread writing a different file, but in log4j2, no matter how many AsyncLoggers there are, they all share a LoggerContext. As a result, only one thread gets data from the Ringbuffer and writes the log file, the amount of logs written per unit time is affected, and the throughput does not increase, no matter how the parameters are adjusted.
Is there any way around this scenario?

[ppkarwasz]

In Log4j 2 there are two types of asynchronous components:

• Asynchronous loggers, that use a single LMAX RingBuffer per logger context.
• Asynchronous appenders, that use a separate blocking BlockingQueue per appender. Note that the appenders themselves are in the same logger context. Using a DisruptorBlockingQueue you can even use a ring buffer as underlying BlockingQueue implementation (additional dependencies required).

Therefore with async loggers you'll have a single async thread, with async appenders you'll have multiple async threads.

Remark: This seems like an XY problem. What is the ultimate goal you want to achieve? If a single thread is not able to handle the logging from the rest of the application, maybe you are logging too much?

Note that most logging performance benchmarks assume an unrealistic situation, where all an application does is producing log events.

[CrazyHZM]

Thank you for your reply.

I hope that under the same traffic pressure, after upgrading to log4j2, I can still output the same amount of log content.

As you said, in log4j1 with async append, each appender is a BlockingQueue, and it has its own consumer to handle the LogEvent. This ensures that in cases where there are multiple Appends within an application, we can have multiple threads handling each file write individually.
For example:
Logger1 -> Appender1 -> queue1 -> Consumer Thread1 -> write
Logger2 -> Appender2 -> queue2 -> Consumer Thread2 -> write

However, after upgrading to lo4j2, because all Asynclogggers share a LoggerContext, it means that they share a Ringbuffer. Similarly, there is only one consumer thread to handle log events and file writes. Per unit of time. It outputs a much smaller number of logs than log4j1 uses.

For example(This can only be done by one consuming thread):
Logger1 -> Ringbuffer1 -> Consumer Thread1 -> Appender1 -> write
Logger2 -> Ringbuffer1 -> Consumer Thread1 -> Appender2 -> write

By the way, in log4j2 I'm using RollingFileAppender +AsyncLogger.

If we can bind a different LoggerContext or Ringbuffer for the different Appender, we can achieve the same effect as the async appender. Or there are other ways to help me achieve my goals.

@ppkarwasz

[ppkarwasz]

@CrazyHZM,

For example(This can only be done by one consuming thread):
Appender1 -> Ringbuffer1 -> Consumer Thread1 -> write
Appender2 -> Ringbuffer1 -> Consumer Thread1 -> write

The way AsyncLogger works is different:

• AsyncLogger creates a log event and adds it to the ring buffer.
• The asynchronous thread calls each appender (Appender1, Appender2) in sequence.

The RingBuffer component precedes appenders in the logging pipeline.

If we can bind a different LoggerContext or Ringbuffer for the different Appender, we can achieve the same effect as the async appender. Or there are other ways to help me achieve my goals.

The Async appender is still available in Log4j 2. See the example configuration:

<Configuration xmlns="https://logging.apache.org/xml/ns"
               xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
               xsi:schemaLocation="
                   https://logging.apache.org/xml/ns
                   https://logging.apache.org/xml/ns/log4j-config-2.xsd">
  <Appenders>
    <File name="FILE"
          fileName="app.log">
      <JsonTemplateLayout/>
    </File>
    <Async name="ASYNC">
      <AppenderRef ref="FILE"/>
    </Async>
  </Appenders>
  <Loggers>
    <Root level="INFO">
      <AppenderRef ref="ASYNC"/>
    </Root>
  </Loggers>
</Configuration>

Note: if you are using the Async appender, you should not use AsyncLogger. Otherwise, each log event will have to traverse two asynchronous barriers.

The advantage of AsyncLogger over Async Appender is that the former can be garbage-free, while the latter always creates temporary log event objects. This doesn't mean that AsyncLogger is always better than Async Appender. Only profiling your application can tell.

[CrazyHZM]

My mistake. I changed the description.

• AsyncLogger creates a log event and adds it to the ring buffer.
• The asynchronous thread calls each appender (Appender1, Appender2) in sequence.

Yes, the problem with our current project is the limited consumption capacity of Consumer Thread1.
One more question: Is there a performance boost for log4j1 and log4j2 Async appenders?

Thanks for your answer. @ppkarwasz

[ppkarwasz]

One more question: Is there a performance boost for log4j1 and log4j2 Async appenders?

The Async appender in Log4j 2 has been enhanced with a configurable BlockingQueueFactory. See Blocking queue factories on our staging site. You can try DisruptorBlockingQueue and JCToolsBlockingQueue to have different performance characteristics

Performance has many aspects and is very sensitive to the logging characteristics of the application:

• in general async loggers and appenders reduce the latency of the calling thread, allowing it to finish earlier. However, the latency is worse than in the synchronous case if the queue is full.
• from my personal benchmarks, a single logging thread can fill up 50% or more of the writing speed of modern SSDs.
• if we look at the sustained logging rate, using sync or async loggers and sync or async appenders, doesn't make much difference.

[vy]

@CrazyHZM, Log4j 1 and Log4j 2 are technically two completely different implementations. I strongly advise you to go with the simplest possible Log4j 2 configuration (i.e., use synchronous loggers; no async. appender and no async. logging) and assess the performance. If you think performance is an issue, we have sufficient guidance in the Log4j performance documentation. If you still cannot pinpoint a particular performance issue, you can always consult us again. But right now... I have the feeling you're trying to solve a problem that doesn't exist yet. Hence, please go with the simplest setup, i.e., the defaults.