How to minimize the time between the scheduled time and the actual execution time?

[vking34]

My problem

When the producer schedule a lot of tasks, there are some tasks that are not executed in the scheduled time while using critical priority.

For example:

2021-02-22 08:47:33.072 ERROR 1 --- [QueueConsumer-5] c.c.a.a.bots.workers.FlashBidBotWorker   : 45835 - error delay short fb: 1613982642301
...

This task should be executed on 1613982642301 (2021-02-22T08:30:41.954) but actually, it is executed on 2021-02-22T08:47:33.072. It was later for about 16 minutes. That is so painful.

My situation

I set the STRICT priority mode:

@Component
public class RqueueConfiguration {

    @Bean
    public SimpleRqueueListenerContainerFactory simpleRqueueListenerContainerFactory() {
        SimpleRqueueListenerContainerFactory factory = new SimpleRqueueListenerContainerFactory();

        RedisStandaloneConfiguration redisConfig = new RedisStandaloneConfiguration(System.getenv("REDIS_HOST"), Integer.parseInt(System.getenv("REDIS_PORT")));
        redisConfig.setDatabase(Integer.parseInt(System.getenv("REDIS_JOB_DB")));
        LettuceConnectionFactory redisConnectionFactory = new LettuceConnectionFactory(redisConfig);
        redisConnectionFactory.afterPropertiesSet();
        factory.setRedisConnectionFactory(redisConnectionFactory);
        factory.setPriorityMode(PriorityMode.STRICT);

        return factory;
    }
}

I enqueue tasks with the critical priority

bot.setNextExecutionTime(System.currentTimeMillis() + delayForNextPhase);
rqueueMessageSender.enqueueInWithPriority(SHORT_FLASH_BID_BOT_QUEUE, JobPriority.CRITICAL, bot, delayForNextPhase);

And here one of my RqueueListeners:

@RqueueListener(
            value = SHORT_FLASH_BID_BOT_QUEUE,
            numRetries = "0",
            concurrency = "9-11",
            priority = "critical=12,high=8,medium=4,low=1",
            active = "${cz-auction-bot.worker}")
private void shortFlashBid(Bot bot) {
...
}

How can I make sure the tasks executed on time?
Thank u so much.

[sonus21]

What happens if you increase concurrency to higher number? From my understanding you're enqueuing large number of jobs and your listeners takes longer time to process.

can you increase concurrency to 20 or 30 and verify?

[sonus21]

• What do you see in Scheduled tabs?
• Can you inspect this critical queue => Click on Queue link => Click on Scheduled, please report delay, if you see negative number?
  Report Time left from this window

There was some issue in sonatype sync, it did not sync two of the last versions to Maven central. It's uploaded here https://repo1.maven.org/maven2/com/github/sonus21

[Dbasdeo]

I noticed the same problem as well. When load testing every thing works great until we hit about 115 RPS. At this point messages start to become more and more delayed from when they should be scheduled, going all the way up until the hundreds of seconds.

I've tried upping the concurrency and the batch size but neither seems to make a difference.

sh-4.2# curl -X GET 'http://localhost:4352/monitoring/metrics' | grep "scheduler-queue"
processing_queue_size{application="xxx",key="__rq::p-queue::{scheduler-queue}",queue="scheduler-queue",} 69.0
delayed_queue_size{application="xxx",key="__rq::d-queue::{scheduler-queue}",queue="scheduler-queue",} 225.0
queue_size{application="xxx",key="__rq::queue::{scheduler-queue}",queue="scheduler-queue",} 12303.0

[Dbasdeo]

[sonus21]

@Dbasdeo you've provided three metrics here

processing queue size: 69 means 69 threads are likely running

delayed_queue_size: 225 means 225 messages are pending in scheduled queue, these messages should be moved to scheduler-queue as soon as message reaches their time.

queue_size: 12303 means there're 12303 pending messages in queue, this is happening since your worker is taking longer time to process messages, only way to solve this issue is to scale your listeners i.e. increase listener servers count.

There has been some improvement in this area, check version 2.8.0 and see if that helps in anyway.

[Dbasdeo]

Was able to resolve this with a couple more tweaks to my code. Thank you for the help.