[S3 realibility] Redeliver event dead letters to the queue

[hungphan227]

Why

When one of S3 is down for a long time, blob events would be stored in event dead letters table. James already has a task to redeliver dead letters back to the queue.

Unfortunately, EventDeadLettersRedeliverService (single instance) only has one EventDeadLetters instance -> only one EventSerializer instance (currently it is MailboxEventSerializer). Therefore, it does not work with blob event dead letters.

Refactoring code is needed to apply EventDeadLettersRedeliverService for blob event dead letters

How

• Move EventSerializer from CassandraEventDeadLettersDAO to CassandraEventDeadLetters
• Update retrieveFailedEvent method in CassandraEventDeadLettersDAO to return String instead of Event
• Create method getEventSerializer in EventBus interface
• Remove EventDeadLetters attribute in EventDeadLettersRedeliverService
• Create new interface EventDeadLettersService which contains two methods:

Mono<Event> failedEvent(Group registeredGroup, InsertionId failDeliveredInsertionId, EventSerializer serializer);
Mono<Void> remove(Group registeredGroup, InsertionId failDeliveredInsertionId, EventSerializer serializer);

• In EventDeadLettersRedeliverService, get events by using EventDeadLettersService

DoD

Integration tests

[hungphan227]

welcome any thoughts on this

[hungphan227]

@chibenwa if you have other thoughts please comment

[quantranhong1999]

Sounds ok to me

[quantranhong1999]

Maybe make this a James ticket as it would benefit EventDeadLetters to reprocess JMAP events with JmapEventSerializer too (rather than only MailboxEventSerializer)?

[chibenwa]

I think this would be the simplest indeed:

• Create method eventSerializer in EventBus interface
• Add a multibinder with all EventBusses
• EventDeadLetter reprocess service would take that set of eventbus. When reprocessing:
  • List all event bus, foreach
  • Try to deserialize the event
  • if deserialize then reprocess the event
  • if not deserialized then try next event bus.