Exclusive incomplete events republication #871
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I've been thinking about republishing incomplete events on restart. there are two things i think we could improve
a) the incomplete events should run after the spring boot health check is green, otherwise a sufficiently large number of incomplete events could cause the program to fail to start in time on Kubernetes or Cloud Foundry, and then we'd get even more missed events, and the cascading dogpile would be horrendous.
b) we could provide a cluster-wide locking and exclusive implementation, so that when, say, five instances of the same service restart, only one actually processes the incomplete events. Spring Cloud introduced a distributed locking mechanism 8 years ago, and it moved into Spring Integration sometime before the pandemic. So, ive introduced support for that in this branch. This of course starts to muddy the code, so I extracted out the default behavior (resubmit all, assuming the property is set) and this exclusive locking behavior (resubmit all, assuming property is set and the lock name is specified) into two separate classes, with appropriate auto configuration so that one gets chosen if the other doesn't.
This PR establishes a new interface for things that will reprocess outstanding events on restart. I've moved them to run at the same time as
ApplicationRunner
does, a bit later in the lifecycle. There are now two implementations. The default behavior is as it has always been: to run all outstanding incomplete events. There's also a new implementation guarded by a new property (establishing a lock name) and a check for the correct classes on the classpath that uses Spring Integration's distributed lock support. Obviously, Spring Integration doesn't offer locks for every persistence tier that Spring Modulith supports, so perhaps we could expand Spring Integration's support or also offer schedlock support as an alternative, additional autoconfiguration.