Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix number of events stored in EventStore and adjust aggregation methods #46

Merged
merged 3 commits into from
Jan 19, 2024

Conversation

pitagoras3
Copy link
Collaborator

Before:

Earlier, each and every event occurring during migration was stored in EventStore. It was not problematic for small migrations, as most types of events are emitted only once per collection migration.

But there are two events - DumpUpdateEvent and RestoreUpdateEvent which are emitted every single time that mongodump/mongorestore writes on stdout. It occurs a lot of times (for long-taking migrations of large databases).

As number of events in EventStore grew up more and more, it was taking much more time to aggregate all the events, and calculate current state of migration.

After:

The most important change in this PR is the fact, that event of given type, can be stored only once in EventStore, and if there was earlier an event of a given type, it will be overriden.

fun storeEvent(event: StateEvent) {
    val eventsForCollection = events.getOrDefault(event.sourceToDestination, ConcurrentHashMap())
    eventsForCollection[event.type] = event
    events[event.sourceToDestination] = eventsForCollection
}

Thanks to that, number of events will be reduced to max of number_of_migrated_collections * types_of_events, which will significantly improve aggregation time.

@pitagoras3 pitagoras3 merged commit 9cd3e4d into main Jan 19, 2024
3 checks passed
@pitagoras3 pitagoras3 deleted the fix-number-of-events-and-aggregation branch January 19, 2024 11:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants