Add deduplication step to sync

[stopfstedt]

Currently, the sync process does not handle duplicate user accounts (duplicate by campus id) at the source.
When the sync runs, all users attached to a given sync source - learner group, cohort, instructor group etc. - are processed individually. this leads to a problem of repeat suspension/reactivation of enrollments for users with duplicate accounts on the Ilios side, where at least one account is enabled and at least one account is disabled.

To fix this, I'm proposing to fix this by adding a user de-duplication step the sync process with the following rules:

• if all dupe accounts are active, then only the most recent one should be processed.
• if all dupe accounts are inactive, then only the most recent one should be processed.
• if there is a mix of active and inactive dupe accounts, then only the most recent active account should be processed.

"most recent" - most recently created, has the max user id value amongst its duplicates.

Edit: tightened up language on the first rule. [ST 2025/01/09]

[michaelchadwick]

@stopfstedt If "most recently created" is the heuristic used, won't a new account made in 2024 that's actually related to an account with 2023 data be used with its "blank" data, obscuring the 2023 data? That case feels like a data entry issue (when new account is created, make sure identical account doesn't already exist), so not sure what post-processing can feasibly do about it.

[stopfstedt]

@stopfstedt If "most recently created" is the heuristic used, won't a new account made in 2024 that's actually related to an account with 2023 data be used with its "blank" data, obscuring the 2023 data? That case feels like a data entry issue (when new account is created, make sure identical account doesn't already exist), so not sure what post-processing can feasibly do about it.

picking most recent here is essentially a tie breaker, we could also pick earliest with the same result. deduplication happens on campus id, we just have to pick one record. from a Moodle perspective, the Ilios user record is ephemeral, we don't store any linkage to concrete Ilios user records on the Moodle side.
the important decision point in all of this is to let the active user record win out over its inactive duplicate(s).
so far, this seem tight enough to me.
if this proposed solution turns out to be incomplete or flawed then i'll amend the ticket.

[jrjohnson]

I'd lean towards erroring in this case and fixing the account (when there are two that are active). For historical data reasons we allow duplicate campus IDs, but it's not my favorite thing. If we error we could then clean up the data and ensure there is only one active account for a campus ID.

[ctam]

I probably don't know enough about Ilios's code now to make any comment. But is there any better way than using the 'max user id value' to determine if an account is 'most recent'? Is there a creation date time field we could use? If not, can that be added? It feels a little code smell with this approach.

"most recent" - most recently created, has the max user id value amongst its duplicates.