Problem: Jobs and microservices appear out of order due to MCPClient forking model

[carlosmcgregor]

Expected behaviour

Items appear in sequential order.

Current behaviour

Items appear out of order. Sometimes, the currentstep=1 appears below a step that has already been completed.

e.g.

Steps to reproduce

1. Start an ingest
2. Let it run / approve the necessary items

Cause

Since the addition of the MCPClient forking model, an unintended consequence has been that items are no longer displayed in sequential order. We believed that this was caused by the following function:

https://github.com/artefactual/archivematica/blob/bade7adeb0ffb4422b6fb289fe13555d0051fca3/src/MCPClient/lib/client/job.py#L62-L69

This function had the unintended consequence that jobs that just finished would get a start_time later than the currentstep=1 job given that the latter would have a timestamp given at object creation and the former would have an updated timestamp when processed.

Disabling this function only solved the issue of the currentstep=1 job appearing below completed steps.

Your environment (version of Archivematica, operating system, other relevant details)
Archivematica: 1.16.0
OS: Ubuntu 22.04

---

For Artefactual use:

Before you close this issue, you must check off the following:

• All pull requests related to this issue are properly linked
• All pull requests related to this issue have been merged
• A testing plan for this issue has been implemented and passed (testing plan information should be included in the issue body or comments)
• Documentation regarding this issue has been written and merged (if applicable)
• Details about this issue have been added to the release notes (if applicable)

[shij13]

Noting also that the incorrect ordering is not consistent. E.g., from the Artefactual sandbox, the "Assign file UUIDs and checksums" microservice is listed in different places in the processing workflow

Since start and finish times for each task/job are recorded only to the second (rather than the millisecond), we are unable to use the timestamps in the database to confirm if jobs and microservices are occurring in the expected order either.

This issue makes it difficult for users to confirm that workflows are happening consistently and with the full information required to complete the task, e.g. in the "Normalize" microservice, "Validate preservation derivatives" job being listed after the "Normalization report" job, or the "Identify file format" job being listed after the "Normalize" decision point

[carlosmcgregor]

After exploring this issue further with @shij13 , we discovered that the main issue has to do with the migration from MySQL 5 to MySQL 8. When it comes to datetime columns, MySQL 5 truncated microseconds, while MySQL 8 rounds up microseconds (unless fractional precision is allowed). This means that jobs that occurred under close succession could end up appearing in disorder as the createdTime would round up to be the same, but the createdTimeDec would push older entries as seeming more recent.

The script provided in your documentation does not address this change in how MySQL 8 handles datetime precision, so datetime objects that were migrated had a precision of 0 (leading to the rounding up of datetime values).

Could you please update your migration script to address this issue? At least when it comes to the Jobs table, freshly installed instances of Archivematica have a column 'createdTime' datetime(6) NOT NULL, which indicates a fractional precision of 6.

Regardless, this could still be a separate issue to the bulk timestamp update which we believe could place the currentstep=1 job as being out of order.