Revert "Revert "Use create_or_find_by to reduce activity status deadlocks""

[jorg-vr]

Reverts #4670

This reintroduces all changes made in #4547 and thus closes #4479

The only change compared to the original pr is that I added reconnect: true to the production DB settings.
I hope that because a roll backed query broke the connection, just reconnecting and continuing should result in an error free flow. (But we'll have to test it to know for sure)

[bmesuere]

I would put this on hold until after the exams.

[bmesuere]

For future reference:

• some submissions failed with internal error, but worked fine when rejudging
• we could not reproduce the problem on naos
• the error occurred "at random", but more with higher traffic. We could trigger it by doing a mass rejudge from the rails console
• the error was triggered by setting the submission status to "running". Upon inspection, it seems that something goes wrong when creating an activity_status which causes a rollback, which causes the mysql connection to crash/drop?

[jorg-vr]

i have been investigating this more, some conclusions:
These rollbacks happen almost always. This pr runs a create before a find, the create is often only required when the status is set to running by the first submission.
=> only a very select subset of these rollbacks causes an issue, DB connection issue
The requests are also always small and fast.
We do not match any of the common reasons why database connection is lost: https://dev.mysql.com/doc/refman/8.0/en/gone-away.html
If I remember correctly we only got an error on the first db request after the rollback.

A potential 'solution' to accept that the connection might fail sometimes, but force an immediate reconnect.
This can be achieved by simply setting 'reconnect: true' in the database yaml
This 'solution' might not last long, as the option has recently been deprecated by mysql: brianmario/mysql2#1322 But it is still unclear how the mysql2 gem will resolve this

[bmesuere]

I don't like this fix. It is a deprecated workaround of a workaround.

In the original code, we do a single retry before raising an error. What if we try more?