Upgrade to django-sodar-core v1.0

[mikkonie]

This one will be a bit more involved due to the large amount of breaking changes. Keeping a note of them here.

This requires first doing bihealth/irods-docker#16 and bihealth/sodar-docker-compose#34. (Edit: not really, some client issues require these to be developed simultaneously)

Tasks

• Upgrade Python requirements
• Upgrade Python versions in CI
• Upgrade Postgres to v16
• Fix breaking Django 4.2 issues
• Update base test class names where used
• Update CORE_API_* references
• Update invite URL names in tests
• Fix DeleteView classes breaking with Django v4.x
• Ensure everything works with renamed timeline app models
• Add OIDC setup
• Update AppSettingAPI app_name arguments into plugin_name N/A
• Update get_object_link() return data to PluginObjectLink
• Update search() return data to PluginSearchResult
• Update ProjectModifyPluginMixin method signatures for plugin_name N/A
• Update AUTH_LDAP*_USER_SEARCH_BASE Django settings
• Set ATOMIC_REQUESTS to True and ensure all works (1)
• Update email sending for settings link usage where applicable N/A
• Upgrade to SODAR Core v1.0.2 final
• Update broker_connection_retry setting in Celery setup (see comments)
• Install versioneer v0.29
• Update install scripts
• Update release cleanup template
• Update iRODS version in CI once 4.3.x package is available
• Ensure iRODS compability in CI
• Ensure DAL works with Whitenoise (see Select2 dependency in DAL v3.9.5 breaks collectstatic with Whitenoise in production sodar-core#1224) (2)

Documentation Updates

• Update dependencies (Postgres, iRODS etc)
• Update SODAR Core API docs (versioning, pagination)
• Timeline API documentation
• Admin alerts email sending Documented in SODAR Core
• User additonal email management
• New user settings
• Remote sync controls in project form
• checkusers management command
• OIDC support (mention status of iRODS support) Add OIDC documentation once iRODS is supported
• Remove SAML mentions
• Timeline list UI
• Update all SODAR Core based screenshots where UI has meaningfully changed

Notes

1. Postponing this for now, as this may create issues with some other things.
2. Best way to 100% make sure this works is to deploy in staging

[mikkonie]

Two tests currently failing in local dev:

======================================================================
FAIL: test_post (isatemplates.tests.test_views.TestISATemplateDeleteView.test_post)
Test POST
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/mikkopen/code/sodar/isatemplates/tests/test_views.py", line 827, in test_post
    self.assertEqual(CookiecutterISATemplate.objects.count(), 0)
AssertionError: 1 != 0

======================================================================
FAIL: test_delete (samplesheets.tests.test_views_taskflow.TestIrodsAccessTicketDeleteView.test_delete)
Test deleting iRODS access ticket
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/mikkopen/code/sodar/samplesheets/tests/test_views_taskflow.py", line 1056, in test_delete
    self.assertIsNone(self.get_irods_ticket(self.ticket))
AssertionError: {<irods.column.Column 2200 TICKET_ID>: 18695, <irods.column.Column 2201 TICKET_STRING>: 'q657xxx3i2x2b8vj', <irods.column.Column 2202 TICKET_TYPE>: 'read', <irods.column.Column 2203 TICKET_USER_ID>: 10002, <irods.column.Column 2204 TICKET_OBJECT_ID>: 18694, <irods.column.Column 2205 TICKET_OBJECT_TYPE>: 'collection', <irods.column.Column 2206 TICKET_USES_LIMIT>: 0, <irods.column.Column 2207 TICKET_USES_COUNT>: 0, <irods.column.Column 2208 TICKET_EXPIRY_TS>: None, <irods.column.Column 2211 TICKET_WRITE_FILE_COUNT>: 0, <irods.column.Column 2212 TICKET_WRITE_FILE_LIMIT>: 0, <irods.column.Column 2213 TICKET_WRITE_BYTE_COUNT>: 0, <irods.column.Column 2214 TICKET_WRITE_BYTE_LIMIT>: 0, <irods.column.Column 2228 TICKET_COLL_NAME>: '/sodarZone/projects/b6/b61a4de5-ea86-4529-b4c6-638b9ec7e334/sample_data/study_745897d3-abca-4598-a6fd-00c66e67f2b5/assay_339bc26f-65b9-49be-8a3f-082b6006f3a6/coll'} is not None

NOTE: At the time of writing all iRODS tests fail in CI, as I'm still using our iRODS 4.2 image which is incompatible with postgres v16.

[mikkonie]

All tests now working locally with iRODS v4.3.1.

[mikkonie]

I fixed iRODS-related tests in CI by temporarily adding a separate PostgreSQL v11 server to be used by iRODS. This can be removed once I have a stable iRODS 4.3 image and switch to using that in CI.

[mikkonie]

The broker_connection_retry configuration no longer works in Celery and should be updated:

[2024-09-09 13:58:19,374: WARNING/MainProcess] /home/xxx/.virtualenvs/sodar1.0/lib/python3.11/site-packages/celery/worker/consumer/consumer.py:507: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
whether broker connection retries are made during startup in Celery 6.0 and above.
If you wish to retain the existing behavior for retrying connections on startup,
you should set broker_connection_retry_on_startup to True.

[mikkonie]

Done and merged into v1.0. A couple of issues remain, I will split them into separate tasks as applicable.