Feature/3959 bg monitoring fixed

[richard-jones]

• Issue: https://github.com/DOAJ/doajPM/issues/3959

Replaces PR: #2419

update bg monitoring settings for mission critical monitoring changes

Applies tighter monitoring constraints on the background jobs, as per this spreadsheet: https://docs.google.com/spreadsheets/d/1XWq-7R9hZmZsBA27eOgs9cO-SfJnLEE0dW50DVtKhpI/edit?gid=543591434#gid=543591434

This PR...

• has scripts to run
• has migrations to run
• adds new infrastructure
• changes the CI pipeline
• affects the public site
• affects the editorial area
• affects the publisher area
• affects the monitoring

Developer Checklist

Developers should review and confirm each of these items before requesting review

• Code meets acceptance criteria from issue
• Unit tests are written and all pass
• User Test Scripts (if required) are written and have been run through
• Project's coding standards are met
  • No deprecated methods are used
  • No magic strings/numbers - all strings are in constants or messages files
  • ES queries are wrapped in a Query object rather than inlined in the code
  • Where possible our common library functions have been used (e.g. dates manipulated via dates)
  • Cleaned up commented out code, etc
  • Urls are constructed with url_for not hard-coded
• Code documentation and related non-code documentation has all been updated
  • Core model documentation has been added to if needed: https://docs.google.com/spreadsheets/d/1lun2S9vwGbyfy3WjIjgXBm05D-3wWDZ4bp8xiIYfImM/edit
  • Events and consumers documentation has been added if needed: https://docs.google.com/spreadsheets/d/1oIeG5vg-blm2MZCE-7YhwulUlSz6TOUeY8jAftdP9JE/edit
• Migation has been created and tested
• There is a recent merge from develop

Reviewer Checklist

Reviewers should review and confirm each of these items before approval
If there are multiple reviewers, this section should be duplicated for each reviewer

• Code meets acceptance criteria from issue
• Unit tests are written and all pass
• User Test Scripts (if required) are written and have been run through
• Project's coding standards are met
  • No deprecated methods are used
  • No magic strings/numbers - all strings are in constants or messages files
  • ES queries are wrapped in a Query object rather than inlined in the code
  • Where possible our common library functions have been used (e.g. dates manipulated via dates)
  • Cleaned up commented out code, etc
  • Urls are constructed with url_for not hard-coded
• Code documentation and related non-code documentation has all been updated
  • Core model documentation has been added to if needed: https://docs.google.com/spreadsheets/d/1lun2S9vwGbyfy3WjIjgXBm05D-3wWDZ4bp8xiIYfImM/edit
  • Events and consumers documentation has been added if needed: https://docs.google.com/spreadsheets/d/1oIeG5vg-blm2MZCE-7YhwulUlSz6TOUeY8jAftdP9JE/edit
• Migation has been created and tested
• There is a recent merge from develop

Testing

This tightens the monitoring on a number of bg jobs, so would be good to validate that this won't cause any major surge in unnecessary reporting.

Probably need to send it live and then adjust.

Deployment

Configuration changes

Adds new values in:

• BG_MONITOR_QUEUED_CONFIG
• BG_MONITOR_ERRORS_CONFIG
• BG_MONITOR_DEFAULT_CONFIG

Monitoring

This may trigger new types of "unstable" alerts in the background jobs reporting, so we should ensure that those alerts come through

[RK206]

Test cases failing from local. Can you please check. Are there any configuration changes required to test?

[RK206]

Failed tests on my local. Not sure if these are specific to my local. Fixed import issue on my local in portality/bll/services/background_task_status.py before testing.

doajtest/unit/test_background_task_status.py:178 (TestBackgroundTaskStatus.test_create_background_status__error_in_period_not_found)
self = <doajtest.unit.test_background_task_status.TestBackgroundTaskStatus testMethod=test_create_background_status__error_in_period_not_found>

    @apply_test_case_config(bg_monitor_errors_config__a)
    def test_create_background_status__error_in_period_not_found(self):
        save_mock_bgjob(JournalCSVBackgroundTask.__action__,
                        status=constants.BGJOB_STATUS_ERROR,
                        created_before_sec=1000000000)
    
        status_dict = background_task_status.create_background_status()
    
        journal_csv_dict = status_dict['queues']['scheduled_short']['errors'].get(JournalCSVBackgroundTask.__action__, {})
    
>       assert is_stable(status_dict['status'])
E       AssertionError: assert False
E        +  where False = is_stable('unstable')

test_background_task_status.py:189: AssertionError

doajtest/unit/test_background_task_status.py:106 (TestBackgroundTaskStatus.test_create_background_status__invalid_last_completed__events_queue)
self = <doajtest.unit.test_background_task_status.TestBackgroundTaskStatus testMethod=test_create_background_status__invalid_last_completed__events_queue>

    @apply_test_case_config(bg_monitor_last_completed__now)
    def test_create_background_status__invalid_last_completed__events_queue(self):
        save_mock_bgjob(JournalCSVBackgroundTask.__action__,
                        queue_id=constants.BGJOB_QUEUE_ID_EVENTS,
                        status=constants.BGJOB_STATUS_COMPLETE, )
    
        status_dict = background_task_status.create_background_status()
    
        assert not is_stable(status_dict['status'])
>       self.assert_unstable_dict(status_dict['queues'].get('events', {}))

test_background_task_status.py:116: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

val = {'err_msgs': [], 'errors': {'application_autochecks': {'err_msgs': [], 'in_monitoring_period': 0, 'status': 'stable', ...table'}, 'ingest_articles': {'err_msgs': [], 'last_run': None, 'last_run_status': None, 'status': 'stable'}, ...}, ...}

    @staticmethod
    def assert_unstable_dict(val):
>       assert not is_stable(val.get('status'))
E       AssertionError: assert not True
E        +  where True = is_stable('stable')
E        +    where 'stable' = <built-in method get of dict object at 0x165c907c0>('status')
E        +      where <built-in method get of dict object at 0x165c907c0> = {'err_msgs': [], 'errors': {'application_autochecks': {'err_msgs': [], 'in_monitoring_period': 0, 'status': 'stable', 'total': 0}, 'article_bulk_create': {'err_msgs': [], 'in_monitoring_period': 0, 'status': 'stable', 'total': 0}, 'article_bulk_delete': {'err_msgs': [], 'in_monitoring_period': 0, 'status': 'stable', 'total': 0}, 'ingest_articles': {'err_msgs': [], 'in_monitoring_period': 0, 'status': 'stable', 'total': 0}, ...}, 'last_completed_job': '2024-11-04T04:46:47Z', 'last_run_successful': {'application_autochecks': {'err_msgs': [], 'last_run': None, 'last_run_status': None, 'status': 'stable'}, 'article_bulk_create': {'err_msgs': [], 'last_run': None, 'last_run_status': None, 'status': 'stable'}, 'article_bulk_delete': {'err_msgs': [], 'last_run': None, 'last_run_status': None, 'status': 'stable'}, 'ingest_articles': {'err_msgs': [], 'last_run': None, 'last_run_status': None, 'status': 'stable'}, ...}, ...}.get

test_background_task_status.py:104: AssertionError

doajtest/unit/test_background_task_status.py:232 (TestBackgroundTaskStatus.test_create_background_status__queued_valid_oldest)
self = <doajtest.unit.test_background_task_status.TestBackgroundTaskStatus testMethod=test_create_background_status__queued_valid_oldest>

    @apply_test_case_config(bg_monitor_queued_config__a)
    def test_create_background_status__queued_valid_oldest(self):
        save_mock_bgjob(JournalCSVBackgroundTask.__action__,
                        status=constants.BGJOB_STATUS_QUEUED, )
    
        status_dict = background_task_status.create_background_status()
        print(json.dumps(status_dict, indent=4))
    
        journal_csv_dict = status_dict['queues']['scheduled_short']['queued'].get(JournalCSVBackgroundTask.__action__, {})
    
>       assert is_stable(status_dict['status'])
E       AssertionError: assert False
E        +  where False = is_stable('unstable')

test_background_task_status.py:243: AssertionError

doajtest/unit/test_background_task_status.py:205 (TestBackgroundTaskStatus.test_create_background_status__queued_valid_total)
self = <doajtest.unit.test_background_task_status.TestBackgroundTaskStatus testMethod=test_create_background_status__queued_valid_total>

    @apply_test_case_config(bg_monitor_queued_config__zero_total)
    def test_create_background_status__queued_valid_total(self):
        save_mock_bgjob(JournalCSVBackgroundTask.__action__,
                        status=constants.BGJOB_STATUS_COMPLETE, )
    
        status_dict = background_task_status.create_background_status()
    
        journal_csv_dict = status_dict['queues']['scheduled_short']['queued'].get(JournalCSVBackgroundTask.__action__, {})
    
>       assert is_stable(status_dict['status'])
E       AssertionError: assert False
E        +  where False = is_stable('unstable')

test_background_task_status.py:215: AssertionError

doajtest/unit/test_background_task_status.py:130 (TestBackgroundTaskStatus.test_create_background_status__valid_last_completed)
self = <doajtest.unit.test_background_task_status.TestBackgroundTaskStatus testMethod=test_create_background_status__valid_last_completed>

    @apply_test_case_config(bg_monitor_last_completed__a)
    def test_create_background_status__valid_last_completed(self):
        save_mock_bgjob(JournalCSVBackgroundTask.__action__,
                        queue_id=constants.BGJOB_QUEUE_ID_EVENTS,
                        status=constants.BGJOB_STATUS_COMPLETE, )
        save_mock_bgjob(AnonExportBackgroundTask.__action__,
                        queue_id=constants.BGJOB_QUEUE_ID_SCHEDULED_LONG,
                        status=constants.BGJOB_STATUS_COMPLETE, )
    
        status_dict = background_task_status.create_background_status()
    
>       assert is_stable(status_dict['status'])
E       AssertionError: assert False
E        +  where False = is_stable('unstable')

test_background_task_status.py:142: AssertionError

doajtest/unit/test_background_task_status.py:145 (TestBackgroundTaskStatus.test_create_background_status__valid_last_completed__no_record)
self = <doajtest.unit.test_background_task_status.TestBackgroundTaskStatus testMethod=test_create_background_status__valid_last_completed__no_record>

    @apply_test_case_config(bg_monitor_last_completed__now)
    def test_create_background_status__valid_last_completed__no_record(self):
        status_dict = background_task_status.create_background_status()
>       assert is_stable(status_dict['status'])
E       AssertionError: assert False
E        +  where False = is_stable('unstable')

test_background_task_status.py:149: AssertionError

[richard-jones]

@RK206 The testing was a bit confused now there is an additional monitoring stream, so I have replaced it with a parameterised test that gets all the variations and that appears to be working fine.

[RK206]

should it import from portality?

[RK206]

same as above. import from portality?

[richard-jones]

Ah, that's a pycharm thing, it sometimes does the imports for you, and it does them relative, which seems to work locally but not anywhere else. Very annoying!