Feature/392 cron check task

[zamuzakki]

This is PR for ConservationInternational/cplus-plugin#392

[github-actions]

☂️ Python Coverage

current status: ✅

Overall Coverage

Lines	Covered	Coverage	Threshold	Status
3168	3073	97%	0%	🟢

New Files

No new covered files...

Modified Files

No covered modified files...

updated for commit: 7686e88 by action🐍

[codecov]

Codecov Report

Attention: Patch coverage is 98.46154% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 97.07%. Comparing base (327fde5) to head (7686e88).

Files	Patch %	Lines
...ngo_project/cplus_api/tasks/check_scenario_task.py	96.15%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #27      +/-   ##
==========================================
+ Coverage   97.01%   97.07%   +0.05%     
==========================================
  Files          59       60       +1     
  Lines        3019     3076      +57     
==========================================
+ Hits         2929     2986      +57     
  Misses         90       90              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[danangmassandy]

@zamuzakki could you check this answer in SO thread? He explains very well about the state of a task in Celery. We need to ensure that we don't stop a task which running time is more than 2H, so can we check the 'running' state from AsyncResult before stopping the job?

I also want to explain the 'PENDING' and 'QUEUED' in our task status enum:

1. PENDING: task has been sent to the celery, but worker has not picked it yet.
2. QUEUED: task is received/acknowledged by worker and it's still in the worker's queue. I'm not sure how many items in the worker's queue, could you perhaps find out about it?

You can do a testing by making a long task, then trigger that task to run 10-20 times to the 'cplus' queue. There will be some tasks with QUEUED status and other tasks with PENDING status.
From my observation in different project, if a task is already in the QUEUED state and the worker is restarted, then the task is basically gone. The worker will not pick that task anymore.
Meanwhile, if the task is still in PENDING state and the worker is restarted, the worker is still able to pick that task. I am guessing that celery stores the PENDING task list in redis. When worker acknowledges a task, celery removes it from the list.

In the future we should also handle the QUEUED tasks that have been lost, but it's kind of difficult right now because the AsyncResult will return the task state as pending. One possible way is to check against the queue (reserved tasks) in the worker.
Below is the screenshot of flower monitoring tool. The lifecycle of a task as follows: celery task list -> worker's reserved list -> active list -> processed/revoked list.