Task manager refactor

[fosterseth]

Major changes

Split TaskManager into 3 separate managers that can run in parallel

• DependencyManager processes tasks that might need dependencies spawned, and flips dependencies_processed to True after resolving any dependencies
• TaskManager looks at pending tasks with dependencies_processed True. It is responsible for fitting the task to a control instance and execution instance, and submitting the task to the dispatcher.
• WorkflowManager looks only at running workflows and decides whether it can spawn new jobs

Save progress before timing out manager

• After 5 minutes, the task manager PID will be killed. However, there is now a grace period that allows the managers to bail out and save whatever progress it has made before it is killed.

Cache preferred_instance_groups and task_impact

• These two fields on UnifiedJob are fairly expensive, and were being calculated repeatedly across task manager runs. Now, these fields are calculated and set in .create_unified_job, thus no extra DB hits accrue when accessing these fields.

the parent issue tracker can be found here https://github.com/ansible/tower/issues/5943

• fix up unit / functional tests
  • PR [task_manager_refactor] Fix unit / functional tests #12534
• fix up integration / collections tests
• green on test suite
• performance and scale team results
• merge it

ISSUE TYPE

• New or Enhanced Feature

COMPONENT NAME

• API

AWX VERSION

awx: 21.3.1.dev72+g9c74678c61

[jbradberry]

the .join() call also does not need the parameter to be a list comprehension. An implicit generator expression (i.e. just remove the [ and ]) will work just fine, like with the any() call I mentioned previously.

[AlanCoding]

I don't see why we don't add help_text to this (consistent with other new field). If you are not sure if it's taking effect, see GET listed in the OPTIONS data. It's mundane work, but those who work with our API will appreciate it.

[AlanCoding]

I'm still really bothered that we are putting timings on _schedule and not schedule, because this misses timings of on_commit actions, which we know are a constant point of failure. Sure, this could be handled by filing a followup issue, but I really don't want it to get lost or slip into the next release, because it undermines the very point of the task manager metrics. We can still hit the dispatcher timeout limits, with no forewarning by the metrics. Until we fix this, the metrics are not really to be believed.

just the other day, I was able to see this by a new mechanism:

tools_awx_1 | 2022-08-08 17:40:42,465 ERROR    [252b9dcc] awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.run_task_manager(*[], **{}
tools_awx_1 | Traceback (most recent call last):
tools_awx_1 |   File "/awx_devel/awx/main/dispatch/worker/task.py", line 102, in perform_work
tools_awx_1 |     result = self.run_callable(body)
tools_awx_1 |   File "/awx_devel/awx/main/dispatch/worker/task.py", line 77, in run_callable
tools_awx_1 |     return _call(*args, **kwargs)
tools_awx_1 |   File "/awx_devel/awx/main/scheduler/tasks.py", line 15, in run_task_manager
tools_awx_1 |     TaskManager().schedule()
tools_awx_1 |   File "/awx_devel/awx/main/scheduler/task_manager.py", line 657, in schedule
tools_awx_1 |     start_time = time.time()
tools_awx_1 |   File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/transaction.py", line 290, in __exit__
tools_awx_1 |     connection.set_autocommit(True)
tools_awx_1 |   File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 420, in set_autocommit
tools_awx_1 |     self.run_and_clear_commit_hooks()
tools_awx_1 |   File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 653, in run_and_clear_commit_hooks
tools_awx_1 |     func()
tools_awx_1 |   File "/awx_devel/awx/main/scheduler/task_manager.py", line 294, in post_commit
tools_awx_1 |     task_cls.apply_async(
tools_awx_1 |   File "/awx_devel/awx/main/dispatch/publish.py", line 87, in apply_async
tools_awx_1 |     with pg_bus_conn() as conn:
tools_awx_1 |   File "/usr/lib64/python3.9/contextlib.py", line 119, in __enter__
tools_awx_1 |     return next(self.gen)
tools_awx_1 |   File "/awx_devel/awx/main/dispatch/__init__.py", line 50, in pg_bus_conn
tools_awx_1 |     conn = psycopg2.connect(dbname=conf['NAME'], host=conf['HOST'], user=conf['USER'], password=conf['PASSWORD'], port=conf['PORT'], **conf.get("OPTIONS", {}))
tools_awx_1 |   File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg2/__init__.py", line 126, in connect
tools_awx_1 |     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
tools_awx_1 | psycopg2.OperationalError: FATAL:  sorry, too many clients already

This shows that the on_commit logic is even more vulnerable than thought, because it requires establishing a new database connection. When approaching connection limits, this guarantees we leave jobs stuck in waiting. That may or may not be exasperated failing to close unused connections, not sure yet.