Large netbox dynamic inventory ouput doesn't allow sync hosts

[proletariandrift]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that AWX is open source software provided for free and that I might not receive a timely response.
• I am NOT reporting a (potential) security vulnerability. (These should be emailed to [email protected] instead.)

Bug Summary

While using Netbox dynamic inventory script, which outputs inventory just for 1 host, it works fine. But if using more then 1 host awx-dispatcher gets an error:

awx-dispatcher stderr | 2024-02-26 14:52:49,591 ERROR    [ac98e042] awx.main.tasks.jobs inventory_update 4882 (running) Post run hook errored.
awx-dispatcher stderr | Traceback (most recent call last):
awx-dispatcher stderr |   File "/awx_devel/awx/main/tasks/jobs.py", line 636, in run
awx-dispatcher stderr |     self.post_run_hook(self.instance, status)
awx-dispatcher stderr |   File "/awx_devel/awx/main/tasks/jobs.py", line 1645, in post_run_hook
awx-dispatcher stderr |     with open(expected_output) as f:
awx-dispatcher stderr | FileNotFoundError: [Errno 2] No such file or directory: '/tmp/awx_4882_5jqtcokc/artifacts/4882/output.json'

AWX version

22.3.0

Select the relevant components

• UI
• UI (tech preview)
• API
• Docs
• Collection
• CLI
• Other

Installation method

docker development environment

Modifications

no

Ansible version

No response

Operating system

Ubuntu 18.04.5 LTS

Web browser

Chrome

Steps to reproduce

I use such Netbox inventory script for getting all active virtual machines and group it for their role:

plugin: netbox.netbox.nb_inventory
api_endpoint: "https://netbox.example.com"
timeout: 300
validate_certs: false
leading_separator: false
keyed_groups:
  - key: custom_fields.vm_role
query_filters:
  - status: active
  - cluster: cluster1
device_query_filters:
  - device_type: "null"

If use it localy with ansible, it outputs 1 group with 2 host in each:

ansible-inventory --list -i netbox-inventory.yaml

{
    "_meta": {
        "hostvars": {
            "server1": {
                "cluster": "cluster1",
                "cluster_group": "cluster_group1",
                "cluster_type": "cluster_type1",
                "custom_fields": {
                    "vm_option_1":  null,
                    "vm_option_2": null,
                    "vm_option_3": null,
                    "vm_option_4": null,
                    "vm_option_5": null,
                    "vm_role": "role1"
                },
                "disk": 15,
                "is_virtual": true,
                "local_context_data": [
                    null
                ],
                "locations": [],
                "memory": 1024,
                "platforms": [
                    "linux"
                ],
                "regions": [
                    "city1",
                    "country1"
                ],
                "services": [],
                "site_groups": [
                    "site_group1"
                ],
                "sites": [
                    "site1"
                ],
                "status": {
                    "label": "Active",
                    "value": "active"
                },
                "tags": [],
                "tenants": [
                    "test"
                ],
                "vcpus": 1.0
            },
            "server2": {
                "cluster": "cluster1",
                "cluster_group": "cluster_group1",
                "cluster_type": "cluster_type1",
                "custom_fields": {
                    "vm_option_1":  null,
                    "vm_option_2": null,
                    "vm_option_3": null,
                    "vm_option_4": null,
                    "vm_option_5": null,
                    "vm_role": "role1"
                },
                "disk": 10,
                "is_virtual": true,
                "local_context_data": [
                    null
                ],
                "locations": [],
                "memory": 1024,
                "platforms": [
                    "linux"
                ],
                "regions": [
                    "city1",
                    "country1"
                ],
                "services": [],
                "site_groups": [
                    "site_group1"
                ],
                "sites": [
                    "site1"
                ],
                "status": {
                    "label": "Active",
                    "value": "active"
                },
                "tags": [],
                "vcpus": 1.0
            }
        }
    },
    "all": {
        "children": [
            "ungrouped",
            "group1"
        ]
    },
    "group1": {
        "hosts": [
            "server1",
            "server2"
        ]
    }
}

And if using it in AWX while syncing:

awx-uwsgi stderr | 2024-02-26 16:13:13,005 INFO     [706eb8ab] awx.analytics.job_lifecycle inventoryupdate-4922 created
awx-dispatcher stderr | 2024-02-26 16:13:13,137 INFO     [706eb8ab] awx.analytics.job_lifecycle inventoryupdate-4922 acknowledged
awx-dispatcher stderr | 2024-02-26 16:13:13,204 INFO     [706eb8ab] awx.analytics.job_lifecycle inventoryupdate-4922 controller node chosen
awx-dispatcher stderr | 2024-02-26 16:13:13,205 INFO     [706eb8ab] awx.analytics.job_lifecycle inventoryupdate-4922 execution node chosen
awx-dispatcher stderr | 2024-02-26 16:13:13,275 INFO     [706eb8ab] awx.analytics.job_lifecycle inventoryupdate-4922 waiting
awx-dispatcher stderr | 2024-02-26 16:13:13,451 INFO     [706eb8ab] awx.analytics.job_lifecycle inventoryupdate-4922 pre run
awx-dispatcher stderr | 2024-02-26 16:13:17,758 INFO     [706eb8ab] awx.analytics.job_lifecycle inventoryupdate-4922 preparing playbook
awx-dispatcher stderr | 2024-02-26 16:13:17,814 INFO     [706eb8ab] awx.analytics.job_lifecycle inventoryupdate-4922 running playbook
awx-dispatcher stderr | 2024-02-26 16:13:17,823 INFO     [706eb8ab] awx.analytics.job_lifecycle inventoryupdate-4922 work unit id received
awx-dispatcher stderr | 2024-02-26 16:13:17,838 INFO     [706eb8ab] awx.analytics.job_lifecycle inventoryupdate-4922 work unit id assigned
awx-receiver stderr | 2024-02-26 16:14:15,235 INFO     [706eb8ab] awx.main.commands.run_callback_receiver Starting EOF event processing for Job 4922
awx-dispatcher stderr | 2024-02-26 16:14:15,240 INFO     [706eb8ab] awx.analytics.job_lifecycle inventoryupdate-4922 post run
awx-dispatcher stderr | 2024-02-26 16:14:15,265 ERROR    [706eb8ab] awx.main.tasks.jobs inventory_update 4922 (running) Post run hook errored.
awx-dispatcher stderr | Traceback (most recent call last):
awx-dispatcher stderr |   File "/awx_devel/awx/main/tasks/jobs.py", line 636, in run
awx-dispatcher stderr |     self.post_run_hook(self.instance, status)
awx-dispatcher stderr |   File "/awx_devel/awx/main/tasks/jobs.py", line 1645, in post_run_hook
awx-dispatcher stderr |     with open(expected_output) as f:
awx-dispatcher stderr | FileNotFoundError: [Errno 2] No such file or directory: '/tmp/awx_4922_kfiu_9ak/artifacts/4922/output.json'
awx-receiver stderr | 2024-02-26 16:14:15,287 INFO     [706eb8ab] awx.analytics.job_lifecycle inventoryupdate-4922 stats wrapup finished
awx-dispatcher stderr | 2024-02-26 16:14:15,391 INFO     [706eb8ab] awx.analytics.job_lifecycle inventoryupdate-4922 finalize run

But if I use inventory which get only 1 server, it works ok:

plugin: netbox.netbox.nb_inventory
api_endpoint: "https://netbox.example.com"
timeout: 300
validate_certs: false
leading_separator: false
keyed_groups:
  - key: custom_fields.vm_role
query_filters:
  - name: server1
  - status: active
  - cluster: cluster1
device_query_filters:
  - device_type: "null"

Expected results

While using dynamic inventory with large output, every host and group should be handled.

Actual results

If dynamic inventory outputs more then 1 host, sync job crushes with error.

Additional information

No response

[thedoubl3j]

hey @proletariandrift this might be a good question for the forum but one follow up question, using the docker dev environment is not supported. Are you able to reproduce on a kube install?

[proletariandrift]

Thanks!
Unfortunately, there is no way to test in k8s.