mock --init occasinally fails with ERROR: stat: path should be string, bytes, os.PathLike or integer, not NoneType

[kdudka]

Short description of the problem

mock --init occasionally fails with a traceback. mock --scrub=root-cache does not fix it:

>>> 2024-03-27 01:33:04	"/usr/bin/mock" "-r" "rhel-9-x86_64" "--plugin-option=tmpfs:keep_mounted=True" "--config-opts=print_main_output=True" "--init"
INFO: mock.py version 5.2 starting (python version = 3.9.16, NVR = mock-5.2-1.kdudka.5.el9), args: /usr/libexec/mock/mock -r rhel-9-x86_64 --plugin-option=tmpfs:keep_mounted=True --config-opts=print_main_output=True --init
Start(bootstrap): init plugins
INFO: selinux enabled
Finish(bootstrap): init plugins
Start: init plugins
INFO: selinux enabled
Finish: init plugins
INFO: Signal handler active
Start: run
Start: clean chroot
Finish: clean chroot
Mock Version: 5.2
INFO: Mock Version: 5.2
INFO: Package manager dnf detected and used (fallback)
Start: chroot init
ERROR: stat: path should be string, bytes, os.PathLike or integer, not NoneType
Traceback (most recent call last):
  File "/usr/libexec/mock/mock", line 1079, in <module>
    exitStatus = main()
  File "/usr/lib/python3.9/site-packages/mockbuild/trace_decorator.py", line 93, in trace
    result = func(*args, **kw)
  File "/usr/libexec/mock/mock", line 836, in main
    result = run_command(options, args, config_opts, commands, buildroot)
  File "/usr/lib/python3.9/site-packages/mockbuild/trace_decorator.py", line 93, in trace
    result = func(*args, **kw)
  File "/usr/libexec/mock/mock", line 867, in run_command
    commands.init()
  File "/usr/lib/python3.9/site-packages/mockbuild/trace_decorator.py", line 93, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.9/site-packages/mockbuild/backend.py", line 167, in init
    self.buildroot.initialize(**kwargs)
  File "/usr/lib/python3.9/site-packages/mockbuild/trace_decorator.py", line 93, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.9/site-packages/mockbuild/buildroot.py", line 198, in initialize
    self._init(prebuild=prebuild)
  File "/usr/lib/python3.9/site-packages/mockbuild/trace_decorator.py", line 93, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.9/site-packages/mockbuild/buildroot.py", line 281, in _init
    self._setup_nosync()
  File "/usr/lib/python3.9/site-packages/mockbuild/trace_decorator.py", line 93, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.9/site-packages/mockbuild/buildroot.py", line 882, in _setup_nosync
    if not os.path.isdir(self.tmpdir):
  File "/usr/lib64/python3.9/genericpath.py", line 42, in isdir
    st = os.stat(s)
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType

!!! 2024-03-27 01:33:05	warning: failed to init mock profile (rhel-9-x86_64), trying to scrub cache...

>>> 2024-03-27 01:33:05	"/usr/bin/mock" "-r" "rhel-9-x86_64" "--plugin-option=tmpfs:keep_mounted=True" "--config-opts=print_main_output=True" "--scrub=root-cache"
INFO: mock.py version 5.2 starting (python version = 3.9.16, NVR = mock-5.2-1.kdudka.5.el9), args: /usr/libexec/mock/mock -r rhel-9-x86_64 --plugin-option=tmpfs:keep_mounted=True --config-opts=print_main_output=True --scrub=root-cache
Start(bootstrap): init plugins
INFO: selinux enabled
Finish(bootstrap): init plugins
Start: init plugins
INFO: selinux enabled
Finish: init plugins
INFO: Signal handler active
Start: run
Start: scrub ['root-cache']
INFO: scrubbing root-cache for rhel-9-x86_64
Finish: scrub ['root-cache']
Finish: run

>>> 2024-03-27 01:33:07	"/usr/bin/mock" "-r" "rhel-9-x86_64" "--plugin-option=tmpfs:keep_mounted=True" "--config-opts=print_main_output=True" "--init"
INFO: mock.py version 5.2 starting (python version = 3.9.16, NVR = mock-5.2-1.kdudka.5.el9), args: /usr/libexec/mock/mock -r rhel-9-x86_64 --plugin-option=tmpfs:keep_mounted=True --config-opts=print_main_output=True --init
Start(bootstrap): init plugins
INFO: selinux enabled
Finish(bootstrap): init plugins
Start: init plugins
INFO: selinux enabled
Finish: init plugins
INFO: Signal handler active
Start: run
Start: clean chroot
Finish: clean chroot
Mock Version: 5.2
INFO: Mock Version: 5.2
INFO: Package manager dnf detected and used (fallback)
Start: chroot init
ERROR: stat: path should be string, bytes, os.PathLike or integer, not NoneType
Traceback (most recent call last):
  File "/usr/libexec/mock/mock", line 1079, in <module>
    exitStatus = main()
  File "/usr/lib/python3.9/site-packages/mockbuild/trace_decorator.py", line 93, in trace
    result = func(*args, **kw)
  File "/usr/libexec/mock/mock", line 836, in main
    result = run_command(options, args, config_opts, commands, buildroot)
  File "/usr/lib/python3.9/site-packages/mockbuild/trace_decorator.py", line 93, in trace
    result = func(*args, **kw)
  File "/usr/libexec/mock/mock", line 867, in run_command
    commands.init()
  File "/usr/lib/python3.9/site-packages/mockbuild/trace_decorator.py", line 93, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.9/site-packages/mockbuild/backend.py", line 167, in init
    self.buildroot.initialize(**kwargs)
  File "/usr/lib/python3.9/site-packages/mockbuild/trace_decorator.py", line 93, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.9/site-packages/mockbuild/buildroot.py", line 198, in initialize
    self._init(prebuild=prebuild)
  File "/usr/lib/python3.9/site-packages/mockbuild/trace_decorator.py", line 93, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.9/site-packages/mockbuild/buildroot.py", line 281, in _init
    self._setup_nosync()
  File "/usr/lib/python3.9/site-packages/mockbuild/trace_decorator.py", line 93, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.9/site-packages/mockbuild/buildroot.py", line 882, in _setup_nosync
    if not os.path.isdir(self.tmpdir):
  File "/usr/lib64/python3.9/genericpath.py", line 42, in isdir
    st = os.stat(s)
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType

!!! 2024-03-27 01:33:08	fatal error: failed to init mock profile (rhel-9-x86_64)

Output of rpm -q mock

mock-5.2-1.kdudka.5.el9.noarch

Any additional notes

Output of `mock --debug-config`

$ mock -r rhel-9-x86_64 --debug-config
INFO: mock.py version 5.2 starting (python version = 3.9.16, NVR = mock-5.2-1.kdudka.5.el9), args: /usr/libexec/mock/mock -r rhel-9-x86_64 --debug-config
Start(bootstrap): init plugins
INFO: selinux enabled
Finish(bootstrap): init plugins
Start: init plugins
INFO: selinux enabled
Finish: init plugins
INFO: Signal handler active
Start: run
config_opts['additional_packages'] = []
config_opts['bootstrap_image'] = 'registry.access.redhat.com/ubi9/ubi'
config_opts['build_log_fmt_str'] = '%(message)s'
config_opts['cache_alterations'] = False
config_opts['chroot_name'] = 'rhel-9-x86_64'
config_opts['chroot_setup_cmd'] = 'groupinstall build'
config_opts['chrootuid'] = 1000
config_opts['cleanup_on_failure'] = False
config_opts['cleanup_on_success'] = False
config_opts['config_file'] = '/etc/mock/rhel-9-x86_64.cfg'
config_opts['config_path'] = '/etc/mock'
config_opts['dist'] = 'el9'
config_opts['dnf.conf'] = ('\n'
'[main]\n'
'cachedir=/var/cache/dnf\n'
'debuglevel=1\n'
'logfile=/var/log/yum.log\n'
'reposdir=/dev/null\n'
'retries=20\n'
'obsoletes=1\n'
'gpgcheck=0\n'
'assumeyes=1\n'
'keepcache=1\n'
'install_weak_deps=0\n'
'best=1\n'
'\n'
'# repos\n'
'\n'
'[rhel-9.4.0-build]\n'
'name=rhel-9.4.0-build\n'
'baseurl=https://download.eng.bos.redhat.com/brewroot/repos/rhel-9.4.0-build/latest/x86_64\n'
'module_hotfixes=True\n'
'\n'
'[rhel-9.4.0-appstream]\n'
'name=rhel-9.4.0-appstream\n'
'baseurl=https://download.eng.bos.redhat.com/rhel-9/nightly/RHEL-9/latest-RHEL-9.4.0/compose/AppStream/x86_64/os/\n'
'priority=254\n'
'module_hotfixes=True\n'
'skip_if_unavailable=True\n'
'\n'
'# needed for builds that have not passed gating yet\n'
'[rhel-9.4.0-devtest-build]\n'
'name=rhel-9.4.0-devtest-build\n'
'baseurl=https://download.eng.bos.redhat.com/brewroot/repos/rhel-9.4.0-devtest-build/latest/x86_64\n'
'module_hotfixes=True\n'
'skip_if_unavailable=True\n'
'\n'
'[epel9]\n'
'name=epel9\n'
'gpgcheck=0\n'
'baseurl=https://dl.fedoraproject.org/pub/epel/9/Everything/x86_64/\n'
'priority=255\n'
'module_hotfixes=True\n'
'skip_if_unavailable=True\n'
'\n'
'[cov06-rhel-9]\n'
'name=cov06-rhel-9\n'
'gpgcheck=0\n'
'baseurl=https://cov06.lab.eng.brq2.redhat.com/repos/rhel-9-x86_64/\n'
'metadata_expire=60\n'
'priority=8\n'
'module_hotfixes=True\n'
'skip_if_unavailable=True\n'
'\n'
'[kdudka-shellcheck]\n'
'name=kdudka-shellcheck\n'
'baseurl=https://coprbe.devel.redhat.com/results/kdudka/shellcheck/epel-9-$basearch/\n'
'type=rpm-md\n'
'gpgcheck=0\n'
'gpgkey=https://coprbe.devel.redhat.com/results/kdudka/shellcheck/pubkey.gpg\n'
'repo_gpgcheck=0\n'
'enabled=1\n'
'enabled_metadata=1\n'
'cachedir=/var/cache/dnf\n'
'keepcache=1\n'
'module_hotfixes=True\n'
'skip_if_unavailable=True\n'
'\n'
'[kdudka-csdiff]\n'
'name=kdudka-csdiff\n'
'baseurl=https://coprbe.devel.redhat.com/results/kdudka/csdiff/epel-9-$basearch/\n'
'type=rpm-md\n'
'gpgcheck=0\n'
'gpgkey=https://coprbe.devel.redhat.com/results/kdudka/csdiff/pubkey.gpg\n'
'repo_gpgcheck=0\n'
'enabled=1\n'
'enabled_metadata=1\n'
'cachedir=/var/cache/dnf\n'
'keepcache=1\n'
'module_hotfixes=True\n'
'skip_if_unavailable=True\n'
'\n'
'[kdudka-cppcheck]\n'
'name=kdudka-cppcheck\n'
'baseurl=https://coprbe.devel.redhat.com/results/kdudka/cppcheck/rhel-9.dev-$basearch/\n'
'type=rpm-md\n'
'gpgcheck=0\n'
'gpgkey=https://coprbe.devel.redhat.com/results/kdudka/cppcheck/pubkey.gpg\n'
'repo_gpgcheck=0\n'
'enabled=1\n'
'enabled_metadata=1\n'
'cachedir=/var/cache/dnf\n'
'keepcache=1\n'
'module_hotfixes=True\n'
'skip_if_unavailable=True\n'
'\n'
'[kdudka-pylint]\n'
'name=kdudka-pylint\n'
'baseurl=https://coprbe.devel.redhat.com/results/kdudka/pylint/rhel-9.dev-$basearch/\n'
'type=rpm-md\n'
'gpgcheck=0\n'
'gpgkey=https://coprbe.devel.redhat.com/results/kdudka/pylint/pubkey.gpg\n'
'repo_gpgcheck=0\n'
'enabled=1\n'
'enabled_metadata=1\n'
'cachedir=/var/cache/dnf\n'
'keepcache=1\n'
'module_hotfixes=True\n'
'skip_if_unavailable=True\n')
config_opts['dnf_warning'] = False
config_opts['enable_disable_repos'] = []
config_opts['files'] = {'etc/hosts': '127.0.0.1 localhost localhost.localdomain\n'
'::1 localhost localhost.localdomain localhost6 '
'localhost6.localdomain6\n'}
config_opts['isolation'] = 'nspawn'
config_opts['macros'] = {'%_buildhost': 'cov03.lab.eng.brq.redhat.com',
'%_host': 'x86_64-redhat-linux-gnu',
'%_host_cpu': 'x86_64',
'%_rpmfilename': '%%{NAME}-%%{VERSION}-%%{RELEASE}.%%{ARCH}.rpm',
'%_topdir': '/builddir/build',
'%dist': '.el9',
'%distribution': 'Red Hat, Inc.',
'%packager': 'Red Hat, Inc.',
'%vendor': 'Red Hat, Inc.'}
config_opts['nspawn_args'] = ['--capability=cap_ipc_lock',
'--bind=/dev/mapper/control',
'--bind=/dev/fuse',
'--bind=/dev/loop-control',
'--bind=/dev/loop0',
'--bind=/dev/loop1',
'--bind=/dev/loop2',
'--bind=/dev/loop3',
'--bind=/dev/loop4',
'--bind=/dev/loop5',
'--bind=/dev/loop6',
'--bind=/dev/loop7',
'--bind=/dev/loop8',
'--bind=/dev/loop9',
'--bind=/dev/loop10',
'--bind=/dev/loop11']
config_opts['plugin_conf'] = {'bind_mount_enable': True,
'bind_mount_opts': {'basedir': '/var/lib/mock/rhel-9-x86_64',
'cache_topdir': '/var/cache/mock',
'cachedir': '/var/cache/mock/rhel-9-x86_64',
'create_dirs': False,
'dirs': [],
'resultdir': '/var/lib/mock/rhel-9-x86_64/result',
'root': 'rhel-9-x86_64'},
'ccache_enable': False,
'ccache_opts': {'basedir': '/var/lib/mock/rhel-9-x86_64',
'cache_topdir': '/var/cache/mock',
'cachedir': '/var/cache/mock/rhel-9-x86_64',
'compress': None,
'dir': '/var/cache/mock/rhel-9-x86_64/ccache/u1000/',
'max_cache_size': '4G',
'resultdir': '/var/lib/mock/rhel-9-x86_64/result',
'root': 'rhel-9-x86_64'},
'chroot_scan_enable': False,
'chroot_scan_opts': {'basedir': '/var/lib/mock/rhel-9-x86_64',
'cache_topdir': '/var/cache/mock',
'cachedir': '/var/cache/mock/rhel-9-x86_64',
'only_failed': True,
'regexes': ['^[^k]?core(\.\d+)?$', '\.log$'],
'resultdir': '/var/lib/mock/rhel-9-x86_64/result',
'root': 'rhel-9-x86_64'},
'compress_logs_enable': False,
'compress_logs_opts': {'basedir': '/var/lib/mock/rhel-9-x86_64',
'cache_topdir': '/var/cache/mock',
'cachedir': '/var/cache/mock/rhel-9-x86_64',
'command': 'gzip',
'resultdir': '/var/lib/mock/rhel-9-x86_64/result',
'root': 'rhel-9-x86_64'},
'hw_info_enable': True,
'hw_info_opts': {'basedir': '/var/lib/mock/rhel-9-x86_64',
'cache_topdir': '/var/cache/mock',
'cachedir': '/var/cache/mock/rhel-9-x86_64',
'resultdir': '/var/lib/mock/rhel-9-x86_64/result',
'root': 'rhel-9-x86_64'},
'lvm_root_enable': False,
'lvm_root_opts': {'basedir': '/var/lib/mock/rhel-9-x86_64',
'cache_topdir': '/var/cache/mock',
'cachedir': '/var/cache/mock/rhel-9-x86_64',
'pool_name': 'mockbuild',
'resultdir': '/var/lib/mock/rhel-9-x86_64/result',
'root': 'rhel-9-x86_64'},
'mount_enable': True,
'mount_opts': {'basedir': '/var/lib/mock/rhel-9-x86_64',
'cache_topdir': '/var/cache/mock',
'cachedir': '/var/cache/mock/rhel-9-x86_64',
'dirs': [],
'resultdir': '/var/lib/mock/rhel-9-x86_64/result',
'root': 'rhel-9-x86_64'},
'overlayfs_enable': False,
'overlayfs_opts': {'basedir': '/var/lib/mock/rhel-9-x86_64',
'cache_topdir': '/var/cache/mock',
'cachedir': '/var/cache/mock/rhel-9-x86_64',
'resultdir': '/var/lib/mock/rhel-9-x86_64/result',
'root': 'rhel-9-x86_64'},
'package_state_enable': False,
'package_state_opts': {'available_pkgs': False,
'basedir': '/var/lib/mock/rhel-9-x86_64',
'cache_topdir': '/var/cache/mock',
'cachedir': '/var/cache/mock/rhel-9-x86_64',
'installed_pkgs': True,
'resultdir': '/var/lib/mock/rhel-9-x86_64/result',
'root': 'rhel-9-x86_64'},
'pm_request_enable': False,
'pm_request_opts': {'basedir': '/var/lib/mock/rhel-9-x86_64',
'cache_topdir': '/var/cache/mock',
'cachedir': '/var/cache/mock/rhel-9-x86_64',
'resultdir': '/var/lib/mock/rhel-9-x86_64/result',
'root': 'rhel-9-x86_64'},
'procenv_enable': False,
'procenv_opts': {'basedir': '/var/lib/mock/rhel-9-x86_64',
'cache_topdir': '/var/cache/mock',
'cachedir': '/var/cache/mock/rhel-9-x86_64',
'resultdir': '/var/lib/mock/rhel-9-x86_64/result',
'root': 'rhel-9-x86_64'},
'root_cache_enable': True,
'root_cache_opts': {'age_check': True,
'basedir': '/var/lib/mock/rhel-9-x86_64',
'cache_topdir': '/var/cache/mock',
'cachedir': '/var/cache/mock/rhel-9-x86_64',
'compress_program': 'pigz',
'decompress_program': None,
'dir': '/var/cache/mock/rhel-9-x86_64/root_cache/',
'exclude_dirs': ['./proc',
'./sys',
'./dev',
'./tmp/ccache',
'./var/cache/yum',
'./var/cache/dnf',
'./var/log'],
'extension': '.gz',
'max_age_days': 15,
'resultdir': '/var/lib/mock/rhel-9-x86_64/result',
'root': 'rhel-9-x86_64'},
'rpkg_preprocessor_enable': False,
'rpkg_preprocessor_opts': {'basedir': '/var/lib/mock/rhel-9-x86_64',
'cache_topdir': '/var/cache/mock',
'cachedir': '/var/cache/mock/rhel-9-x86_64',
'cmd': '/usr/bin/preproc-rpmspec %(source_spec)s '
'--output %(target_spec)s',
'requires': ['preproc-rpmspec'],
'resultdir': '/var/lib/mock/rhel-9-x86_64/result',
'root': 'rhel-9-x86_64'},
'selinux_enable': True,
'selinux_opts': {'basedir': '/var/lib/mock/rhel-9-x86_64',
'cache_topdir': '/var/cache/mock',
'cachedir': '/var/cache/mock/rhel-9-x86_64',
'resultdir': '/var/lib/mock/rhel-9-x86_64/result',
'root': 'rhel-9-x86_64'},
'showrc_enable': False,
'showrc_opts': {'basedir': '/var/lib/mock/rhel-9-x86_64',
'cache_topdir': '/var/cache/mock',
'cachedir': '/var/cache/mock/rhel-9-x86_64',
'resultdir': '/var/lib/mock/rhel-9-x86_64/result',
'root': 'rhel-9-x86_64'},
'sign_enable': False,
'sign_opts': {'basedir': '/var/lib/mock/rhel-9-x86_64',
'cache_topdir': '/var/cache/mock',
'cachedir': '/var/cache/mock/rhel-9-x86_64',
'cmd': 'rpmsign',
'opts': '--addsign %(rpms)s',
'resultdir': '/var/lib/mock/rhel-9-x86_64/result',
'root': 'rhel-9-x86_64'},
'tmpfs_enable': False,
'tmpfs_opts': {'basedir': '/var/lib/mock/rhel-9-x86_64',
'cache_topdir': '/var/cache/mock',
'cachedir': '/var/cache/mock/rhel-9-x86_64',
'keep_mounted': False,
'max_fs_size': None,
'mode': '0755',
'required_ram_mb': 900,
'resultdir': '/var/lib/mock/rhel-9-x86_64/result',
'root': 'rhel-9-x86_64'},
'yum_cache_enable': True,
'yum_cache_opts': {'basedir': '/var/lib/mock/rhel-9-x86_64',
'cache_topdir': '/var/cache/mock',
'cachedir': '/var/cache/mock/rhel-9-x86_64',
'max_age_days': 30,
'max_metadata_age_days': 30,
'online': True,
'resultdir': '/var/lib/mock/rhel-9-x86_64/result',
'root': 'rhel-9-x86_64'}}
config_opts['print_main_output'] = True
config_opts['releasever'] = 'el9'
config_opts['root'] = 'rhel-9-x86_64'
config_opts['root_log_fmt_str'] = '%(levelname)s %(filename)s:%(lineno)d: %(message)s'
config_opts['rpmbuild_arch'] = 'x86_64'
config_opts['rpmbuild_networking'] = True
config_opts['rpmbuild_timeout'] = 86400
config_opts['state_log_fmt_str'] = '%(asctime)s - %(message)s'
config_opts['target_arch'] = 'x86_64'
config_opts['verbose'] = 1
Finish: run

[praiskup]

The buggy logic has been deactivated if config_opts['nosync'] = False (will be in Mock 5.6+) which we think is configured mostly everywhere because nosync is broken. If this is not the case, please tell us.

The problem with the nosync logic is triggered by Mock's locking, and concurrent Mock processes running against the same chroot (e.g. rhel-9-x86_64). When one Mock process performs --init, the other fails to acquire the lock and expects the chroot has already been initialized.. I'd be surprised if nosync was the only symptom of this concurrency bug.

[praiskup]

Thank you for reporting the problem!

[kdudka]

@praiskup Thanks for checking! The captured output is from csmock, which never runs mock in parallel on the same chroot. csmock implements its own locking for the whole scans (because csmock invokes mock repeatedly for each scan). Anyway, I do not remember when I saw this traceback the last time. And we have not updated mock on OSH workers since I reported this. So it might have been a bug somewhere else, which has been fixed meanwhile.

[kdudka]

An OSH scan of kernel-5.14.0-70.103.1.el9_0 failed with the same traceback yesterday, internally tracked as https://issues.redhat.com/browse/OSH-665. I will try to update mock to 5.6 in the hope that it will mitigate this issue.