Build Fedora 40 anaconda-iso failed on aarch64

[henrywang]

Build Fedora 40 aarch64 anaconda-iso on aarch64 server with command
sudo podman run --rm -it --privileged --pull=newer --tls-verify=false --security-opt label=type:unconfined_t -v /var/ARTIFACTS/work-anaconda-isozv2m_v9i/tmt/plans/bib-image/anaconda-iso/discover/default-0/tests/output:/output -v /var/lib/containers/storage:/var/lib/containers/storage quay.io/centos-bootc/bootc-image-builder:latest --type anaconda-iso --tls-verify=false --target-arch aarch64 --chown 0:0 --rootfs xfs 192.168.100.1:5000/bootc-workflow-test:55nk.

• The building failed at org.osbuild.copy stage.
• Do not have this issue on Fedora 40 x86_64 anaconda-iso building.
• Do not have this issue on CS9 and RHEL 9 anaconda iso.

org.osbuild.copy: 21de03a7fecdd9cde2fb0fefb5285360fcba6e337a1a9d96c4f5316c5b75f400 {
  "paths": [
    {
      "from": "input://tree/",
      "to": "mount://device/"
    }
  ]
}
device/device (org.osbuild.loopback): loop0 acquired (locked: False)
mount/device (org.osbuild.ext4): mounting /dev/loop0 -> /store/tmp/buildroot-tmp-qrdohjkr/mounts/
copying '/run/osbuild/inputs/tree/.' -> '/run/osbuild/mounts/.'
mount/device (org.osbuild.ext4): sync: error syncing '/store/tmp/buildroot-tmp-qrdohjkr/mounts/': Input/output error
mount/device (org.osbuild.ext4): Traceback (most recent call last):
mount/device (org.osbuild.ext4):   File "/usr/lib/osbuild/mounts/org.osbuild.ext4", line 63, in <module>
mount/device (org.osbuild.ext4):     main()
mount/device (org.osbuild.ext4):   File "/usr/lib/osbuild/mounts/org.osbuild.ext4", line 59, in main
mount/device (org.osbuild.ext4):     service.main()
mount/device (org.osbuild.ext4):   File "/usr/lib/python3.12/site-packages/osbuild/host.py", line 252, in main
mount/device (org.osbuild.ext4):     self.stop()
mount/device (org.osbuild.ext4):   File "/usr/lib/python3.12/site-packages/osbuild/mounts.py", line 127, in stop
mount/device (org.osbuild.ext4):     self.umount()
mount/device (org.osbuild.ext4):   File "/usr/lib/python3.12/site-packages/osbuild/mounts.py", line 215, in umount
mount/device (org.osbuild.ext4):     self.sync()
mount/device (org.osbuild.ext4):   File "/usr/lib/python3.12/site-packages/osbuild/mounts.py", line 223, in sync
mount/device (org.osbuild.ext4):     subprocess.run(["sync", "-f", self.mountpoint],
mount/device (org.osbuild.ext4):   File "/usr/lib64/python3.12/subprocess.py", line 571, in run
mount/device (org.osbuild.ext4):     raise CalledProcessError(retcode, process.args,
mount/device (org.osbuild.ext4): subprocess.CalledProcessError: Command '['sync', '-f', '/store/tmp/buildroot-tmp-qrdohjkr/mounts/']' returned non-zero exit status 1.
device/device (org.osbuild.loopback): Traceback (most recent call last):
device/device (org.osbuild.loopback):   File "/usr/lib/osbuild/devices/org.osbuild.loopback", line 178, in <module>
device/device (org.osbuild.loopback):     main()
device/device (org.osbuild.loopback):   File "/usr/lib/osbuild/devices/org.osbuild.loopback", line 174, in main
device/device (org.osbuild.loopback):     service.main()
device/device (org.osbuild.loopback):   File "/usr/lib/python3.12/site-packages/osbuild/host.py", line 252, in main
device/device (org.osbuild.loopback):     self.stop()
device/device (org.osbuild.loopback):   File "/usr/lib/python3.12/site-packages/osbuild/devices.py", line 124, in stop
device/device (org.osbuild.loopback):     self.close()
device/device (org.osbuild.loopback):   File "/usr/lib/osbuild/devices/org.osbuild.loopback", line 159, in close
device/device (org.osbuild.loopback):     self.lo.clear_fd_wait(self.fd, 30)
device/device (org.osbuild.loopback):   File "/usr/lib/python3.12/site-packages/osbuild/loop.py", line 265, in clear_fd_wait
device/device (org.osbuild.loopback):     raise TimeoutError("waiting for loop device timed out")
device/device (org.osbuild.loopback): TimeoutError: waiting for loop device timed out
Traceback (most recent call last):
  File "/usr/bin/osbuild", line 33, in <module>
    sys.exit(load_entry_point('osbuild==124', 'console_scripts', 'osbuild')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/osbuild/main_cli.py", line 181, in osbuild_cli
    r = manifest.build(
        ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/osbuild/pipeline.py", line 477, in build
    res = pl.run(store, monitor, libdir, debug_break, stage_timeout)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/osbuild/pipeline.py", line 376, in run
    results = self.build_stages(store,
              ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/osbuild/pipeline.py", line 348, in build_stages
    r = stage.run(tree,
        ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/osbuild/pipeline.py", line 146, in run
    with contextlib.ExitStack() as cm:
  File "/usr/lib64/python3.12/contextlib.py", line 610, in __exit__
    raise exc_details[1]
  File "/usr/lib64/python3.12/contextlib.py", line 595, in __exit__
    if cb(*exc_details):
       ^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/tempfile.py", line 946, in __exit__
    self.cleanup()
  File "/usr/lib64/python3.12/tempfile.py", line 950, in cleanup
    self._rmtree(self.name, ignore_errors=self._ignore_cleanup_errors)
  File "/usr/lib64/python3.12/tempfile.py", line 930, in _rmtree
    _shutil.rmtree(name, onexc=onexc)
  File "/usr/lib64/python3.12/shutil.py", line 759, in rmtree
    _rmtree_safe_fd(stack, onexc)
  File "/usr/lib64/python3.12/shutil.py", line 703, in _rmtree_safe_fd
    onexc(func, path, err)
  File "/usr/lib64/python3.12/shutil.py", line 700, in _rmtree_safe_fd
    onexc(os.unlink, fullname, err)
  File "/usr/lib64/python3.12/shutil.py", line 698, in _rmtree_safe_fd
    os.unlink(entry.name, dir_fd=topfd)
OSError: [Errno 30] Read-only file system: '/store/tmp/buildroot-tmp-qrdohjkr/mounts'
2024/08/03 11:05:43 error: cannot run osbuild: running osbuild failed: exit status 1

[chunfuwen]

Look like it can successfully build out iso for fedora-40 on aarch64 with below command(without --target-arch aarch64 option):

sudo podman run --rm -it --privileged --pull=newer --security-opt label=type:unconfined_t -v /var/lib/libvirt/images/output:/output -v /var/lib/libvirt/images/config.json:/config.json   -v /var/lib/libvirt/images/auth.json:/run/containers/0/auth.json  quay.io/centos-bootc/bootc-image-builder:latest  --type anaconda-iso --tls-verify=true  --config /config.json  quay.io/fedora/fedora-bootc:40  --chown 107:107  --rootfs ext4 '

....

2024-08-14 03:58:05,519 process          L0470 DEBUG| [stdout] bootiso-tree:	fc0fb59fcc21d531d3e81bfe4df620c0f784d7f3e13521362917054597041b28
2024-08-14 03:58:05,519 process          L0470 DEBUG| [stdout] bootiso:  	92da29a858a96726301387982499ac230358cedd37d7f60c45bb0cbbbeccb95d
2024-08-14 03:58:07,511 process          L0470 DEBUG| [stdout] Build complete!
2024-08-14 03:58:07,511 process          L0470 DEBUG| [stdout] Results saved in
2024-08-14 03:58:07,511 process          L0470 DEBUG| [stdout] 

[henrywang]

In my case, the Fedora 42 has the same issue.
Fedora 42: https://artifacts.osci.redhat.com/testing-farm/9eb5647e-cc61-4fd3-8c82-d3d5a8ab6842/