Wrong fs type, bad option, bad superblock on /dev/loop0p4

[siegy22]

Running on:

• RHEL 9
• Podman 4.4.1

Trying to build a vmdk for my vsphere environment. Running into the following issue:

$ sudo podman run --rm -it --privileged -v /var/lib/containers/storage:/var/lib/containers/storage -v .:/output --security-opt label=type:unconfined_t   --pull newer quay.io/centos-bootc/bootc-image-builder:latest --local --rootfs xfs --type vmdk quay.io/fedora/fedora-bootc:40

Generating manifest manifest-vmdk.json
DONE
Building manifest-vmdk.json
starting -Pipeline source org.osbuild.containers-storage: 1b31ca58fda2e9876d28fb1128f607c63aa0b70a1829fe921a499e089bd797a5
Build
  root: <host>
Pipeline build: 7c62b923dcc6dfbc975b7b610e0b2db50d1d0da374ede68db834e07f8cdc9f13
Build
  root: <host>
  runner: org.osbuild.fedora38 (org.osbuild.fedora38)
org.osbuild.container-deploy: a5c82b527787522bd4c9fd54d9819ab761615cb5293c113092f6b0e30ff17ddc {}
Getting image source signatures
Copying blob sha256:17c2ddfc5a1d4c66eaf1dafdd5bc50882a23c3e5250926cb9cfe734abffc2f64
Copying blob sha256:b1bc5c0fb77df66e970553b0fac7730ac3440fc85e5c619fc01929a83b926a8e
Copying blob sha256:d73015ef64a206468b1ae5acb5d788d7c2e85acbaaa1ad12b11e16ff746c5afa
Copying blob sha256:3e88df441d46735e79afb4e83cf7b23f55a89b13bce03db5a5360d3940cb5a34
Copying blob sha256:6148eca8b30e543ec29be02b2f25506288ba9b25f04e159f6f156857bf7b8f0b
Copying blob sha256:6a342e4616374a75e068e98251beb59d4692ee4f7656f8804a29e1942261d5bb
Copying blob sha256:9cf524c7b562f7a026ee1066792536b810f3d63a2ccf108e4b07a45247ba0e2f
Copying blob sha256:71b47348b71a95aaa247773cbae2912b3686d6e53a7bcba5bc3dd31e59b073a7
Copying blob sha256:c180314d13488858087de9a55763146a44273580d951b9ce06fe5a2da3b1f830
Copying blob sha256:9d44d11c336781375dcca8a32d2b6a9093d55c8f33a20a1c37abfa080b93d714
Copying blob sha256:6739a9594b6de1839ee1a7bd3e9e13b0b4f2470a94bbd1b7d0ab0df90e1cd1b8
Copying blob sha256:1a01f717287b80117fdb300ecb5264b61ed484d02919ea34cfdca809b308aaa4
Copying blob sha256:6cc36bfb5ffb759ff5819d37950ff6ff734447bdd2b7ae020cb5643d468c3ac9
Copying blob sha256:f010736ee2bc6e113a49108f136d2700d95bfa75ba44a4aaab5c55dcb4ab2054
Copying blob sha256:c2029f0e6393410fbe4dd3dcd2ba483aa093e593d127c40abf7bb5f2751dd2d4
Copying blob sha256:528b6bf9aef6fcaa09aad1299e74ec7225a7724443eeb17308f5e54a693417b8
Copying blob sha256:ec8c5d36012a7975668f4021be78d1b652bf1f689786387d25e6d4beb57531ca
Copying blob sha256:a577fc356696b8410390865fa96c35df017ca097357c5b990fcbda2f82b828c0
Copying blob sha256:7b8786e9b0204edc0a93a485e1f954b7d4b17c3b59255f3ce8d16c749712515e
Copying blob sha256:23990a9d06c75e0e065d29808ee2adb76bb89b6ffabe09eeed9d3d6f4e486d75
Copying blob sha256:7715e67a4f38b0e9cd5204836972fc92bec999992c69c8e2f460c131000c66df
Copying blob sha256:cea791ffab36136da7b6c033105b5e8827b0715006329a99194bbf10bfb55632
Copying blob sha256:abb89fb10111ecae072ca16f02ffc4579ee437b3d00c70a99532d66fafb1fb5f
Copying blob sha256:0cb1e50daaf3f4545626a7c9503756fb2818d9abc295e997300c729d28b70c86
Copying blob sha256:6d2d6f147084df80de1de1709fe7726fa28c09d5a7c470a931c87ba48142cda4
Copying blob sha256:bc1eb84c69894a5b3f2663932732c0a346544791a2ae9ff786339a566e8f4fab
Copying blob sha256:4f860ed51ed900a652f3b2c707aa2117c4b3235490f2186e7d05a09e3d8bae02
Copying blob sha256:242ac141f50068fb490bf545336dfede80144f77429944288e1d6e53790b77d1
Copying blob sha256:a45f077b33fe26506200dc1f2081fb53130f6180dbe2dece418711ca506d757b
Copying blob sha256:4693f47bf6d057771ba12d620cf80316958fa0fcf44a769c7d2d8bc14ae5f211
Copying blob sha256:a7abe4eb94cd36eaa7ae37b32d23f356f50a4a937b72291955a3111b52d0279b
Copying blob sha256:a7cb9205f70a03cda256f67adb611037e11ca7bff6e09fc25d104b12b9495a5b
Copying blob sha256:4645c2e2ae86b1e64dd029123a60a2450fae4e839c07e342a72a6c6ff00e213f
Copying blob sha256:7439614dfaa84f4809461a53b3af55b843fe53364f7a54da945e88a41a2b2df8
Copying blob sha256:4e0a19503bd08de2d2ed869e95f939a1942fcbf713c3a3ed99b2ee4a688963d6
Copying blob sha256:a58970a0dbb4f100058daefb661952094ccc28409ad2b2e5978d35d62b82007e
Copying blob sha256:91846fbd046b9cb62be007f373713b6d816daff87acf1c7e4e26356f2752db4a
Copying blob sha256:4ce0435de78b4d2af47057e3013061f8fb578134f701eba158e0c09de694f260
Copying blob sha256:ded15ce631fd3fb8502f4d5e19d30bcdc1bf2029bfe08de5fceed70b8d5aa475
Copying blob sha256:31eb82066ce4ea1b7d13672348781e05b1f185c46f1ed09001f1102f0d3e62f8
Copying blob sha256:bc041f079f049c5bedb2978fe0cbc55f67579b2eb2e0bf9cf6627f0cea7c1fb0
Copying blob sha256:d3b58645baf8770c64d808c122988ca635b70e2379ebe6afabb71022a52cb94a
Copying blob sha256:d4d45306513b33da58ba63c3ea528135fce67fc7ce3237ee69806aa8c8a36fe7
Copying blob sha256:52b2e3447cfd528f3c71a5735a032196dc81b86d863073913eca4b2afd78eda4
Copying blob sha256:d6af311274695c7f08700aab52e4005f521594e2873c82af767487eed91f7502
Copying blob sha256:995ae245b6953b182ea8edd96fc0b0e62eaa301cffe006984bcfb9e558fe8ef6
Copying blob sha256:b4e9b1561e6bd5270ab5f313200272ac7bcfad5d6a875b013cc7d57c507cb42f
Copying blob sha256:018b4abefe21d214f35d746fa597ac61c6aff2654bfeef37e734e2ed8897a9e8
Copying blob sha256:30e42707e5cc9ea42d635757d203729dcd48fd5876d363a0de6b3f0715f86e6b
Copying blob sha256:e54e7443e8a1c615577509f30da75e767d77a4952dd81bc8635a6c35dec287bd
Copying blob sha256:c57d8c434a15e524c0ea2ba690cf774bbb56aa86d565520978a8c7072f55451a
Copying blob sha256:bbc74d78022b20c96f7df61a9bfe3f5e302151a330b138e47816ca1ca01354d9
Copying blob sha256:c474d16efa919b716303f18e9711630c8e6a95788201c69ea8af4cde7565d27a
Copying blob sha256:7cea5aad4710ccd1922a152367da43de8af778566a6954bd0ae0f70581c03d83
Copying blob sha256:49d8ec005974b222e7bad34a7c9cec6256a86776e6d590482e6f8bd3146409bd
Copying blob sha256:deeae36ad7f3c1960909983776b35544ad71681baf13d200e6ed0c34afda1418
Copying blob sha256:cc9b2e4be0cff7a943880cf4427b19c15f350e39d63c0ae881b97a0b53dce064
Copying blob sha256:cc627e2d169ed28c3fe5615b5d93732ed33be94779bb5d1f019eea207cd6933c
Copying blob sha256:c05e9b603acf89040c0de8fb691d87f40e95187810783c7b02da524e679b9ac9
Copying blob sha256:be97d85b6806007defa8ae8f1b218197257d83a445cbd7c676a5151c90f4922d
Copying blob sha256:9ced770a11d55f695f685622bcbc52578660d15f214cd3a02c47df2bb14d29e1
Copying blob sha256:3ce8a2c57a943fc942306cd0207ccb0b85f675bb19b736c00b38dd8909b08d9f
Copying blob sha256:be1bda446b9cc52192be63cbee6e2c1e20c496e1602c25a990610cf7917ea70f
Copying blob sha256:e1e12e8bb544f33130d337e7344afc4fa367b967d1aff5a6f372b0c0fe9c4325
Copying blob sha256:12787d84fa137cd5649a9005efe98ec9d05ea46245fdc50aecb7dd007f2035b1
Copying config sha256:d7c36e87ba1c488cebbab4709b0b8cda5fd09ddc09e87779b585e8f8816b6d30
Writing manifest to image destination
d7c36e87ba1c488cebbab4709b0b8cda5fd09ddc09e87779b585e8f8816b6d30
Untagged: docker.io/library/tmp-container-deploy-30161886013631:latest
Deleted: d7c36e87ba1c488cebbab4709b0b8cda5fd09ddc09e87779b585e8f8816b6d30

⏱  Duration: 95s
org.osbuild.selinux: 7c62b923dcc6dfbc975b7b610e0b2db50d1d0da374ede68db834e07f8cdc9f13 {
  "file_contexts": "etc/selinux/targeted/contexts/files/file_contexts",
  "exclude_paths": [
    "/sysroot"
  ],
  "labels": {
    "/usr/bin/mount": "system_u:object_r:install_exec_t:s0",
    "/usr/bin/ostree": "system_u:object_r:install_exec_t:s0",
    "/usr/bin/umount": "system_u:object_r:install_exec_t:s0"
  }
}

⏱  Duration: 19s
Pipeline image: 8e295b8eac1a86102e831c4fa7ae8fee2aa4bc1292287ae4f7c4828238fe9a13
Build
  root: 7c62b923dcc6dfbc975b7b610e0b2db50d1d0da374ede68db834e07f8cdc9f13
  runner: org.osbuild.linux (org.osbuild.linux)
org.osbuild.truncate: 3a3c362c7886520627a5dec9d6f50bb9a19615e5efe54f6efa7f56d411c5d9c8 {
  "filename": "disk.raw",
  "size": "10737418240"
}

⏱  Duration: 0s
org.osbuild.sfdisk: af4f02c151fc3765e620850ab42c269e401ec8666268a67a726c07f026631fc0 {
  "label": "gpt",
  "uuid": "D209C89E-EA5E-4FBD-B161-B461CCE297E0",
  "partitions": [
    {
      "bootable": true,
      "size": 2048,
      "start": 2048,
      "type": "21686148-6449-6E6F-744E-656564454649",
      "uuid": "FAC7F1FB-3E8D-4137-A512-961DE09A5549"
    },
    {
      "size": 1026048,
      "start": 4096,
      "type": "C12A7328-F81F-11D2-BA4B-00A0C93EC93B",
      "uuid": "68B2905B-DF3E-4FB3-80FA-49D1E773AA33"
    },
    {
      "size": 2097152,
      "start": 1030144,
      "type": "0FC63DAF-8483-4772-8E79-3D69D8477DE4",
      "uuid": "CB07C243-BC44-4717-853E-28852021225B"
    },
    {
      "size": 17844191,
      "start": 3127296,
      "type": "0FC63DAF-8483-4772-8E79-3D69D8477DE4",
      "uuid": "6264D520-3FB9-423F-8AB8-7A0A8E3D3562"
    }
  ]
}
device/device (org.osbuild.loopback): loop0 acquired (locked: True)
label: gpt
label-id: D209C89E-EA5E-4FBD-B161-B461CCE297E0
start="2048", size="2048", type="21686148-6449-6E6F-744E-656564454649", uuid="FAC7F1FB-3E8D-4137-A512-961DE09A5549", bootable
start="4096", size="1026048", type="C12A7328-F81F-11D2-BA4B-00A0C93EC93B", uuid="68B2905B-DF3E-4FB3-80FA-49D1E773AA33"
start="1030144", size="2097152", type="0FC63DAF-8483-4772-8E79-3D69D8477DE4", uuid="CB07C243-BC44-4717-853E-28852021225B"
start="3127296", size="17844191", type="0FC63DAF-8483-4772-8E79-3D69D8477DE4", uuid="6264D520-3FB9-423F-8AB8-7A0A8E3D3562"
{
   "partitiontable": {
      "label": "gpt",
      "id": "D209C89E-EA5E-4FBD-B161-B461CCE297E0",
      "device": "/dev/loop0",
      "unit": "sectors",
      "firstlba": 2048,
      "lastlba": 20971486,
      "sectorsize": 512,
      "partitions": [
         {
            "node": "/dev/loop0p1",
            "start": 2048,
            "size": 2048,
            "type": "21686148-6449-6E6F-744E-656564454649",
            "uuid": "FAC7F1FB-3E8D-4137-A512-961DE09A5549"
         },{
            "node": "/dev/loop0p2",
            "start": 4096,
            "size": 1026048,
            "type": "C12A7328-F81F-11D2-BA4B-00A0C93EC93B",
            "uuid": "68B2905B-DF3E-4FB3-80FA-49D1E773AA33"
         },{
            "node": "/dev/loop0p3",
            "start": 1030144,
            "size": 2097152,
            "type": "0FC63DAF-8483-4772-8E79-3D69D8477DE4",
            "uuid": "CB07C243-BC44-4717-853E-28852021225B"
         },{
            "node": "/dev/loop0p4",
            "start": 3127296,
            "size": 17844191,
            "type": "0FC63DAF-8483-4772-8E79-3D69D8477DE4",
            "uuid": "6264D520-3FB9-423F-8AB8-7A0A8E3D3562"
         }
      ]
   }
}

⏱  Duration: 1s
org.osbuild.mkfs.fat: 5c7253871a9f15c7a3c2989a75f1797413f697f34d038adced61502cbe10b396 {
  "volid": "7B7795E7"
}
device/device (org.osbuild.loopback): loop0 acquired (locked: True)
mkfs.fat 4.2 (2021-01-31)

⏱  Duration: 1s
org.osbuild.mkfs.xfs: 933824326232a39cdfe5687b4e7b04fb2c6e49386c6a91e0a7cdefeb322ec300 {
  "uuid": "a3428caf-7d8d-4abf-a931-69b53b528ada",
  "label": "boot"
}
device/device (org.osbuild.loopback): loop0 acquired (locked: True)
meta-data=/dev/loop0             isize=512    agcount=4, agsize=65536 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=1
         =                       reflink=1    bigtime=1 inobtcount=1 nrext64=1
data     =                       bsize=4096   blocks=262144, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=16384, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
Discarding blocks...Done.

⏱  Duration: 1s
org.osbuild.mkfs.xfs: 50af9e8efa475e586a468608addeaf82cff708b2ba7aecb4a68aa28c882053ff {
  "uuid": "20986a46-cb72-4750-9d57-4948e92d7e06",
  "label": "root"
}
device/device (org.osbuild.loopback): loop0 acquired (locked: True)
meta-data=/dev/loop0             isize=512    agcount=4, agsize=557631 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=1
         =                       reflink=1    bigtime=1 inobtcount=1 nrext64=1
data     =                       bsize=4096   blocks=2230523, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=16384, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
Discarding blocks...Done.

⏱  Duration: 0s
org.osbuild.bootc.install-to-filesystem: da13e96da33e7840f5dba373e0eae38f8cee478c0830e88fd5dfc6f2a47c51c2 {
  "kernel-args": [
    "rw",
    "console=tty0",
    "console=ttyS0"
  ],
  "target-imgref": "quay.io/fedora/fedora-bootc:40"
}
device/disk (org.osbuild.loopback): loop0 acquired (locked: False)
mount/- (org.osbuild.xfs): mounting /dev/loop0p4 -> /store/tmp/buildroot-tmp-9h8jhdtm/mounts/
mount/- (org.osbuild.xfs): already unmounted: /store/tmp/buildroot-tmp-9h8jhdtm/mounts/
Traceback (most recent call last):
  File "/usr/bin/osbuild", line 33, in <module>
    sys.exit(load_entry_point('osbuild==128', 'console_scripts', 'osbuild')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/osbuild/main_cli.py", line 182, in osbuild_cli
    r = manifest.build(
        ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/osbuild/pipeline.py", line 477, in build
    res = pl.run(store, monitor, libdir, debug_break, stage_timeout)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/osbuild/pipeline.py", line 376, in run
    results = self.build_stages(store,
              ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/osbuild/pipeline.py", line 348, in build_stages
    r = stage.run(tree,
        ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/osbuild/pipeline.py", line 222, in run
    data = mntmgr.mount(mount)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/osbuild/mounts.py", line 98, in mount
    path = client.call("mount", args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/osbuild/host.py", line 348, in call
    ret, _ = self.call_with_fds(method, args)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/osbuild/host.py", line 384, in call_with_fds
    raise error
osbuild.host.RemoteError: RuntimeError: b'mount: /store/tmp/buildroot-tmp-9h8jhdtm/mounts: wrong fs type, bad option, bad superblock on /dev/loop0p4, missing codepage or helper program, or other error.\n       dmesg(1) may have more information after failed mount system call.' (code: 32)
   File "/usr/lib/python3.12/site-packages/osbuild/host.py", line 268, in serve
    reply, reply_fds = self._handle_message(msg, fds)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/osbuild/host.py", line 301, in _handle_message
    ret, fds = self.dispatch(name, args, fds)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/osbuild/mounts.py", line 131, in dispatch
    r = self.mount(args)
        ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/osbuild/mounts.py", line 199, in mount
    raise RuntimeError(f"{msg} (code: {code})") from e

2024/09/08 10:02:47 error: cannot run osbuild: running osbuild failed: exit status 1

Copied the command from the fedora-bootc documentation: https://docs.fedoraproject.org/en-US/bootc/provisioning-vsphere/

[ondrejbudai]

Can you share your host's /etc/os-release? I presume that the issue is that your kernel is too old to handle an xfs partition created by Fedora's mkfs.xfs. I see you are running RHEL 9, I didn't see it, sorry. :(

Yeah, this won't work, we use mkfs.xfs from the container itself to create a filesystem, but Fedora's mkfs.xfs by default creates a partition with new enough features that RHEL 9's kernel cannot mount it. Can you try running the same command on Fedora (or any other system with newer kernel than RHEL)? Another option is to try ext4, but maybe you will run into a similar issue.

I wonder what we can do here to fix it. Maybe we can allow people to pass custom options to mkfs.ext4 so they can work around these incompatibilities?

[siegy22]

Yeah just tried running it on a Fedora 40 box. Seems to work just fine. So the minimum kernel version would be what? 6.x I guess?

[siegy22]

Added in the docs, can be closed as this isn't anything that can be handled by bootc-image-builder, I guess?