Yumrepofsv2

[NeilHanlon]

Yumrepofsv1 was created with specific goals in mind. The architecture therein has not scaled particularly well with the use of Peridot for building 9.x, and will continue to have tremendous pain points as we grow and add 8.x into the mix.

yumrepofs v2 should be able to:

• Always be able to regenerate states
• Keep/provide old artifacts in a repository
• Repositories are "hashed" by default (this means sorted into alphanumeric directories) rather than task ID

[nazunalika]

yumrepofs catalogs should also have an easier way to define noarch packages that only belong to a specific architecture's repositories without using the glob matching and exclude sections. there are cases where edk2 (as an example) has portions in most arches, but certain noarch packages don't exist in all of them, and it's not clear without looking at the excludes at the very top of a generated catalog. (the way it is is still fine, but something to think about for future state.)

[NeilHanlon]

yumrepofs catalogs should also have an easier way to define noarch packages that only belong to a specific architecture's repositories without using the glob matching and exclude sections. there are cases where edk2 (as an example) has portions in most arches, but certain noarch packages don't exist in all of them, and it's not clear without looking at the excludes at the very top of a generated catalog. (the way it is is still fine, but something to think about for future state.)

this is making me think about some generic/additional filtering layer that would allow us to make these changes.. or, at least, a more "clear" way than is defined by the proto currently. I think there is probably a better way to represent and/or override the placement of NEVRAs into repositories. In my head, it feels as though we could have some post-process thing which rearranges or sorts the packages if we need to make customizations. I may be thinking about this wrong though.

[nazunalika]

I was looking at it like this. We have an exclude_filter that may encompass many things (whether it's what was filtered in the pungi config globally or what packages (like noarch) that does not exist in a given arch based on the prepopulate.json). And then the package section lists what will be there, arch or noarch, as a general rule.

## Start: Global filter
exclude_filter {
  repo_match: "^(BaseOS|AppStream|CRB|HighAvailability|ResilientStorage|SAP|SAPHANA)$"
  arch {
    key: "*"
    glob_match: "kernel-rt*"
    glob_match: "javapackages-bootstrap"
    glob_match: "insights-client*"
  }
}
exclude_filter {
  repo_match: "^.*$"
  arch {
    key: "*"
    glob_match: "glibc32"
    glob_match: "libgcc32"
    glob_match: "*openh264*"
    glob_match: "python3-openipmi"
    glob_match: "OpenIPMI-perl"
    glob_match: "rhc"
    glob_match: "rhc-worker-playbook"
  }
  arch {
    key: "ppc64le"
    glob_match: "SLOF"
    glob_match: "guestfs-tools"
    glob_match: "libguestfs"
    glob_match: "libvirt-daemon-kvm"
    glob_match: "libvirt-daemon-driver-qemu"
    glob_match: "qemu-kiwi"
    glob_match: "qemu-kvm"
    glob_match: "supermin"
    glob_match: "virt-manager"
    glob_match: "virt-v2v"
    glob_match: "virt-p2v"
    glob_match: "virt-top"
    glob_match: "cockpit-machines"
  }
  arch {
    key: "s390x"
    glob_match: "rust-std-static-wasm32-unknown-unknown"
    glob_match: "rust-std-static-wasm32-wasi"
  }
}
## End: Global filter
## Start: per-repo exclude filters (using edk2 as the example, ". . ." represents more entries that are redacted for clarity)
exclude_filter {
  repo_match: "^AppStream$"
  arch {
    key: "ppc64le"
    . . .
    glob_match: "edk2-aarch64"
    glob_match: "edk2-ovmf"
    . . .
  }
  arch {
    key: "s390x"
    . . .
    glob_match: "edk2-aarch64"
    glob_match: "edk2-ovmf"
    . . .
  }
  arch {
    key: "ppc64le"
    glob_match: "edk2-aarch64"
    . . .
  }
  arch {
    key: "x86_64"
    glob_match: "edk2-ovmf"
    . . .
  }
}
exclude_filter {
  repo_match: "^CRB$"
  arch {
    key: "x86_64"
    glob_match: "edk2-ovmf"
    . . .
  }
  arch {
    key: "ppc64le"
    glob_match: "edk2-tools-doc"
    . . .
  }
  arch {
    key: "s390x"
    glob_match: "edk2-tools-doc"
    . . .
  }
  arch {
    key: "aarch64"
    glob_match: "edk2-aarch64"
    . . .
  }
}
## End: per-repo exclude filters
. . .
## edk2 package example
package {
  name: "edk2"
  type: PACKAGE_TYPE_NORMAL_FORK
  repository {
    name: "AppStream"
    include_filter: "edk2-aarch64.noarch"
    include_filter: "edk2-ovmf.noarch"
  }
  repository {
    name: "CRB"
    include_filter: "edk2-ovmf.noarch"
    include_filter: "edk2-tools-doc.noarch"
    include_filter: "edk2-tools.aarch64"
    include_filter: "edk2-aarch64.noarch"
    include_filter: "edk2-tools.x86_64"
  }
}
## end edk2 package example

In an ideal world, I can see exclude_filter mainly listening out global stuff as listed by filter_packages in pungi, and then the package stanzas being specific on what's actually included. For example:

## Start: Global filter_packages from pungi
exclude_filter {
  repo_match: "^(BaseOS|AppStream|CRB|HighAvailability|ResilientStorage|SAP|SAPHANA)$"
  arch {
    key: "*"
    glob_match: "kernel-rt*"
    glob_match: "javapackages-bootstrap"
    glob_match: "insights-client*"
  }
}
exclude_filter {
  repo_match: "^.*$"
  arch {
    key: "*"
    glob_match: "glibc32"
    glob_match: "libgcc32"
    glob_match: "*openh264*"
    glob_match: "python3-openipmi"
    glob_match: "OpenIPMI-perl"
    glob_match: "rhc"
    glob_match: "rhc-worker-playbook"
  }
  arch {
    key: "ppc64le"
    glob_match: "SLOF"
    glob_match: "guestfs-tools"
    glob_match: "libguestfs"
    glob_match: "libvirt-daemon-kvm"
    glob_match: "libvirt-daemon-driver-qemu"
    glob_match: "qemu-kiwi"
    glob_match: "qemu-kvm"
    glob_match: "supermin"
    glob_match: "virt-manager"
    glob_match: "virt-v2v"
    glob_match: "virt-p2v"
    glob_match: "virt-top"
    glob_match: "cockpit-machines"
  }
  arch {
    key: "s390x"
    glob_match: "rust-std-static-wasm32-unknown-unknown"
    glob_match: "rust-std-static-wasm32-wasi"
  }
}
## End: global filter

# edk2 package example
package {
  name: "edk2"
  type: PACKAGE_TYPE_NORMAL_FORK
  repository {
    name: "AppStream"
    arch {
      key: "x86_64"
      include_filter: "edk2-ovmf.noarch"
    }
    arch {
      key: "aarch64"
      include_filter: "edk2-aarch64.noarch"
    }
  }
  repository {
    name: "CRB"
    arch {
      key: "aarch64"
      include_filter: "edk2-ovmf.noarch"
      include_filter: "edk2-tools-doc.noarch"
    }
    arch {
      key: "x86_64"
      include_filter: "edk2-aarch64.noarch"
      include_filter: "edk2-tools-doc.noarch"
    }
    arch {
      key: "ppc64le"
      include_filter: "edk2-aarch64.noarch"
      include_filter: "edk2-ovmf.noarch"
    }
    arch {
      key: "s390x"
      include_filter: "edk2-aarch64.noarch"
      include_filter: "edk2-ovmf.noarch"
    }
    include_filter: "edk2-tools.aarch64"
    include_filter: "edk2-tools.x86_64"
  }
}

This keys off by arch in the package -> repository sections, thus reducing the amount of general exclude_filter at the top of the catalog.