Replies: 15 comments 78 replies
-
Sign-offsReviewedApproved |
Beta Was this translation helpful? Give feedback.
-
|
Is producing an SBOM for these fetched RPMs in-scope? Just asking because I don't see any mention of it |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
Whatever. I'm just saying that |
Beta Was this translation helpful? Give feedback.
-
How so? it points exactly where the package was fetched from. The fact that it may point to an internal resource is out of scope of this project though, that's caller's responsibility to process the generated SBOM accordingly, including following any company security practices wrt/ not leaking such information, e.g. the lifetime of the SBOM might only be up until the point an analyzer tool verifies there's nothing else in the container image beyond what the SBOM claims it to be and then either the SBOM is discarded completely or further transformed to replace the internal resources with official pointers to publicly accessible repos, either way, it's beyond cachi2. |
Beta Was this translation helpful? Give feedback.
-
|
Ack, if there is a post-processing for SBOMs, then url may work. I assumed SBOMs are treated as immutable. |
Beta Was this translation helpful? Give feedback.
-
Well, ideally they should be, but as you can clearly see there are limitations, IOW you either post-process it before publishing or you don't publish it at all, depends on the exact use case. |
Beta Was this translation helpful? Give feedback.
-
Are possible modifications discussed here: rpm-software-management/rpm#2908 (comment) going to be reflected in the lockfile format proposed here? I think it should be. |
Beta Was this translation helpful? Give feedback.
-
|
Based on recent discussions, wouldn't be better to stick to the current format where each package in the list of packages contains repoid (rather then grouping packages per repoid)? |
Beta Was this translation helpful? Give feedback.
-
Technically noone in the discussion went to such detail to even consider something else, in fact they didn't even agree on the very basis of the format. I'm only trying to declutter the format by decreasing redundant information to find the least common denominator to consolidate it. |
Beta Was this translation helpful? Give feedback.
-
|
The need for and the rest is left to the caller. The reason for that is that by doing and
Not only cachi2 mandates the repos to be exposed locally under |
Beta Was this translation helpful? Give feedback.
-
|
@eskultety I cannot comment on the first half as it's not me who defines scope of cachi2 vs rest of Konflux build process. Ralph or some other architects (?) will need to decide here. About the createrepo question I'm not sure what you mean by |
Beta Was this translation helpful? Give feedback.
-
@Tojaj this is what I wanted to know, so it's always just about metadata, that clarifies my |
Beta Was this translation helpful? Give feedback.
-
cachi2 is a tool not a direct Konflux component (or Konflux-only component, at least I'd like to treat it that way), so Konflux as a consumer of this feature would have to deal with this on their own. I truly don't like having this RPM feature tailored too much to the first consumer that comes and generating |
Beta Was this translation helpful? Give feedback.
-
|
In the current package managers workflow, users need to use a few commands in order to both prefetch and prepare the content to be used in the build. To illustrate: # fetch the dependencies
cachi2 fetch-deps npm
# generate env variables that need to be used when performing the build/install comand (e.g. npm install)
cachi2 generate-env ./cachi2-output -o ./cachi2.env
# modify the source files so that the package manager uses the packages from a local folder
cachi2 inject-files ./cachi2-outputSo I think |
Beta Was this translation helpful? Give feedback.
-
My preference would still be |
Beta Was this translation helpful? Give feedback.
-
This design doesn't factor in any potential GPG usage, does factoring it in change anything about this design? IOW to ensure a high level of SLSA in the secure supply chain we might want to be able signing the created repository and hence ensuring their integrity and authenticity for the rest of the steps in a pipeline for which supporting GPG might be suitable. |
Beta Was this translation helpful? Give feedback.
-
|
Signatures of the RPMs themselves are not going to be altered in any way by this process. Whatever is using them downstream in the build process should verify their signatures (though this may need cooperation from the step that writes the repo file) in whatever way is appropriate. Do you want cachi2 to sign the created repodata? I don't see any blocker for that in the proposal, though I would not put that in scope for the first proof-of-concept. |
Beta Was this translation helpful? Give feedback.
-
|
I agree it's definitely out of scope for a POC, I just wanted to raise awareness of the issue and ideally make it part of the design, but mark it as a future extension should it make any kind of difference as far as the design goes (I don't know whether it does or not at this point just yet), I just wanted to make sure we know about it. As for the signing itself, I don't think we should sweep that possibility off the table, strictly from implementation POV that should be just one additional independent step. |
Beta Was this translation helpful? Give feedback.
This comment has been hidden.
This comment has been hidden.
-
|
The first sentence is saying that there are no requirements on the URL for each RPM. It must point to the RPM. It does not have to point to a repository. It could just as well point to some generic artifact storage. The prefetching tool doesn't need any repodata to download individual RPMs. As an example, you could replace the first URL in the initial example with Module metadata is more complicated. There is no single file you could point the lockfile to in order to get full information about particular module (and only that module). Hence the possible need to actually understand the data more. |
Beta Was this translation helpful? Give feedback.
-
|
okay, makes more sense now. |
Beta Was this translation helpful? Give feedback.
-
Can you elaborate some more on the modularity use case to get full picture of the ultimate future architecture? |
Beta Was this translation helpful? Give feedback.
-
|
There is no design for the module part yet. Here's a stream of rambling ideas: for modules you need RPMs and metadata. The RPMs I would expect to be handled by the existing proposal, there's nothing special about them that would prevent that. The content origin repository used by the (currently non-existing, not designed) tool that produces the lockfile would have a single YAML file in the repodata that contains all modular information for all modules. Maybe the lockfile could use this URL. Maybe the content could be used as a whole, or it could be somehow filtered to preserve only the needed objects.. A different option would be to somehow find the files in another storage where there's single YAML document per file. This could be done for ModuleStreams by by looking them up in MBS. I don't know of a reasonable way to find the source of Defaults though. |
Beta Was this translation helpful? Give feedback.
-
Sources are arch independent, shouldn't they be under |
Beta Was this translation helpful? Give feedback.
-
@Tojaj so if IIUC given that there's only ever a single source container for all arches it is hence unacceptable for the pipeline process to invoke cachi2 multiple times with multiple different lockfiles? |
Beta Was this translation helpful? Give feedback.
-
|
@eskultety the current format allows you to have sepearate lockfiles if you want to. I propose to opt to a single file by default for cachi2, as I'm not aware of any content production process that would benefit from multiple files (that brings risk of inconsistency). On the other hand I'm aware that production processes for Containers, RPMs, ISOs, VHD, AMIs, QCOW2s, etc. that we produce en masse would all benefit from a single file and its inherently guaranteed consistency of package set and package versions between arches. (I mentioned examples here rpm-software-management/dnf5#833 (comment)) Cachi2 support for multiple lock files can be always added when there is a real use-case for that. YAGNI. I don't see any blocker for supporting both scenarios.
|
Beta Was this translation helpful? Give feedback.
-
Well, that however doesn't mean it should be ^this lockfile, rather a template file that needs to be both a single file and what is actually maintained in each project, I think having a parametrized template lockfile is much easier to maintain long term, and you also don't have any consistency problems, everything is perfectly consistent. |
Beta Was this translation helpful? Give feedback.
-
|
I honestly don't understand this comment. You know that there will be a tooling to generate this |
Beta Was this translation helpful? Give feedback.
-
|
Sure. You said:
The risk of inconsistency only exists if there's another external factor to the lockfile maintenance and that could be either human factor or a process that generates the lockfile. If we can agree on the base premise that what you're propose here simply isn't going to be the file than anyone wished to maintain manually, then generation of this file is the only actor where the inconsistency (i.e. a bug) can stem from. So, in that case, if a a generating program can easily generate a correct fully exploded lockfile as proposed here with a list of architectures included, it can easily generate one per architecture and remain perfectly consistent, so what you're describing seems like a complete non-issue to me. You then continue
You say that all services would all benefit from a single file. My question is what is that single file? Because all those services you mentioned are pretty much fully automated, so existence of a single lockfile containing all architectures vs having one per-architecture shouldn't play any role in an automated environment, especially apparently since the plan has been to generate the proposed lockfile (out of scope for cachi2) all along. To me your argument about the importance of a single file only makes sense when we start talking about the actual origin, template (or source if you will) for the lockfile format you're proposing here in this discussion that someone somewhere will have to maintain in order to generate this lockfile for the supply chain pipeline. Note that I haven't even hinted at any discussion about a template file being already in works, I only pitched the general idea of a generating template file being the ONLY source of truth for these lockfile solely based on my reasoning (and understanding) above, i.e. maintenance of the input file (whichever is going to be the one). My motivation here is to keep the file as lean as possible and I somehow cannot wrap my head around the need to specify multiple architectures in a single file and jumping over a few more hoops rather than keeping the file simple and more straightforward for the price of a potential (although marginal) performance penalty for the whole supply chain pipeline. Despite your earlier claim that your proposed format doesn't prevent anyone to specify a single arch, while true, my impression is that while there is a small penalty for generating per-arch lockfiles this format as is is both a bit harder to process and definitely harder to maintain whoever ends up dealing with it. |
Beta Was this translation helpful? Give feedback.
-
|
I updated the lockfile format with items |
Beta Was this translation helpful? Give feedback.
-
What's the point of this field, cachi2 won't process |
Beta Was this translation helpful? Give feedback.
-
|
I am opening a new thread here to find out what is the blocker for the design to be approved. |
Beta Was this translation helpful? Give feedback.
-
|
@onosek from my perspective it would be to get some final conclusion on the need of the multiarch list spec design in the format and what the requirements on cachi2 are when it comes to invoking |
Beta Was this translation helpful? Give feedback.
-
|
@onosek I'd almost forget (thanks @brunoapimentel for reminding me), SBOM content needs to be part of the design as well, we touched it in the discussion, but to avoid any misunderstanding we'll need SBOM too. |
Beta Was this translation helpful? Give feedback.
-
|
Erik also mentioned that |
Beta Was this translation helpful? Give feedback.
-
|
I know the tool to generate the lockfile is out of scope, but would there be a way to tell if the content of sources actually match the .rpm files that are listed in the lockfile? And how much do we care about this accuracy? The reason I'm asking this is because, in our current pipeline, this is an important matter, to the point that pre-compiled binaries for languages such as Python are completely forbidden, and compilation must be done from sources during build time. |
Beta Was this translation helpful? Give feedback.
-
|
May I ask you why you want to use SRPM and binary RPM matching for content verification? If signature verification is not enough then we have a problem because we depend on that a lot. It is important to ensure that downloaded file has correct checksum. But if someone replace the file then it can also modify checksum in manifesto. It is important to ensure that RPM verifies signature when they get installed. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the clarification. I believe the case of Python wheels are different, since we're pulling them from PyPI and have no other means to verify that the source distribution actually matches the content of the wheel. If we believe that trusting the signatures is enough, I'm fine with it, I just think it needs to be a explict decision, maybe have a small section about it in the design. Some follow up questions:
|
Beta Was this translation helpful? Give feedback.
-
As for RPMs, the builder container's should have the RPM database properly populated so every package will be implicitly verified before its installation, so any 3rd party package would fail installation unless the the corresponding GPG public key is baked into the builder container. So, I wonder what added value we'd add by checking the signatures 1 step ahead, provided checksum verification is in place? As for SRPMs, I imagine the story would be the same. Note that both the lockfile and a set of GPG public keys is input to cachi2 (likely coming from a single source) so it could be all compromised and so we'd have no way of verifying it, essentially again leaving the problem for later stages of the supply chain pipeline to deal with.
That would inherently mean that SRPMs always need to be specified in the lockfile alongside RPMs which isn't going to be the case once the RPM support lands since the vast majority of consumers is only ever going to be interested in doing |
Beta Was this translation helpful? Give feedback.
-
In the context of Konflux, the SRPMs will be made available in source containers, so it could be a way to guarantee we're distributing accurate sources. If we think of Cachi2 as only responsible for fetching, then we could offload this verification to a later part of the pipeline. I'm imagining a scenario in which the SRPMs listed in the lockfile are not correct, even if by mistake of whoever put the file together. Does Cachi2 need to care about it? Maybe not, but let's make this explicit in the design so that whatever pipeline is using Cachi2 can decide how to deal with the consequences.
Interesting, so SRPMs won't be mandatory? I can totally see this use case, I think it makes sense, but I hadn't thought about it before. |
Beta Was this translation helpful? Give feedback.
-
I get that point, but as I explained I don't think cachi2 has enough data to verify this correctly apart from dealing with checksums.
Well, it must be, because a for app containers SRPMs are completely irrelevant, that's been the missing link all along, you can do 'pip install foo' in your builder container, but any platform dependency (e.g. RPM) would have to fetched online and not hermetically. |
Beta Was this translation helpful? Give feedback.
-
|
Generate .repo files on the cachi2's output? Currently, there are these options:
Thoughts? |
Beta Was this translation helpful? Give feedback.
-
|
I like to option proposed by @brunoapimentel above, thus I vote for the 3) compromise |
Beta Was this translation helpful? Give feedback.
-
|
Yeah, that makes sense to me too. The generate-env and inject-files examples really helped illustrate how the createrepo call shouldn't go directly in the fetch-deps command. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks Ralph. Just to clarify ... you mentioned What about |
Beta Was this translation helpful? Give feedback.
-
|
No, I think we're suggesting to move that out of fetch-deps too. I think the criticism is that createrepo is generating new content that's not the same as the repo metadata for the repo that the rpms were fetched from. It would be doing more than just fetching. (We had an idea at one point that maybe we could just literally fetch the remote repo metadata and have cachi2 provide a local copy of it, but @Tojaj pointed out that that remote repo metadata might have full url references in it, which would be fragile.) Unfortunately, that task of creating the repo metadata doesn't fit with an existing cachi2 command. Looks like it will need a new command to me, like Bruno suggested. Maybe The function of the |
Beta Was this translation helpful? Give feedback.
-
|
Seems like this one has been settled. Please take my comment here in consideration when moving forward with a subcommand: #465 (reply in thread) |
Beta Was this translation helpful? Give feedback.
-
|
I updated the design based on your comments. Summary of changes:
|
Beta Was this translation helpful? Give feedback.
-
|
And so on, depending on the package managers used by the request. I believe we'd better still keep the same base layout and add rpms on top of it: |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the comment.
|
Beta Was this translation helpful? Give feedback.
-
Come on, that's just not going to work in practice. You say "doesn't require changes to Containerfile"...yes, but in reality it will because users will need to add packages in two places, and would lead to a desire to deduplicate. Our world really does not need another RPM lockfile format. We have a lot of tools already in this space (xref osbuild/osbuild#1134 ); there's one actively used in fedora-coreos and osbuild too. Lockfile handling should be in dnf. Also, we should add a declarative format there (that isn't a lockfile necessarily). |
Beta Was this translation helpful? Give feedback.
-
|
Using spec file is nice idea, but it means some requirements will be not fulfill. Be honest I always prefer to use existing format rather to create a new one. One of the requirement was to verify that a package that is going in is exactly the same. When a package is from Red Hat distribution we can trust NEVRA, because it represent always the same package even checksum differs (due to a different signature). Architecture is also something that is not fixed in SPEC requires - package abc-0:1-1.x86_64 might be installed by |
Beta Was this translation helpful? Give feedback.
-
|
For any situation in which the input requirements provide multiple solutions...I think we should probably default to erroring out. This happened a while ago, I was using Today dnf will just pick something, but I think we need to require unambiguous inputs in a context like this. I mean, in the end the "lockfile generator" tool is going to need to do the same depsolve! And if it's just using the dnf heuristics, that isn't really better...because it may just suddenly change when the container owner goes to update their container for other reasons. (Also without nice "diffing" logic/tools, it'd be pretty easy to miss package changes in a giant pile of generated YAML) So I guess my opinion here remains that in general, instead of trying to have each container include a giant set of pinned packages, we should do rpm-software-management/dnf5#833 (comment) Which is basically - the base container image (identified by an immutable tag or digest) also pins an immutable set of rpms that will never change - including identifying them by specific sha256 digest - imagine basically we stick the repodata (which includes exact digests) as OCI artifacts. Then by combining two things:
We get the combined desired effects here - and the only thing that needs to be included in each individual container is using e.g.: And as long as they have a |
Beta Was this translation helpful? Give feedback.
-
I don't think I understand how this would work. But the example you gave here makes more sense to me: rpm-software-management/dnf5#833 (comment) base: quay.io/fedora/rpms:40.20240325@sha256:...
packages:
- foo
- bar
- baz
repo-packages:
image: quay.io/fedora/rpms:40.20240320@sha256:...
packages:
- kernel
- iptablesThis is not really that different from the rpms.in.yaml format used in https://github.com/konflux-ci/rpm-lockfile-prototype?tab=readme-ov-file#whats-the-input_file, except that the repos in contentOrigin are "immutable". If all the input repos are immutable, then specifying the direct dependencies should indeed be enough for reproducibility (disregarding potential changes between versions of the tool that does the resolving). If cachi2 could verify that all the sources are immutable, then a lockfile could perhaps be optional (but IMO it should still be a recommended option, as others have expressed in rpm-software-management/dnf5#833). |
Beta Was this translation helpful? Give feedback.
-
I followed up in rpm-software-management/dnf5#833 (comment) Doing that linkage is not strictly required - we can implement "versioned pinned repository lockfiles" without it. The ultimate point here is basically that instead of again having a giant list of RPMs and their sha256sums, we just pin to a set of immutable repos + versions. But it just IMO works much more nicely together to do the pinning from the base image by default, instead of it being an opt-in. |
Beta Was this translation helpful? Give feedback.
-
|
OK so, we've been experimenting with using plain I guess probably these two subthreads should be re combined and taken to an upstream place where we try to really "thin the herd" of the wild array of package lists (mainly YAML or TOML variants) that we have. The "content resolver" stuff example would be pretty high on my list of things that needs to go away and be replaced with something standard. "comps" would be a close second. |
Beta Was this translation helpful? Give feedback.
-
|
Based on the comments #502 I made some changes in the design:
The output now looks like: |
Beta Was this translation helpful? Give feedback.
-
@onosek can you point me to ^this requirement? I may have missed something in this discussion. More specifically, if these fields are not mandatory, how can we check their authenticity and confidently put entries about such packages to the SBOM? |
Beta Was this translation helpful? Give feedback.
-
|
AFAIK the authenticity of packages should be verified by GPG keys and when verified, missing checksum might not be a big issue. But in case we don't have such GPG verification so far (or do we?) checksums should be present. |
Beta Was this translation helpful? Give feedback.
-
AFAIK signing RPMs is not mandatory so rpm might not complain about such package, hence we want to check checksums as a first-level protection. Of course, when GPG keys are in place, rpm will take care of the rest. However, neither seems to be sufficient on its own. That said, I guess the other package managers kind of set the precedent in the SBOM contents. The thing is, if we design this as optional, it is going to become harder in the future to make it mandatory by definition; conversely, one can always relax existing constraints without affecting the user base. |
Beta Was this translation helpful? Give feedback.
-
I stand corrected on this statement. |
Beta Was this translation helpful? Give feedback.
-
|
I think checksums should be mandatory. I would feel a lot more confident about working with a lockfile if I could verify the content and not just the metadata/NEVRA. One thing we've become aware of in image builder a few times is that the NEVRA->checksum mapping is, unfortunately, not always unique (unless my information is outdated,
For RPMs/dnf, the checksum is provided in the package metadata directly so there's no need to compute it when resolving package NEVRA (and dependencies). It's of course not "trivial" to get when writing a lockfile by hand, but for a tool querying a repository, it's quite straightforward (see for example the dnf depsolver for osbuild / image builder https://github.com/osbuild/osbuild/blob/7c04e1c596acf38cf8f3a5983207626c599f4cdd/tools/osbuild-depsolve-dnf). |
Beta Was this translation helpful? Give feedback.
-
Introduction
Containers want to install RPMs at build time. This is not possible when the build doesn’t have access to the negslirtwork (hermetic build). This feature is meant to enable such workflow.
The general idea is that the developer specifies ahead of time what RPMs they want to have available. In a step before the actual build, cachi2 takes this information, downloads all of the packages and prepares a repository that will be available during build.
Requirements
Block schema
The image shows the main function blocks, input and outputs. In the first phase (MVP) only a subset of the whole functionality will be delivered. That means processing the lock file, and gathering (downloading) all required items (RPMs, source RPMs) into a directory structure that will represent input for a container build process. The basic fetch command will produce SBOM with metadata of fetched RPMs too.
As a next (optional) step, the other cachi2 command (
generate-envorinject-files) will process the structure created in the previous step and will generate repositories and .repo file.rpms.lock.yaml
Works as a configuration input file for prefetching. The format allows multiple architectures to be listed.
or values are not mandatory - when not specified, a random repoid name is generated.
checksum key and are not mandatory either - when not specified, (S)RPM file verification won't be performed after download.
An example of the format follows.
Input
Output
The given output path is populated with the following structure. The
"deps/rpm" part - this part of the path is for compatibility with other package manager modules cachi2 works with (for example "deps/npm").
The SBOM file format is CycloneDX and PURLs is generated as per specificiation
Example:
The optional 2nd step will add metadata
Content of cachi2.repo file (with as many repositories included as used in the lockfile):
Note that the repo identifiers in the filesystem structure match the identifiers in the repo file, and come from the lockfile.
If wasn't specified, a random value is generated. And the section will additionally contain the key
namewith an additional comment.The subdirectory for the corresponding architecture can be mounted into the build container to /etc/yum.repos.d/.
Benefits
Important things to keep in mind
What about …
… handling dependencies
It doesn’t. Users are required to include all transitive dependencies in the rpms.lock.yaml file. Ideally, a tool would do this for them.
The prefetching step doesn’t really care about dependencies. It just downloads and prepares what it’s told to.
If there is a missing dependency, there will be a failure at build time.
If the lockfile lists additional packages, the build will not use them, so it’s just wasting time and bandwidth at the prefetching step (and possibly including irrelevant sources in source containers).
Authors
Author: lsedlar
Corrections and addons: tmlcoch, onosek
Beta Was this translation helpful? Give feedback.
All reactions