spdx: initial sbom framwork + spdx encoder #1468

BradLugo · 2025-01-14T03:54:42Z

Description

These changes introduce the initial SBOM framework into claircore, generating an SPDX 2.3 JSON document from a claircore.IndexReport.

Design

Design doc: https://docs.google.com/document/d/1dRKMWjmkpYO5oN_Y-S_0R2vwyvcdXHCNejM3C-F2DIc/edit?tab=t.0

hdonnay · 2025-01-14T15:40:07Z

Should be moved out of the pkg directory.

Rambling on why

The pkg in a package name is just line noise: we all know it's a go package. Marking something for "external" use is done in the opposite direction: internal is actually enforced by the compiler. If the API for use is squishier than that, it's on humans to observe it anyway.

BradLugo · 2025-01-15T19:44:31Z

If anyone has a suggestion on testing something that bakes in time.Now() the output, please let me know. Example of the problem in my current unit test:

            - 		"created":  string("2025-01-14T21:48:35Z"),
            + 		"created":  string("2025-01-14T09:51:40Z"),

hdonnay · 2025-01-15T20:08:56Z

If anyone has a suggestion on testing something that bakes in time.Now() the output, please let me know.

You can put together a cmp.Option for cmp that turns these back into time.Time and allows some delta, or just ignores them.

sbom/sbom.go

sbom/spdx/encoder.go

BradLugo · 2025-01-16T22:35:37Z

I'll start using fixup commits from this point on to help reviewers until I get approvals, then I'll rebase all the fixup commits.

dcaravel

LGTM

sbom/spdx/encoder_test.go

Signed-off-by: Brad Lugo <[email protected]>

sbom/spdx/encoder.go

sbom/sbom.go

sbom/spdx/encoder.go

sbom/spdx/encoder_test.go

go.mod

sbom/spdx/encoder.go

sbom/sbom.go

sbom/spdx/encoder.go

sbom/sbom.go

RTann · 2025-01-30T19:26:40Z

sbom/sbom.go

+	"io"
+)
+
+// Encoder is an interface to convert a claircore.IndexReport into an io.Reader


Suggested change

// Encoder is an interface to convert a claircore.IndexReport into an io.Reader

// Encoder is an interface to write a Software Bill of Materials (SBOM) representation of a [claircore.IndexReport] to an output stream.

Don't forget about this one ^ :)

sbom/spdx/encoder.go

sbom/spdx/encoder_test.go

dcaravel

LGTM, tested on a few images (namely nginx:1.24.0 and quay.io/rhacs-eng/main:4.6.2) comparing the v4.IndexReport from stackrox to the produced SBOMs, and the output was as expected / explainable.

RTann

Approving to ensure this can move along. No true blockers from me, though perhaps the import ordering may be one? I also ask for consideration of the strconv.Atoi comments (do we need these calls?) and the comment about breaking out of the loop

RTann · 2025-02-05T00:26:56Z

sbom/sbom.go

+	"io"
+)
+
+// Encoder is an interface to convert a claircore.IndexReport into an io.Reader


Don't forget about this one ^ :)

sbom/spdx/encoder.go

RTann · 2025-02-05T04:44:05Z

sbom/spdx/encoder.go

+
+		// Record Distributions for this package.
+		if r.Distribution != nil {
+			rDistId, err := strconv.Atoi(r.Distribution.ID)


just realized, do we use the IDs for anything other than converting them into ints? If not, then can we just keep them as strings and make the map key string instead of int? Same idea for the slices. Not sure if we have to keep doing this conversion

We convert them to ints to sort them. The return from IndexReport.IndexRecords() is non-deterministic because it iterates over IndexReport.Packages, which is a map. In order to have a deterministic output, both to test against and for users to be able to run a diff between reports, we need to have some sort of ordering structuring. This is what I came up with, but I'm happy to revisit how we do it (perhaps in a follow up PR 🙂)

Some of this context is captured in a comment here: Some context here: https://github.com/quay/claircore/pull/1468/files#diff-9730b2b5baa2c38c549f0b3c1077fddf654896f316b79c2c8f3f77d5fb7ccd1cR273-R276

so if we just cared about sorting for determinism, then I think we can keep it as strings. If we also want to have the IDs in increasing order, then converting to int makes sense. Feel free to change in a followup if you decide to change it

sbom/spdx/encoder.go

RTann · 2025-02-05T04:55:26Z

sbom/spdx/encoder.go

+					return nil, err
+				}
+
+				srcPkg, ok := pkgs[rSrcPkgId]


won't block the PR over this, but wondering why we add the source here instead of just waiting until we see it? Do we not always see it? In the case we add it here and then see it again later, do we risk potentially duplicating data?

@crozzy will likely be able to explain this better, but my understanding is that source packages aren't always recorded as root-level packages. In fact, I think rhcc source packages are the only ones that are also recorded as root-level packages.
For clarity, when I say "root-level packages" here, I'm referring to IndexReport.Packages as opposed to IndexReport.Packages.Source.

Yeah, I think it depends on what the Scan() method kicks back. If the indexer reports something as a source I think we should add it as a source package in the SBOM.

RTann · 2025-02-05T05:00:54Z

don't forget to sign your commits :) Also looks like there are still a bunch. Seems like you are on top of that, though, with the fixup! prefixes

BradLugo · 2025-02-05T21:13:53Z

don't forget to sign your commits :) Also looks like there are still a bunch. Seems like you are on top of that, though, with the fixup! prefixes

Yup, thank you! It's a new thing I'm trying. The fixup commits have a message body detailing more about the commit and giving reviewers an easier history to work with. Then, when I get approvals from all participating reviewers, I rebase --autosquash at the end to clean up the history and pass CI. Not sure if this is helpful or not. Would love yall's feedback on it!

sbom/sbom.go

crozzy

LGTM, nice work

Adding a function to be able to convert index reports into SPDX documents and SPDX documents into index reports. Signed-off-by: crozzy <[email protected]> Signed-off-by: Brad Lugo <[email protected]>

BradLugo · 2025-02-06T23:42:58Z

autosquashing commits; no changes.

Unavailable for follow up review

BradLugo changed the title ~~spdx: implement initial sbom framwork + spdx encoder~~ spdx: initial sbom framwork + spdx encoder Jan 14, 2025

BradLugo force-pushed the add-spdx-converter branch from 0e7bc81 to ae67e46 Compare January 14, 2025 19:14

BradLugo mentioned this pull request Jan 14, 2025

Initial SBOM framework crozzy/claircore#75

Closed

BradLugo force-pushed the add-spdx-converter branch 3 times, most recently from 11f31ac to be73a74 Compare January 14, 2025 19:28

crozzy reviewed Jan 15, 2025

View reviewed changes

sbom/sbom.go Outdated Show resolved Hide resolved

sbom/spdx/encoder.go Outdated Show resolved Hide resolved

dcaravel reviewed Jan 16, 2025

View reviewed changes

BradLugo force-pushed the add-spdx-converter branch from 3abf86b to aa194f1 Compare January 16, 2025 22:33

BradLugo marked this pull request as ready for review January 16, 2025 22:49

BradLugo requested a review from a team as a code owner January 16, 2025 22:49

BradLugo requested review from hdonnay, dcaravel and crozzy and removed request for a team January 16, 2025 22:49

dcaravel previously approved these changes Jan 16, 2025

View reviewed changes

BradLugo commented Jan 16, 2025

View reviewed changes

sbom/spdx/encoder_test.go Outdated Show resolved Hide resolved

BradLugo marked this pull request as draft January 21, 2025 19:09

BradLugo dismissed dcaravel’s stale review via 6211778 January 21, 2025 23:28

BradLugo force-pushed the add-spdx-converter branch from ac99a3b to 40685d2 Compare January 22, 2025 01:00

sbom: add initial encoder interface

03e369e

Signed-off-by: Brad Lugo <[email protected]>

BradLugo force-pushed the add-spdx-converter branch 3 times, most recently from 29c8071 to f8bb863 Compare January 23, 2025 21:27

BradLugo marked this pull request as ready for review January 23, 2025 21:27

BradLugo force-pushed the add-spdx-converter branch from f8bb863 to 795e5f2 Compare January 23, 2025 21:31

hdonnay previously requested changes Jan 24, 2025

View reviewed changes

sbom/spdx/encoder.go Outdated Show resolved Hide resolved

sbom/spdx/encoder.go Outdated Show resolved Hide resolved

sbom/sbom.go Outdated Show resolved Hide resolved

sbom/spdx/encoder.go Outdated Show resolved Hide resolved

sbom/spdx/encoder_test.go Outdated Show resolved Hide resolved

RTann mentioned this pull request Jan 25, 2025

ROX-27146: SBOM Scanner V4 Matcher API stackrox/stackrox#13756

Merged

6 tasks

RTann reviewed Jan 25, 2025

View reviewed changes

BradLugo requested review from RTann and hdonnay January 28, 2025 21:40

dcaravel reviewed Jan 29, 2025

View reviewed changes

sbom/spdx/encoder.go Outdated Show resolved Hide resolved

sbom/spdx/encoder.go Outdated Show resolved Hide resolved

RTann reviewed Jan 31, 2025

View reviewed changes

dcaravel previously approved these changes Feb 4, 2025

View reviewed changes

BradLugo requested a review from RTann February 4, 2025 23:49

RTann previously approved these changes Feb 5, 2025

View reviewed changes

BradLugo dismissed stale reviews from RTann and dcaravel via 9fcd250 February 5, 2025 21:16

RTann reviewed Feb 5, 2025

View reviewed changes

sbom/sbom.go Outdated Show resolved Hide resolved

BradLugo requested a review from RTann February 6, 2025 18:05

RTann approved these changes Feb 6, 2025

View reviewed changes

crozzy approved these changes Feb 6, 2025

View reviewed changes

spdx: add converter for index reports

0075f4a

Adding a function to be able to convert index reports into SPDX documents and SPDX documents into index reports. Signed-off-by: crozzy <[email protected]> Signed-off-by: Brad Lugo <[email protected]>

BradLugo force-pushed the add-spdx-converter branch from 9127d9d to 0075f4a Compare February 6, 2025 23:42

BradLugo merged commit 79e0035 into quay:main Feb 7, 2025
6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

spdx: initial sbom framwork + spdx encoder #1468

spdx: initial sbom framwork + spdx encoder #1468

BradLugo commented Jan 14, 2025 •

edited

Loading

hdonnay commented Jan 14, 2025 •

edited

Loading

BradLugo commented Jan 15, 2025

hdonnay commented Jan 15, 2025

BradLugo commented Jan 16, 2025

dcaravel left a comment

RTann Jan 30, 2025

RTann Feb 5, 2025

dcaravel left a comment

RTann left a comment

RTann Feb 5, 2025

RTann Feb 5, 2025

BradLugo Feb 5, 2025 •

edited

Loading

RTann Feb 5, 2025

RTann Feb 5, 2025

BradLugo Feb 5, 2025

crozzy Feb 6, 2025

RTann commented Feb 5, 2025 •

edited

Loading

BradLugo commented Feb 5, 2025 •

edited

Loading

crozzy left a comment

BradLugo commented Feb 6, 2025 •

edited

Loading

	// Encoder is an interface to convert a claircore.IndexReport into an io.Reader
	// Encoder is an interface to write a Software Bill of Materials (SBOM) representation of a [claircore.IndexReport] to an output stream.

spdx: initial sbom framwork + spdx encoder #1468

spdx: initial sbom framwork + spdx encoder #1468

Conversation

BradLugo commented Jan 14, 2025 • edited Loading

Description

Design

hdonnay commented Jan 14, 2025 • edited Loading

BradLugo commented Jan 15, 2025

hdonnay commented Jan 15, 2025

BradLugo commented Jan 16, 2025

dcaravel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dcaravel left a comment

Choose a reason for hiding this comment

RTann left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BradLugo Feb 5, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RTann commented Feb 5, 2025 • edited Loading

BradLugo commented Feb 5, 2025 • edited Loading

crozzy left a comment

Choose a reason for hiding this comment

BradLugo commented Feb 6, 2025 • edited Loading

BradLugo commented Jan 14, 2025 •

edited

Loading

hdonnay commented Jan 14, 2025 •

edited

Loading

BradLugo Feb 5, 2025 •

edited

Loading

RTann commented Feb 5, 2025 •

edited

Loading

BradLugo commented Feb 5, 2025 •

edited

Loading

BradLugo commented Feb 6, 2025 •

edited

Loading