Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: create manifest.wall #142

Merged
merged 88 commits into from
Sep 30, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
88 commits
Select commit Hold shift + click to select a range
5e21a18
feat: create manifest.wall
letFunny Jul 1, 2024
07ca4d7
address review comments
letFunny Jul 2, 2024
af60a6f
feat: parse generate property in sdf
letFunny Jul 2, 2024
55bd81d
redo manifest package to not unmarshall everything
letFunny Jul 2, 2024
e96e87b
feat: internal manifest package
letFunny Jul 2, 2024
26022fd
fix slicer tests
letFunny Jul 2, 2024
dfba7b3
lint errors
letFunny Jul 2, 2024
1ea56d1
remove unnecessary constants
letFunny Jul 2, 2024
1f1177a
do not aggregate manifest twice for tests
letFunny Jul 3, 2024
fe51371
minor rename
letFunny Jul 5, 2024
655b17a
add generate to README
letFunny Jul 5, 2024
401c86e
make validate optional and use structs for prefixes
letFunny Jul 9, 2024
8600db6
indent manifest jsonwall in tests
letFunny Jul 9, 2024
e2c2645
address review comments
letFunny Jul 10, 2024
140ab91
change maps for structs with omitempty, jsonwall iterator
letFunny Jul 10, 2024
3faa139
move path specification to function
letFunny Jul 10, 2024
94ff54a
remove redundant comment
letFunny Jul 10, 2024
131472b
conflict resolution works properly with generate and solve bugfix
letFunny Jul 10, 2024
b8f91bf
make GetGeneratePath private and change name
letFunny Jul 10, 2024
426ca94
remove incomplete error messages
letFunny Jul 10, 2024
e16eb2e
performance optimization
letFunny Jul 10, 2024
eff44b1
remove redundant check that could never occur
letFunny Jul 10, 2024
01f01f2
avoid using slices.Concat which is not in Go 1.21
letFunny Jul 10, 2024
2ee0286
refactor manifest iteration with generic helper
letFunny Jul 12, 2024
a0200f1
restrict generic with interface union type
letFunny Jul 12, 2024
cdaf78d
feat: add IterateContents
letFunny Jul 26, 2024
efa9b44
feat: use higher level API internally in manifest.Validate
letFunny Jul 26, 2024
e6f38d9
validate schema when reading manifest
letFunny Jul 26, 2024
7e19d3f
reword comment per PR suggestion
letFunny Jul 26, 2024
f41c82b
TODO
letFunny Aug 5, 2024
48d58ee
duplication is not an error
letFunny Aug 5, 2024
101c3d4
duplication is ub, not allowed
letFunny Aug 6, 2024
89174b4
Merge branch 'main' into chisel-db-manifest-package
letFunny Aug 6, 2024
301f506
Merge branch 'main' into chisel-db-parse-generate
letFunny Aug 6, 2024
49950a0
documentation and move code for better readability
letFunny Aug 8, 2024
597e3ad
Merge branch 'chisel-db-manifest-package' into chisel-db-create-manifest
letFunny Aug 12, 2024
c4aa806
validate manifest entry, TODO
letFunny Aug 13, 2024
481f69f
pass by pointer instead of value
letFunny Aug 19, 2024
56dd49c
use double quotes
letFunny Aug 19, 2024
8dfcd2b
Read takes io.Reader
letFunny Aug 19, 2024
4e0e33b
remove unnecessary zstd for manifest_test
letFunny Aug 19, 2024
58531e9
address other review comments
letFunny Aug 19, 2024
8fa3b3d
rework algorithm to avoid all the copying into slices
letFunny Aug 19, 2024
b06fdd1
last version of the conflict resolution algorithm
letFunny Aug 20, 2024
1cd1fe1
make failure string order deterministic
letFunny Aug 21, 2024
659218f
remove unnecessary comment and add note about transitiveness
letFunny Aug 27, 2024
53c0785
address review comments
letFunny Aug 28, 2024
e84a5d7
Merge branch 'chisel-db-parse-generate' into chisel-db-create-manifest
letFunny Aug 29, 2024
14850bf
Merge branch 'chisel-db-manifest-package' into chisel-db-create-manifest
letFunny Aug 29, 2024
391119e
working
letFunny Aug 29, 2024
154697b
test mode of manifest
letFunny Aug 29, 2024
cc5b6de
simplify removing extra type
letFunny Aug 29, 2024
6ad27a1
typo
letFunny Aug 29, 2024
475a2c5
simplify
letFunny Aug 29, 2024
c7e9693
remove redundant comments
letFunny Aug 29, 2024
ac59f75
Merge branch 'main' into chisel-db-create-manifest
letFunny Aug 30, 2024
c6f925d
lint
letFunny Aug 30, 2024
9e2ecfb
simplify
letFunny Aug 30, 2024
ee83c93
bugfix: proper until calculation for content created outside packages
letFunny Aug 30, 2024
cb3db41
remove redundant comments
letFunny Aug 30, 2024
93a2218
Merge branch 'main' into chisel-db-create-manifest
letFunny Sep 16, 2024
a9528bd
tidy up
letFunny Sep 16, 2024
3fb0904
move reading manifest to auxiliary function
letFunny Sep 17, 2024
f123d37
testutil.Archive pkgs -> packages
letFunny Sep 17, 2024
d6e048b
make manifest.{FileMode,FileName} private
letFunny Sep 23, 2024
687b21e
archive.Fetch returns PackageInfo
letFunny Sep 23, 2024
212eb46
test for slicer not generating a manifest
letFunny Sep 25, 2024
a662826
rename Hash -> SHA256
letFunny Sep 23, 2024
8a64032
reword comment
letFunny Sep 25, 2024
293fd06
move manifest creation to small functions
letFunny Sep 25, 2024
9b03b95
make LocateManifestSlices public and tested
letFunny Sep 26, 2024
962566a
lint: remove unused variable
letFunny Sep 26, 2024
aaf70b1
add comment to clarify conflict resolution
letFunny Sep 26, 2024
189a3bf
move info logic into helper function
letFunny Sep 27, 2024
7fa6ee9
move report and generateManifest to manifest package
letFunny Sep 27, 2024
07a0549
lint: check error of WalkDir function
letFunny Sep 27, 2024
573427c
manifest.Write interface accepts a Writer
letFunny Sep 27, 2024
f9d1aaf
lint: check error in tests
letFunny Sep 30, 2024
dad0ed7
create report entries for the manifest files to unify logic
letFunny Sep 30, 2024
087d09d
rename LocateManifestSlices to FindPaths
letFunny Sep 30, 2024
a0920b5
manifestAddReport now takes Report
letFunny Sep 30, 2024
2a0371c
better root handling report
letFunny Sep 30, 2024
86c6e7b
Hash -> SHA256
letFunny Sep 30, 2024
e070332
move manifest generation to auxiliary function
letFunny Sep 30, 2024
b39cfe7
better comment
letFunny Sep 30, 2024
8445042
default manifest filename for FindPaths
letFunny Sep 30, 2024
a9ee6ac
remove unused ManifestSlices
letFunny Sep 30, 2024
88a11fa
report cannot be null when writing manifest
letFunny Sep 30, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
165 changes: 165 additions & 0 deletions internal/manifest/manifest.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,16 @@ package manifest
import (
"fmt"
"io"
"io/fs"
"os"
"path/filepath"
"slices"
"sort"
"strings"

"github.com/klauspost/compress/zstd"

"github.com/canonical/chisel/internal/archive"
"github.com/canonical/chisel/internal/jsonwall"
"github.com/canonical/chisel/internal/setup"
)
Expand Down Expand Up @@ -176,6 +182,68 @@ func LocateManifestSlices(slices []*setup.Slice, manifestFileName string) map[st
return manifestSlices
}

type GenerateManifestsOptions struct {
PackageInfo []*archive.PackageInfo
Selection []*setup.Slice
Report *Report
TargetDir string
Filename string
Mode os.FileMode
}

func GenerateManifests(options *GenerateManifestsOptions) error {
manifestSlices := LocateManifestSlices(options.Selection, options.Filename)
if len(manifestSlices) == 0 {
// Nothing to do.
return nil
}
dbw := jsonwall.NewDBWriter(&jsonwall.DBWriterOptions{
Schema: Schema,
})

err := manifestAddPackages(dbw, options.PackageInfo)
if err != nil {
return err
}

err = manifestAddSlices(dbw, options.Selection)
if err != nil {
return err
}

err = manifestAddReport(dbw, options.Report.Entries)
letFunny marked this conversation as resolved.
Show resolved Hide resolved
if err != nil {
return err
}

err = manifestAddManifestPaths(dbw, options.Mode, manifestSlices)
if err != nil {
return err
}

files := []io.Writer{}
for relPath := range manifestSlices {
logf("Generating manifest at %s...", relPath)
absPath := filepath.Join(options.TargetDir, relPath)
if err := os.MkdirAll(filepath.Dir(absPath), 0755); err != nil {
return err
}
file, err := os.OpenFile(absPath, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, options.Mode)
if err != nil {
return err
}
files = append(files, file)
defer file.Close()
}
w, err := zstd.NewWriter(io.MultiWriter(files...))
if err != nil {
return err
}
defer w.Close()
_, err = dbw.WriteTo(w)
return err
}

type prefixable interface {
Path | Content | Package | Slice
}
Expand All @@ -198,3 +266,100 @@ func iteratePrefix[T prefixable](manifest *Manifest, prefix *T, onMatch func(*T)
}
return nil
}

func manifestAddPackages(dbw *jsonwall.DBWriter, infos []*archive.PackageInfo) error {
for _, info := range infos {
err := dbw.Add(&Package{
Kind: "package",
Name: info.Name,
Version: info.Version,
Digest: info.SHA256,
Arch: info.Arch,
})
if err != nil {
return err
}
}
return nil
}

func manifestAddSlices(dbw *jsonwall.DBWriter, slices []*setup.Slice) error {
for _, slice := range slices {
err := dbw.Add(&Slice{
Kind: "slice",
Name: slice.String(),
})
if err != nil {
return err
}
}
return nil
}

func manifestAddReport(dbw *jsonwall.DBWriter, entries map[string]ReportEntry) error {
for _, entry := range entries {
sliceNames := []string{}
for slice := range entry.Slices {
err := dbw.Add(&Content{
Kind: "content",
Slice: slice.String(),
Path: entry.Path,
})
if err != nil {
return err
}
sliceNames = append(sliceNames, slice.String())
}
sort.Strings(sliceNames)
err := dbw.Add(&Path{
Kind: "path",
Path: entry.Path,
Mode: fmt.Sprintf("0%o", unixPerm(entry.Mode)),
Slices: sliceNames,
Hash: entry.Hash,
FinalHash: entry.FinalHash,
letFunny marked this conversation as resolved.
Show resolved Hide resolved
Size: uint64(entry.Size),
Link: entry.Link,
})
if err != nil {
return err
}
}
return nil
}

func manifestAddManifestPaths(dbw *jsonwall.DBWriter, manifestMode os.FileMode, manifestSlices map[string][]*setup.Slice) error {
for path, slices := range manifestSlices {
sliceNames := []string{}
for _, slice := range slices {
err := dbw.Add(&Content{
Kind: "content",
Slice: slice.String(),
Path: path,
})
if err != nil {
return err
}
sliceNames = append(sliceNames, slice.String())
}
sort.Strings(sliceNames)
err := dbw.Add(&Path{
Kind: "path",
Path: path,
Mode: fmt.Sprintf("0%o", unixPerm(manifestMode)),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do we know this mode? The answer is we don't, not here. That's a relatively small issue, but it's also a hint of a larger issue that is quite clear: there's an unfortunate cross-dependency and disconnect right now between the manifest package and the slicer package about how to write stuff to disk.

Let's please try to fix this by:

  1. Introduce fsutil.CreateWriter, which returns (io.WriterCloser, *Entry, error). Let's please do it right, in the sense that we'll have a writerProxy, similar to the readerProxy that we have today, to fill up the hash and size as we go, as that's trivial to do. We won't depend on it, though. See below.

  2. Add the *Entry to the actual Report as soon as we create the *Entry. At this point the entry will still be empty as far as the hash goes, and that's the way it should be as otherwise we'd need to add it at the end, which doesn't work since entries are sorted.

  3. Remove manifestAddManifestPaths and the respective field from the options type.

With these steps, there's nothing special anymore about manifest files here: the slicer cuts it, the report holds it, the manifest writes it down, just like everything else.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One extra detail here: CreateWriter can take the same Options parameter we have in the Create function.

Copy link
Collaborator Author

@letFunny letFunny Sep 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in dad0ed7. Hopefully now everything should be unified and there is nothing special about manifest files.

Slices: sliceNames,
})
if err != nil {
return err
}
}
return nil
}

func unixPerm(mode fs.FileMode) (perm uint32) {
perm = uint32(mode.Perm())
if mode&fs.ModeSticky != 0 {
perm |= 01000
}
return perm
}
Loading
Loading