Support for showing progress (or processing information) while scanning each layer of the image

[yashvardhan-kukreja]

This issue is derived from aquasecurity/trivy#330

Trivy's issue was to display progress while fetching layers but the core logic of fetching and scanning each layer of the provided image is done under fanal and not under trivy. (Trivy calls the Inspect() method of fanal for doing so)

Hence, I am raising this issue and I believe that this capability can be added in fanal (that too in a backwards compatible way) and if it gets merged, then we can use this capability in trivy to display progress while fetching layers.

[knqyf263]

Thank you for copying the issue. Exactly. I added the issue on Trivy to aggregate issues, but it should be implemented in fanal. Here is the relevant part. It depends on google/go-containerregistry.

fanal/artifact/image/image.go

Lines 136 to 163 in 1296f05

	func (a Artifact) uncompressedLayer(diffID string) (string, io.Reader, error) {
	// diffID is a hash of the uncompressed layer
	h, err := v1.NewHash(diffID)
	if err != nil {
	return "", nil, xerrors.Errorf("invalid layer ID (%s): %w", diffID, err)
	}
	layer, err := a.image.LayerByDiffID(h)
	if err != nil {
	return "", nil, xerrors.Errorf("failed to get the layer (%s): %w", diffID, err)
	}
	// digest is a hash of the compressed layer
	var digest string
	if a.isCompressed(layer) {
	d, err := layer.Digest()
	if err != nil {
	return "", nil, xerrors.Errorf("failed to get the digest (%s): %w", diffID, err)
	}
	digest = d.String()
	}
	r, err := layer.Uncompressed()
	if err != nil {
	return "", nil, xerrors.Errorf("failed to get the layer content (%s): %w", diffID, err)
	}
	return digest, r, nil
	}

[yashvardhan-kukreja]

just to confirm, are you sure that it is the function uncompressedLayer() which would involve the logic of showing progress?

Correct me if I am wrong, but in my opinion, the logic of fetching and showing should be in or around the walker.walkLayerTar() function, because it is the sole function which consumes time while the inspection of layer happens. All the other functions associated with Inspect() execute almost instantaneuously.
Hence, the display of progress seems apt only under walker.walkLayerTar().

fanal/walker/tar.go

Lines 19 to 62 in 719c92b

	func WalkLayerTar(layer io.Reader, analyzeFn WalkFunc) ([]string, []string, error) {
	var opqDirs, whFiles []string
	tr := tar.NewReader(layer)
	for {
	hdr, err := tr.Next()
	if err == io.EOF {
	break
	}
	if err != nil {
	return nil, nil, xerrors.Errorf("failed to extract the archive: %w", err)
	}
	filePath := hdr.Name
	filePath = strings.TrimLeft(filepath.Clean(filePath), "/")
	fileDir, fileName := filepath.Split(filePath)
	// e.g. etc/.wh..wh..opq
	if opq == fileName {
	opqDirs = append(opqDirs, fileDir)
	continue
	}
	// etc/.wh.hostname
	if strings.HasPrefix(fileName, wh) {
	name := strings.TrimPrefix(fileName, wh)
	fpath := filepath.Join(fileDir, name)
	whFiles = append(whFiles, fpath)
	continue
	}
	if isIgnored(filePath) {
	continue
	}
	if hdr.Typeflag == tar.TypeSymlink || hdr.Typeflag == tar.TypeLink || hdr.Typeflag == tar.TypeReg {
	err = analyzeFn(filePath, hdr.FileInfo(), tarOnceOpener(tr))
	if err != nil {
	return nil, nil, xerrors.Errorf("failed to analyze file: %w", err)
	}
	}
	}
	return opqDirs, whFiles, nil
	}

[avats-dev]

@yashvardhan-kukreja @knqyf263 can anyone guide me a bit how to resolve this? I want to do this. Thanks.

[yashvardhan-kukreja]

@avats-dev thanks for your willingness to pick it up.
Actually, I ve already hopped onto this and started working around on it.
But you can surely stick around on this thread and subscribe to it to get a good context around this issue hence, understanding a good chunk of codebase for solving any other current/future issues. I hope that's fine with you

Also, for your assistance I ll document my findings time to time on this thread to make it convenient for you to understand what stuff is going around 😊

[avats-dev]

@yashvardhan-kukreja ok, great 👍

[yashvardhan-kukreja]

@knqyf263 The input to the walkLayerTar function is the layer which is of the type io.Reader. Now, the problem is that io.Reader doesn't tell the total size of the object (layer).
So, we can't track percentage of progress/completion of fetching of layers through that.

Although, the Layer interface had a Size() function (denoting the size of the layer) but I tried to look into it and I couldn't quite understand how to correlate the output of Size() function with the execution of walkLayerTar function to track the progress of fetching of layer.

[yashvardhan-kukreja]

So, @knqyf263 , can u tell me if I am going in the right direction or wrong and what should I look next into to implement this? A heads up regarding this would be nice :)