Digest bugfix and refactoring

Switched to using crane.Digest to retrieve an image's digest. This simplifies the code and ensures consistency of all stored digests.
These digests can also be used to pull their images, as opposed to the manifest digests retrieved via the v2 API
Updated the README.md to more closely resemble the proposal in goharbor's community repo.
Updated MAKEFILE
Updated images.json to use a correct digest

README.md

@@ -1,64 +1,96 @@
-# Harbor Satellite — Brings Container Registries to the Edge
+# Proposal: `Harbor Satellite`
 
-The project aims to decentralize container registries for better accessibility to edge devices.
-Satellite registries can be used in stateful or stateless mode with the intention to function as a primary registry for the edge location, or as a fallback option if the central registry is unavailable. Satellite is crucial for operating in areas with limited or no internet, or when you need to distribute images to thousands or edge registries.
+Authors: Vadim Bauer / [Vad1mo](https://github.com/Vad1mo), Csaba Almasi, Philip Laine, David Huseby / [dhuseby](https://github.com/dhuseby),  Roald Brunell / [OneFlyingBanana](https://github.com/OneFlyingBanana)
 
-## Primary Use Cases
+## Abstract
 
-- Overcome connectivity issues that affect software distribution to edge locations.
-- Mange distribution of containerized software to edge locations.
-- Managing hundreds or thousands of container registries at edge locations.
-- Works nicely with Kubernetes, yet can work without any container runtime, or as an edge bootstrap instance.
+Harbor Satellite aims to bring Harbor container registries to edge locations, ensuring consistent, available, and integrity-checked images for edge computing environments. This proposal outlines the development of a stateful, standalone satellite that can function as a primary registry for edge locations and as a fallback option if the central Harbor registry is unavailable.
 
-## Why
+## Background
+
+In recent years, containers have extended beyond their traditional cloud environments, becoming increasingly prevalent in remote and edge computing contexts. These environments often lack reliable internet connectivity, posing significant challenges in managing and running containerized applications due to difficulties in fetching container images. To address this, the project aims to decentralize container registries, making them more accessible to edge devices. The need for a satellite that can operate independently, store images on disk, and run indefinitely with stored data is crucial for maintaining operations in areas with limited or no internet connectivity.
+
+## Proposal
+
+The proposed change is to develop "Harbor Satellite", an extension to the existing Harbor container registry. This extension will enable the operation of decentralized registries on edge devices.
+
+Harbor Satellite will synchronize with the central Harbor registry, when Internet connectivity permits it, allowing it to receive and store images. This will ensure that even in environments with limited or unreliable internet connectivity, containerized applications can still fetch their required images from the local Harbor Satellite.
+
+Harbor Satellite will also include a toolset enabling the monitoring and management of local decentralized registries.
+
+## Non-Goals
+
+T.B.D.
+
+## Rationale
+
+Deploying a complete Harbor instance on edge devices in poor/no coverage areas could prove problematic since :
 
-Deploying a complete Harbor instance on edge devices in poor/no coverage areas could prove problematic, since :
 - Harbor wasn't designed to run on edge devices.(e.g. Multiple processes, no unattended mode)
 - Harbor could behave unexpectedly in poor/no connectivity environments.
 - Managing hundreds or thousands of container registries is not operationally feasible with Harbor
 - Harbor would be too similar to a simple registry mirror
 
-Harbor Satellite aims to be resilient, lightweight and will be able to keep functioning independently of Harbor instances.
+Harbor Satellite aims to be resilient, lightweight and will be able to keep functioning independently from Harbor instances.
 
-##  How it Works
+## Compatibility
 
-Harbor Satellite synchronizes with the central Harbor registry, when Internet connectivity permits it, allowing it to receive and store images. This will ensure that even in environments with limited or unreliable internet connectivity, containerized applications can still fetch their required images from the local Harbor Satellite.
+Compatibility with all container registries or edge devices can't be guaranteed.
 
-Harbor Satellite will also include a toolset enabling the monitoring and management of local decentralized registries.
+## Implementation
 
+### Overall Architecture
 
-## Typical Use Cases
+Harbor Satellite, at its most basic, will run in a single container and will be divided in the following 2 components :
 
-### Architecture
-Harbor Satellite, at its most basic, will run in a single container and will be divided into the following 2 components :
-
-- **Satellite** : Is responsible for moving artifacts from upstream, identifying the source and reading the list of images that needs to be replicated. Satellite also modifies the container runtime configuration, so that the container runtime does not fetch images from remote.
-- **OCI Registry** : Is an embedded registry responsible for storing required OCI artifacts locally.
-- **Ground Control** : Is a component of Harbor and is responsible for serving a constructed list of images that need to be present on this edge location.
+- **Satellite** : Is responsible for moving artifacts from upstream (using Skopeo/Crane/Other), identifying the source and reading the list of images that needs to be replicated. Satellite will also be able to modify and manage the container runtimes. configuration in sync so that container runtime does not fetch images from remote.
+- **OCI Registry** : Is responsible for storing required OCI artifacts locally (using zotregistry or docker registry).
+- **Ground Control** : Is a component of Harbor and is responsible for serving a Harbor Satellite with the list of images it needs.
 
 ![Basic Harbor Satellite Diagram](docs/images/harbor-satellite-overview.svg)
 
+<p align="center"><em>Basic Harbor Satellite Diagram</em></p>
+
+### Specific Use Cases
 
-### Replicating From a Remote Registry to the Edge Registry
+Harbor Satellite may be implemented following 1 or several of 3 different architectures depending on its use cases :
 
-In this use case, the stateless satellite component will handle pulling images from a remote registry and then pushing them to the local OCI registry. This local registry will then be accessible to other local edge devices, who can pull required images directly from it.
+1. **Replicating from a remote registry to a local registry.**  
+In this basic use case, the stateless satellite component will handle pulling images from a remote registry and then pushing them to the local OCI compliant registry. This local registry will then be accessible to other local edge devices who can pull required images directly from it.
+_(A direct access from edge device to the remote registry is still possible when network conditions permit it)._  
+The satellite component may also handle updating container runtime configurations and fetching image lists from Ground Control, a part of Harbor.  
+The stateful local regsitry will also need to handle storing and managing data on local volumes.  
+A typical use case would work as follows :  
+_In an edge computing environment where IoT devices are deployed to a location with limited or no internet connnectivity, these devices need to run containerised images but cannot pull from a central Harbor registry. A local Harbor Satellite instance can be deployed and take up this role while Internet connectivity is unreliable and distribute all required images. Once a reliable connection is re-established, the Harbor Satellite instance will be able to pull required images from its central Harbor registry and thus store up to date images locally._
 
 ![Use Case #1](docs/images/satellite_use_case_1.svg)
+<p align="center"><em>Use case #1</em></p>
 
-### Replicating From a Remote Registry to an Edge Kubernetes Registry
+2. **Replicating from a remote regsitry to a local Spegel Registry**  
+The stateless satellite component send pull instructions to Spegel instances running with each node of a Kubernetes cluster. The node will then directly pull images from a remote registry and share it with other local nodes, removing the need for each of them to individually pull an image from a remote registry.  
+The network interfaces (boundaries) represented in this use case should and will be the same as those represented in use case #1  
+A typical use case would work as follows :  
+_In a larger scale edge computing environment with a significant amount of IoT devices needing to run containerised applications, a single local registry in might not be able to handle the increased amount of demands from edge devices. The solution is to deploy several registries to several nodes who are able to automatically replicate images across each other thanks to Spegel instances running together with each node. The Satellite component will use the same interface to instruct each node when, where and how to pull new images that need to be replicated across the cluster._
 
-The stateless satellite component sends pull instructions to Spegel instances running on each Kubernetes node. The node container runtime will then directly pull images from a remote registry to its internal store. Building on Spegel images are now available for other local nodes, removing the need for each of them to individually pull an image from a remote registry.
-This use case only works in Kubernetes environments, the major advantage of such a setup compared to use case #1 is that it allows to operate a stateful registry on a stateless cluster.  The only dependency satellite has is on spegel.
+![Use Case #2](docs/images/satellite_use_case_2.svg)
+<p align="center"><em>Use case #2</em></p>
 
-![Use Case #1](docs/images/satellite_use_case_2.svg)
+3. **Proxying from a remote regsitry over the local registry**  
+The stateless satellite component will be in charge of configuring the local OCI compliant registry, which will be running in proxy mode only. This local registry will then handle pulling necessary images from the remote registry and serving them up for use by local edge devices.  
+A typical use case would work as follows :  
+_When, for a number of possible different reasons, the remote registry side of the diagram would not be able to produce a list of images to push down to the Harbor Satellite, the Satellite would then act as a proxy and forward all requests from edge devices to the remote registry. This ensures the availability of necessary images without the need for a pre-compiled list of images_
 
+![Use Case #3](docs/images/satellite_use_case_3.svg)
+<p align="center"><em>Use case #3</em></p>
 
-### Proxying From a Remote Registry Over to the Edge Registry
-The stateless satellite component will be responsible for configuring the local OCI registry running in proxy mode and the configuration of the container runtime. This local registry is handing, image pulls from the remote registry and serving them up for use by local edge devices.  
-In a highly dynamic environment where the remote registry operator or edge consumer cannot produce a list of images that need to be present on edge. the Satellite can also act as a remote proxy for edge devices. This ensures the availability of necessary images without the need for a pre-compiled list of images.
+### Container Runtime Configuration
 
-![Use Case #1](docs/images/satellite_use_case_3.svg)
+In each of these use cases, we need to ensure that IoT edge devices needing to run containers will be able to access the registry and pull images from it. To solve this issue, we propose 4 solutions :
 
+1. By using **containerd** or **CRI-O** and  configuring a mirror within them.
+2. By setting up an **HTTP Proxy** to manage and optimize pull requests to the registry.
+3. By **directly referencing** the registry.
+4. By **directly referencing** the registry and using Kubernetes' mutating webhooks to point to the correct registry.
 
 ## Development
 
@@ -67,4 +99,5 @@ The project is currently in active development. If you are interested in partici
 ## Community, Discussion, Contribution, and Support
 
 You can reach the Harbor community and developers via the following channels:
-- [#harbor-satellite on CNCF Slack ](https://cloud-native.slack.com/archives/C06NE6EJBU1)
+
+- [#harbor-satellite on CNCF Slack](https://cloud-native.slack.com/archives/C06NE6EJBU1)

image-list/images.json

@@ -5,7 +5,7 @@
       "repository": "myproject",
       "images": [
         {
-          "name": "album-server@sha256:23d6e6de0b63623e44a1ec59746682c77f197702ae77264eb287fa5119256f8a"
+          "name": "album-server@sha256:71df27326a806ef2946ce502d26212efa11d70e4dcea06ceae612eb29cba398b"
         },
         {
           "name": "album-server"

internal/replicate/replicate.go

@@ -40,9 +40,9 @@ func (r *BasicReplicator) Replicate(ctx context.Context, image string) error {
 
 	// TODO: Implement deletion of images from the local registry that are not present in the source registry
 	// Probably use crane.Catalog to get a list of images in the local registry and compare to incoming image list
+	// Then use crane.Delete to delete those images
 
 	source := getPullSource(image)
-	fmt.Println("Source:", source)
 
 	if source != "" {
 		CopyImage(source)
@@ -107,12 +107,7 @@ func CopyImage(imageName string) error {
 	srcRef := imageName
 	destRef := zotUrl + imageName
 
-	// Delete ./local-oci-layout directory if it already exists
-	if err := os.RemoveAll("./local-oci-layout"); err != nil {
-		return fmt.Errorf("failed to remove directory: %w", err)
-	}
-
-	// Pull the image with additional flags and specify a destination directory
+	// Pull the image and specify a destination directory
 	srcImage, err := crane.Pull(srcRef)
 	if err != nil {
 		return fmt.Errorf("failed to pull image: %w", err)
@@ -126,7 +121,12 @@ func CopyImage(imageName string) error {
 		return fmt.Errorf("failed to push image: %w", err)
 	} else {
 		fmt.Println("Image pushed successfully")
+	}
 
+	// Delete ./local-oci-layout directory if it already exists
+	// This is required because it is a temporary directory used by crane to pull and push images to and from
+	if err := os.RemoveAll("./local-oci-layout"); err != nil {
+		return fmt.Errorf("failed to remove directory: %w", err)
 	}
 
 	return nil

internal/store/http-fetch.go

@@ -8,8 +8,10 @@ import (
 	"io"
 	"net/http"
 	"os"
+	"strings"
 
-	v1 "github.com/opencontainers/image-spec/specs-go/v1"
+	"github.com/google/go-containerregistry/pkg/authn"
+	"github.com/google/go-containerregistry/pkg/crane"
 )
 
 type RemoteImageList struct {
@@ -83,45 +85,24 @@ func (client *RemoteImageList) List(ctx context.Context) ([]Image, error) {
 }
 
 func (client *RemoteImageList) GetDigest(ctx context.Context, tag string) (string, error) {
-	// Construct the URL for fetching the manifest
-	url := client.BaseURL + "/manifests/" + tag
+	// Construct the image reference
+	imageRef := fmt.Sprintf("%s:%s", client.BaseURL, tag)
+	// Remove extra characters from the URL
+	imageRef = imageRef[strings.Index(imageRef, "//")+2:]
+	imageRef = strings.ReplaceAll(imageRef, "/v2", "")
 
 	// Encode credentials for Basic Authentication
 	username := os.Getenv("HARBOR_USERNAME")
 	password := os.Getenv("HARBOR_PASSWORD")
-	auth := base64.StdEncoding.EncodeToString([]byte(username + ":" + password))
-
-	// Create a new HTTP request
-	req, err := http.NewRequest("GET", url, nil)
-	if err != nil {
-		return "", fmt.Errorf("failed to create request: %w", err)
-	}
-
-	// Set the Authorization header
-	req.Header.Set("Authorization", "Basic "+auth)
-
-	req.Header.Add("Accept", "application/vnd.oci.image.manifest.v1+json")
-
-	// Send the request
-	httpClient := &http.Client{}
-	resp, err := httpClient.Do(req)
-	if err != nil {
-		return "", fmt.Errorf("failed to fetch manifest: %w", err)
-	}
-	defer resp.Body.Close()
 
-	// Read the response body
-	body, err := io.ReadAll(resp.Body)
+	// Use crane.Digest to get the digest of the image
+	digest, err := crane.Digest(imageRef, crane.WithAuth(&authn.Basic{
+		Username: username,
+		Password: password,
+	}))
 	if err != nil {
-		return "", fmt.Errorf("failed to read response body: %w", err)
-	}
-
-	// Unmarshal the JSON response
-	var manifestResponse v1.Manifest
-	if err := json.Unmarshal(body, &manifestResponse); err != nil {
-		return "", fmt.Errorf("failed to unmarshal JSON response: %w", err)
+		return "", fmt.Errorf("failed to fetch digest: %w", err)
 	}
 
-	// Return the digest from the config section of the response
-	return string(manifestResponse.Config.Digest), nil
+	return digest, nil
 }