Merge branch 'main' into decouple-zot-registry

.github/labeler.yml

@@ -0,0 +1,8 @@
+# Add the documentation label to all PRs that modify md files
+documentation:
+- changed-files:
+  - any-glob-to-any-file: '**/*.md'
+# Add the golang label to all PRs that modify go files
+golang:
+- changed-files:
+  - any-glob-to-any-file: '**/*.go'

.github/workflows/labeler.yaml

@@ -0,0 +1,13 @@
+name: "Pull Request Labeler"
+on:
+- pull_request_target
+
+jobs:
+  labeler:
+    permissions:
+      contents: read
+      pull-requests: write
+    runs-on: ubuntu-latest
+    steps:
+    - name: Label PR
+      uses: actions/labeler@v5

.github/workflows/lint.yaml

@@ -4,11 +4,10 @@ on:
     branches:
       - main
     types: [submitted]
-    paths:
-      - '**/*.go'
 jobs:
   lint:
-    if: github.event.review.state == 'approved'
+    # the ON keyword is ANY not AND for conditions regarding path. Need to work around with labelling for PRs. 
+    if: contains(github.event.pull_request.labels.*.name, 'golang') && github.event.review.state == 'approved'
     runs-on: ubuntu-latest
     steps:
     - name: Checkout repository

README.md

@@ -1,6 +1,70 @@
-# Harbor Satellite - Brings Container Registries to the EDGE
+# Harbor Satellite — Brings Container Registries to the Edge
 
-Satellite is an extension to [Harbor](https://goharbor.io/) that allows you
-to deploy and operate a lightweight registry on edge locations.
-That can be managed and populated with OCI artifacts by a central Harbor registry.
+The project aims to decentralize container registries for better accessibility to edge devices.
+Satellite registries can be used in stateful or stateless mode with the intention to function as a primary registry for the edge location, or as a fallback option if the central registry is unavailable. Satellite is crucial for operating in areas with limited or no internet, or when you need to distribute images to thousands or edge registries.
 
+## Primary Use Cases
+
+- Overcome connectivity issues that affect software distribution to edge locations.
+- Mange distribution of containerized software to edge locations.
+- Managing hundreds or thousands of container registries at edge locations.
+- Works nicely with Kubernetes, yet can work without any container runtime, or as an edge bootstrap instance.
+
+## Why
+
+Deploying a complete Harbor instance on edge devices in poor/no coverage areas could prove problematic, since :
+- Harbor wasn't designed to run on edge devices.(e.g. Multiple processes, no unattended mode)
+- Harbor could behave unexpectedly in poor/no connectivity environments.
+- Managing hundreds or thousands of container registries is not operationally feasible with Harbor
+- Harbor would be too similar to a simple registry mirror
+
+Harbor Satellite aims to be resilient, lightweight and will be able to keep functioning independently of Harbor instances.
+
+##  How it Works
+
+Harbor Satellite synchronizes with the central Harbor registry, when Internet connectivity permits it, allowing it to receive and store images. This will ensure that even in environments with limited or unreliable internet connectivity, containerized applications can still fetch their required images from the local Harbor Satellite.
+
+Harbor Satellite will also include a toolset enabling the monitoring and management of local decentralized registries.
+
+
+## Typical Use Cases
+
+### Architecture
+Harbor Satellite, at its most basic, will run in a single container and will be divided into the following 2 components :
+
+- **Satellite** : Is responsible for moving artifacts from upstream, identifying the source and reading the list of images that needs to be replicated. Satellite also modifies the container runtime configuration, so that the container runtime does not fetch images from remote.
+- **OCI Registry** : Is an embedded registry responsible for storing required OCI artifacts locally.
+- **Ground Control** : Is a component of Harbor and is responsible for serving a constructed list of images that need to be present on this edge location.
+
+![Basic Harbor Satellite Diagram](docs/images/harbor-satellite-overview.svg)
+
+
+### Replicating From a Remote Registry to the Edge Registry
+
+In this use case, the stateless satellite component will handle pulling images from a remote registry and then pushing them to the local OCI registry. This local registry will then be accessible to other local edge devices, who can pull required images directly from it.
+
+![Use Case #1](docs/images/satellite_use_case_1.svg)
+
+### Replicating From a Remote Registry to an Edge Kubernetes Registry
+
+The stateless satellite component sends pull instructions to Spegel instances running on each Kubernetes node. The node container runtime will then directly pull images from a remote registry to its internal store. Building on Spegel images are now available for other local nodes, removing the need for each of them to individually pull an image from a remote registry.
+This use case only works in Kubernetes environments, the major advantage of such a setup compared to use case #1 is that it allows to operate a stateful registry on a stateless cluster.  The only dependency satellite has is on spegel.
+
+![Use Case #1](docs/images/satellite_use_case_2.svg)
+
+
+### Proxying From a Remote Registry Over to the Edge Registry
+The stateless satellite component will be responsible for configuring the local OCI registry running in proxy mode and the configuration of the container runtime. This local registry is handing, image pulls from the remote registry and serving them up for use by local edge devices.  
+In a highly dynamic environment where the remote registry operator or edge consumer cannot produce a list of images that need to be present on edge. the Satellite can also act as a remote proxy for edge devices. This ensures the availability of necessary images without the need for a pre-compiled list of images.
+
+![Use Case #1](docs/images/satellite_use_case_3.svg)
+
+
+## Development
+
+The project is currently in active development. If you are interested in participating or using the product, [reach out](https://container-registry.com/contact/).
+
+## Community, Discussion, Contribution, and Support
+
+You can reach the Harbor community and developers via the following channels:
+- [#harbor-satellite on CNCF Slack ](https://cloud-native.slack.com/archives/C06NE6EJBU1)

docs/decisions/0001-skopeo-vs-crane.md

@@ -0,0 +1,24 @@
+# Skopeo vs Crane
+
+## Context and Problem Statement
+
+We want to decide wether to use Skopeo or Crane to replicate images from a remote source registry to a local destination registry.
+
+## Considered Options
+
+* [Skopeo](https://github.com/containers/skopeo)
+* [Crane](https://github.com/google/go-containerregistry/tree/main/cmd/crane)
+
+## Decision Outcome
+
+Chosen option: "Crane", because according to our use cases _(listed in the table below)_, Crane works for all of them :
+
+| Use Case | Skopeo | Crane |
+|---|---|---|
+| **Programmable, can be used as an SDK** | No (CLI only or via wrapping) | Yes (CLI + library) |
+| **Configuration can be done via API** | No | Yes |
+| **Move OCI Artifacts** | Yes (copy + delete) | Yes (copy + delete) |
+| **Remapping Artifacts (programmatically)** _( = modify or move image)_ | No | Yes |
+| **Image Consistency Verification** (was it really pulled correctly, is it present) | Yes (inspect manifest and compare hashes) | Yes (validate well-formed image + compare digest of image) |
+| **Memory and CPU consumption on low end devices, Disk consumption** | "Very lightweight CPU usage" / 37.1MB | "Lightweight CPU usage" / 34.7MB |
+| **Dealing with Low bandwidth networks, resume, etc.** | Can report transfer rate and progress per blob, can use `--retry`/`--retry-times` flags, evidence of people using it with [intermittent networking issues](https://github.com/containers/common/issues/654) | No resumable push -> works with single PATCH request, as per [this issue](https://github.com/google/go-containerregistry/issues/1448), can use `.withRetry`, support for [very slow speeds via buffering](https://github.com/google/go-containerregistry/issues/920) / [here](https://github.com/google/go-containerregistry/pull/923) |

docs/decisions/0002-zot-vs-docker-registries.md

@@ -0,0 +1,28 @@
+# Zot registry vs Docker registry
+
+## Context and Problem Statement
+
+In order to start development of Harbor Satellite's first use case, we need to choose which local registry we want to use. While both (and other solutions) are viable, we want to focus on getting started quickly, so deployment and usability ease are our temporary focus.
+
+## Considered Options
+
+* [Zot registry](https://zotregistry.dev/v2.0.3/)
+* [Docker registry](https://hub.docker.com/_/registry)
+
+## Decision Outcome
+
+Chosen option: " ", because after considering elements in the table below, it was the best choice.
+
+| Feature                                | Docker Registry                           | Zot                             |
+|----------------------------------------|-------------------------------------------|---------------------------------|
+| Ease of setup                          | Very easy - with Docker, single CLI command | Easy/Very Easy - with binary (build from source or download) -> run with Docker / with Docker, single CLI command |
+| Lightweight                            | Needs Docker                              | Can run as host-level service from single binary |
+| Fully OCI compliant                    | Yes                                       | Yes                             |
+| Built-in authentication                | Yes                                       | Yes                             |
+| Garbage collection                     | Yes                                       | Yes                             |
+| Storage deduplication                  | No                                        | Yes                             |
+| Signing + verifying container images   | Docker Content Trust  (more complex)      | Cosign built-in                 |
+| Attaching metadata to container images | "Annotations" can work similarly          | Notation built-in               |
+| GUI                                    | No                                        | Yes                             |
+
+[Interesting video about Zot](https://www.youtube.com/watch?v=zOjOF00aQSY&t=4s)

docs/decisions/README.md

@@ -0,0 +1,7 @@
+# Decisions
+
+This directory contains decision records for Harbor Satellite.
+
+For new ADRs, please use [adr-template.md](adr-template.md) or [0000-use-markdown-any-decision-records.md](0000-use-markdown-any-decision-records.md) as basis.
+More information on MADR is available at <https://adr.github.io/madr/>.
+General information about architectural decision records is available at <https://adr.github.io/>.

docs/decisions/_adr-template-long.md

@@ -0,0 +1,79 @@
+---
+# These are optional elements. Feel free to remove any of them.
+status: {proposed | rejected | accepted | deprecated | … | superseded by [ADR-0005](0005-example.md)}
+date: {YYYY-MM-DD when the decision was last updated}
+deciders: {list everyone involved in the decision}
+consulted: {list everyone whose opinions are sought (typically subject-matter experts); and with whom there is a two-way communication}
+informed: {list everyone who is kept up-to-date on progress; and with whom there is a one-way communication}
+---
+# {short title of solved problem and solution}
+
+## Context and Problem Statement
+
+{Describe the context and problem statement, e.g., in free form using two to three sentences or in the form of an illustrative story.
+ You may want to articulate the problem in form of a question and add links to collaboration boards or issue management systems.}
+
+<!-- This is an optional element. Feel free to remove. -->
+## Decision Drivers
+
+* {decision driver 1, e.g., a force, facing concern, …}
+* {decision driver 2, e.g., a force, facing concern, …}
+* … <!-- numbers of drivers can vary -->
+
+## Considered Options
+
+* {title of option 1}
+* {title of option 2}
+* {title of option 3}
+* … <!-- numbers of options can vary -->
+
+## Decision Outcome
+
+Chosen option: "{title of option 1}", because
+{justification. e.g., only option, which meets k.o. criterion decision driver | which resolves force {force} | … | comes out best (see below)}.
+
+<!-- This is an optional element. Feel free to remove. -->
+### Consequences
+
+* Good, because {positive consequence, e.g., improvement of one or more desired qualities, …}
+* Bad, because {negative consequence, e.g., compromising one or more desired qualities, …}
+* … <!-- numbers of consequences can vary -->
+
+<!-- This is an optional element. Feel free to remove. -->
+## Validation
+
+{describe how the implementation of/compliance with the ADR is validated. E.g., by a review or an ArchUnit test}
+
+<!-- This is an optional element. Feel free to remove. -->
+## Pros and Cons of the Options
+
+### {title of option 1}
+
+<!-- This is an optional element. Feel free to remove. -->
+{example | description | pointer to more information | …}
+
+* Good, because {argument a}
+* Good, because {argument b}
+<!-- use "neutral" if the given argument weights neither for good nor bad -->
+* Neutral, because {argument c}
+* Bad, because {argument d}
+* … <!-- numbers of pros and cons can vary -->
+
+### {title of other option}
+
+{example | description | pointer to more information | …}
+
+* Good, because {argument a}
+* Good, because {argument b}
+* Neutral, because {argument c}
+* Bad, because {argument d}
+* …
+
+<!-- This is an optional element. Feel free to remove. -->
+## More Information
+
+{You might want to provide additional evidence/confidence for the decision outcome here and/or
+ document the team agreement on the decision and/or
+ define when this decision when and how the decision should be realized and if/when it should be re-visited and/or
+ how the decision is validated.
+ Links to other decisions and resources might here appear as well.}

docs/decisions/_adr-template-short.md

@@ -0,0 +1,26 @@
+# Use Markdown Any Decision Records
+
+## Context and Problem Statement
+
+We want to record any decisions made in this project independent whether decisions concern the architecture ("architectural decision record"), the code, or other fields.
+Which format and structure should these records follow?
+
+## Considered Options
+
+* [MADR](https://adr.github.io/madr/) 3.0.0 – The Markdown Any Decision Records
+* [Michael Nygard's template](http://thinkrelevance.com/blog/2011/11/15/documenting-architecture-decisions) – The first incarnation of the term "ADR"
+* [Sustainable Architectural Decisions](https://www.infoq.com/articles/sustainable-architectural-design-decisions) – The Y-Statements
+* Other templates listed at <https://github.com/joelparkerhenderson/architecture_decision_record>
+* Formless – No conventions for file format and structure
+
+## Decision Outcome
+
+Chosen option: "MADR 3.0.0", because
+
+* Implicit assumptions should be made explicit.
+  Design documentation is important to enable people understanding the decisions later on.
+  See also [A rational design process: How and why to fake it](https://doi.org/10.1109/TSE.1986.6312940).
+* MADR allows for structured capturing of any decision.
+* The MADR format is lean and fits our development style.
+* The MADR structure is comprehensible and facilitates usage & maintenance.
+* The MADR project is vivid.