Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Quickstart #60

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions .air.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
[build]
bin = "main"
dir = "."
cmd = "go build -o main ."
exclude_dir = ["assets", "tmp", "vendor", "testdata"]
exclude_regex = ["_test.go"]
exclude_unchanged = false
follow_symlink = false
run_after_build = "chmod +x main"
include_ext = ["go"]

[log]
level = "debug"
100 changes: 100 additions & 0 deletions QUICKSTART.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
## Quickstart: Local Setup of Harbor Satellite from Source

To set up Harbor Satellite locally, follow these steps:

### 1. Prerequisites
Ensure you have:
- A Harbor registry instance (or similar OCI-compliant registry).
- Credentials with permission to create robot accounts in the registry.
- The latest version of Dagger installed.

### 2. Set Up the Registry
In this guide, we'll use a Harbor registry instance.

- **Registry Login**: Obtain the username and password for your registry, ensuring it has appropriate permissions.

### 3. Configure Ground Control
Navigate to the `ground-control` directory and set up the following environment variables:

```bash
HARBOR_USERNAME=admin
HARBOR_PASSWORD=Harbor12345
HARBOR_URL=https://demo.goharbor.io

PORT=8080
APP_ENV=local

DB_HOST=localhost
DB_PORT=5432
DB_DATABASE=groundcontrol
DB_USERNAME=postgres # Customize based on your DB config
DB_PASSWORD=password # Customize based on your DB config
```
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Security: Replace sensitive credentials with placeholders.

The example contains actual credentials and URLs which could be security risks if copied directly. Consider:

  1. Using placeholders instead of actual credentials
  2. Adding a warning about securing sensitive information
  3. Mentioning credential management best practices

Apply this diff:

-HARBOR_USERNAME=admin
-HARBOR_PASSWORD=Harbor12345
-HARBOR_URL=https://demo.goharbor.io
+HARBOR_USERNAME=<your_harbor_username>
+HARBOR_PASSWORD=<your_harbor_password>
+HARBOR_URL=<your_harbor_url>

 PORT=8080
 APP_ENV=local

 DB_HOST=localhost
 DB_PORT=5432
 DB_DATABASE=groundcontrol
-DB_USERNAME=postgres       # Customize based on your DB config
-DB_PASSWORD=password       # Customize based on your DB config
+DB_USERNAME=<your_db_username>
+DB_PASSWORD=<your_db_password>
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
```bash
HARBOR_USERNAME=admin
HARBOR_PASSWORD=Harbor12345
HARBOR_URL=https://demo.goharbor.io
PORT=8080
APP_ENV=local
DB_HOST=localhost
DB_PORT=5432
DB_DATABASE=groundcontrol
DB_USERNAME=postgres # Customize based on your DB config
DB_PASSWORD=password # Customize based on your DB config
```
```bash
HARBOR_USERNAME=<your_harbor_username>
HARBOR_PASSWORD=<your_harbor_password>
HARBOR_URL=<your_harbor_url>
PORT=8080
APP_ENV=local
DB_HOST=localhost
DB_PORT=5432
DB_DATABASE=groundcontrol
DB_USERNAME=<your_db_username>
DB_PASSWORD=<your_db_password>
```


### 4. Run Ground Control
To start the Ground Control service, execute the following Dagger command:

```bash
dagger call build-dev --component ground-control up
```

> **Note:** Ensure you have set up Dagger with the latest version before running this command. Ground Control will run on port 8080.

### 5. Configure Satellite
Return to the root project directory:

```bash
cd ..
```

Then navigate to the `satellite` directory and verify that `config.toml` is set up correctly:

```toml
# Whether to use the built-in Zot registry or not
bring_own_registry = false

# IP address and port of the registry
own_registry_adr = "127.0.0.1"
own_registry_port = "8585"

# URL of remote registry or local file path
url_or_file = "https://demo.goharbor.io/v2/myproject/album-server"



# Default path for Zot registry config.json
zotConfigPath = "./registry/config.json"

# Set logging level
log_level = "info"
```
Comment on lines +62 to +82
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Enhance configuration documentation with explanations and validation steps.

The configuration section needs:

  1. Explanations for each configuration option
  2. Steps to validate the configuration
  3. A more generic example URL

Apply this diff:

 # Whether to use the built-in Zot registry or not
 bring_own_registry = false
+# Set to true if you want to use an external registry instead of the built-in Zot registry
 
 # IP address and port of the registry
 own_registry_adr = "127.0.0.1"
 own_registry_port = "8585"
+# These settings are used when bring_own_registry = true
 
 # URL of remote registry or local file path
-url_or_file = "https://demo.goharbor.io/v2/myproject/album-server"
+url_or_file = "<your_registry_url>/v2/<project>/<repository>"
+# Format: https://registry.example.com/v2/project/repository
 
 # Default path for Zot registry config.json
 zotConfigPath = "./registry/config.json"
+# This path must exist and be readable
 
 # Set logging level
 log_level = "info"
+# Available levels: debug, info, warn, error

Add validation steps:

To validate your configuration:
1. Verify file permissions for zotConfigPath
2. Test registry connectivity
3. Confirm port availability


### 6. Register the Satellite with Ground Control
Using `curl` or Postman, make a `POST` request to register the Satellite with Ground Control:

```bash
curl -X POST http://localhost:8080/satellites/register -H "Content-Type: application/json" -d '{ "name": "<satellite_name_here>" }'
```

The response will include a token string. Set this token in the Satellite `.env` file:

```console
TOKEN=<string_from_ground_control>
```

### 7. Start the Satellite
Run the following Dagger command to start the Satellite service:

```bash
dagger call build-dev --component satellite up
```

The Satellite service will start on port 9090. Ensure that the `ground_control_url` is correctly set in the Satellite configuration before launching.


### 8. Finalize and Run
After setting the token, you can now run the Satellite. This setup will launch the Satellite in a container with the following exposed ports:
- **9090** for the Satellite service.
- **8585** for the Zot registry (if configured).

With this setup, your Harbor Satellite should be up and running!
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Add error handling and verification steps to the setup process.

The setup instructions should include:

  1. Error handling for registration failures
  2. Health check endpoints
  3. Troubleshooting steps
  4. Verification of successful setup

Add the following sections:

### Verification Steps
1. Check Ground Control health:
   ```bash
   curl http://localhost:8080/health
  1. Verify Satellite registration:
    curl http://localhost:8080/satellites/status

Troubleshooting

Common issues and solutions:

  • Connection refused: Verify ports are not in use
  • Registration fails: Check Ground Control logs
  • Token issues: Verify token format and permissions

Health Monitoring

Monitor these endpoints:


<!-- This is an auto-generated comment by CodeRabbit -->

27 changes: 15 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,9 @@ Harbor Satellite will synchronize with the central Harbor registry, when Interne

Harbor Satellite will also include a toolset enabling the monitoring and management of local decentralized registries.

## QuickStart
Please refer to the latest setup instructions in QUICKSTART.md file for detailed steps on setting up Harbor Satellite locally.

## Non-Goals

T.B.D.
Expand Down Expand Up @@ -51,29 +54,29 @@ Harbor Satellite, at its most basic, will run in a single container and will be

Harbor Satellite may be implemented following 1 or several of 3 different architectures depending on its use cases :

1. **Replicating from a remote registry to a local registry.**
1. **Replicating from a remote registry to a local registry.**
In this basic use case, the stateless satellite component will handle pulling images from a remote registry and then pushing them to the local OCI compliant registry. This local registry will then be accessible to other local edge devices who can pull required images directly from it.
_(A direct access from edge device to the remote registry is still possible when network conditions permit it)._
The satellite component may also handle updating container runtime configurations and fetching image lists from Ground Control, a part of Harbor.
The stateful local regsitry will also need to handle storing and managing data on local volumes.
A typical use case would work as follows :
_(A direct access from edge device to the remote registry is still possible when network conditions permit it)._
The satellite component may also handle updating container runtime configurations and fetching image lists from Ground Control, a part of Harbor.
The stateful local regsitry will also need to handle storing and managing data on local volumes.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Fix typo in "registry" spelling.

There are instances where "registry" is misspelled as "regsitry".

Apply these corrections:

-regsitry
+registry

Also applies to: 68-68

A typical use case would work as follows :
_In an edge computing environment where IoT devices are deployed to a location with limited or no internet connnectivity, these devices need to run containerised images but cannot pull from a central Harbor registry. A local Harbor Satellite instance can be deployed and take up this role while Internet connectivity is unreliable and distribute all required images. Once a reliable connection is re-established, the Harbor Satellite instance will be able to pull required images from its central Harbor registry and thus store up to date images locally._

![Use Case #1](docs/images/satellite_use_case_1.svg)
<p align="center"><em>Use case #1</em></p>

2. **Replicating from a remote regsitry to a local Spegel Registry**
The stateless satellite component send pull instructions to Spegel instances running with each node of a Kubernetes cluster. The node will then directly pull images from a remote registry and share it with other local nodes, removing the need for each of them to individually pull an image from a remote registry.
The network interfaces (boundaries) represented in this use case should and will be the same as those represented in use case #1
A typical use case would work as follows :
2. **Replicating from a remote regsitry to a local Spegel Registry**
The stateless satellite component send pull instructions to Spegel instances running with each node of a Kubernetes cluster. The node will then directly pull images from a remote registry and share it with other local nodes, removing the need for each of them to individually pull an image from a remote registry.
The network interfaces (boundaries) represented in this use case should and will be the same as those represented in use case #1
A typical use case would work as follows :
_In a larger scale edge computing environment with a significant amount of IoT devices needing to run containerised applications, a single local registry in might not be able to handle the increased amount of demands from edge devices. The solution is to deploy several registries to several nodes who are able to automatically replicate images across each other thanks to Spegel instances running together with each node. The Satellite component will use the same interface to instruct each node when, where and how to pull new images that need to be replicated across the cluster._

![Use Case #2](docs/images/satellite_use_case_2.svg)
<p align="center"><em>Use case #2</em></p>

3. **Proxying from a remote regsitry over the local registry**
The stateless satellite component will be in charge of configuring the local OCI compliant registry, which will be running in proxy mode only. This local registry will then handle pulling necessary images from the remote registry and serving them up for use by local edge devices.
A typical use case would work as follows :
3. **Proxying from a remote regsitry over the local registry**
The stateless satellite component will be in charge of configuring the local OCI compliant registry, which will be running in proxy mode only. This local registry will then handle pulling necessary images from the remote registry and serving them up for use by local edge devices.
A typical use case would work as follows :
_When, for a number of possible different reasons, the remote registry side of the diagram would not be able to produce a list of images to push down to the Harbor Satellite, the Satellite would then act as a proxy and forward all requests from edge devices to the remote registry. This ensures the availability of necessary images without the need for a pre-compiled list of images_

![Use Case #3](docs/images/satellite_use_case_3.svg)
Expand Down
36 changes: 36 additions & 0 deletions ci/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,42 @@ const (

type HarborSatellite struct{}

// build-dev function would start the dev server for the component provided.
func (m *HarborSatellite) BuildDev(
ctx context.Context,
// +optional
// +defaultPath="./ground-control"
source *dagger.Directory,
component string,
) (*dagger.Service, error) {
if component == "satellite" || component == "ground-control" {
golang := dag.Container().
From("golang:latest").
WithMountedCache("/go/pkg/mod", dag.CacheVolume("go-mod")).
WithEnvVariable("GOMODCACHE", "/go/pkg/mod").
WithMountedCache("/go/build-cache", dag.CacheVolume("go-build")).
WithEnvVariable("GOCACHE", "/go/build-cache").
WithMountedDirectory(PROJ_MOUNT, source).
WithWorkdir(PROJ_MOUNT).
WithExec([]string{"go", "install", "github.com/air-verse/air@latest"})

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Improve container setup reliability and consistency.

Several issues need attention:

  1. Using "latest" tag can lead to non-reproducible builds
  2. Air tool version should be pinned for consistency
  3. Missing error handling for Air installation

Apply these changes:

-		golang := dag.Container().
-			From("golang:latest").
+		golang := dag.Container().
+			From(DEFAULT_GO).
+			WithExec([]string{"go", "install", "github.com/air-verse/[email protected]"}).
+			WithExec([]string{"which", "air"}).  // Verify installation

Committable suggestion skipped: line range outside the PR's diff.

if component == "ground-control" {
golang = golang.
WithExec([]string{"ls", "-la"}).
// WithWorkdir("./ground-control/").
// WithExec([]string{"go", "mod", "download"}).
WithExec([]string{"ls", "-la"}).
WithExec([]string{"air", "-c", ".air.toml"}).
WithExposedPort(8080)
}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Clean up development setup and improve error handling.

The ground-control setup needs cleanup and improvements:

  1. Remove debug ls commands
  2. Remove or implement commented code
  3. Add validation for Air config
  4. Use constant for port number
 		if component == "ground-control" {
 			golang = golang.
-				WithExec([]string{"ls", "-la"}).
-				// WithWorkdir("./ground-control/").
-				// WithExec([]string{"go", "mod", "download"}).
-				WithExec([]string{"ls", "-la"}).
+				WithExec([]string{"test", "-f", ".air.toml"}).  // Validate config
 				WithExec([]string{"air", "-c", ".air.toml"}).
-				WithExposedPort(8080)
+				WithExposedPort(8080)  // Consider adding a constant for this

Committable suggestion skipped: line range outside the PR's diff.

// to-do: build else part for the satellite

return golang.AsService(), nil
}

return nil, fmt.Errorf("error: please provide component as either satellite or ground-control")
}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codebase verification

Missing Air configuration file and potential setup issues

The verification revealed several concerns:

  • The .air.toml configuration file referenced in the code is missing from both the root and ground-control directories
  • The ground-control module exists with its own go.mod, but the Air development server setup is incomplete
  • The BuildDev method is only defined but not used elsewhere in the codebase

Required actions:

  • Add the missing .air.toml configuration file in the ground-control directory
  • Ensure the Air development server configuration aligns with the Go module structure
  • Consider documenting the BuildDev usage in the codebase
🔗 Analysis chain

Verify Air configuration and method usage.

Let's verify the integration points:

🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Verify Air configuration and BuildDev usage

# Check if Air config exists
echo "Checking Air configuration files..."
fd -t f ".air.toml$"

# Look for BuildDev method usage
echo -e "\nChecking BuildDev method usage..."
rg "BuildDev.*ground-control" -t go

Length of output: 224


Script:

#!/bin/bash
# Let's broaden our search to get more context

# Check for any Air-related configuration files with different extensions
echo "Checking for Air configuration files with various extensions..."
fd -t f "air\.(toml|conf|config|yaml|yml)$"

# Look for BuildDev method usage with a broader search
echo -e "\nChecking BuildDev method usage (broader search)..."
rg "BuildDev" -A 3 -B 1 -t go

# Check the ground-control directory structure
echo -e "\nChecking ground-control directory structure..."
fd . ground-control

# Look for go.mod in ground-control
echo -e "\nChecking for go.mod in ground-control..."
fd "go.mod$" ground-control

Length of output: 2711


// Build function would start the build process for the name provided. Source should be the path to the main.go file.
func (m *HarborSatellite) Build(
ctx context.Context,
Expand Down
13 changes: 13 additions & 0 deletions ground-control/.air.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
[build]
bin = "main"
dir = "."
cmd = "go build -o main ."
exclude_dir = ["assets", "tmp", "vendor", "testdata"]
exclude_regex = ["_test.go"]
exclude_unchanged = false
follow_symlink = false
run_after_build = "chmod +x main"
include_ext = ["go"]
Comment on lines +1 to +10
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Several improvements needed in build configuration

  1. The binary name "main" is too generic and could lead to conflicts. Consider using a more specific name like "ground-control-dev".
  2. The chmod +x command should specify explicit permissions (e.g., 755) for better security.
  3. Setting exclude_unchanged = false means every file change triggers a rebuild, which could impact development performance.
  4. Consider adding more exclude directories like ".git", "node_modules".

Apply these improvements:

[build]
-  bin = "main"
+  bin = "ground-control-dev"
  dir = "."
-  cmd = "go build -o main ."
+  cmd = "go build -o ground-control-dev ."
-  exclude_dir = ["assets", "tmp", "vendor", "testdata"]
+  exclude_dir = ["assets", "tmp", "vendor", "testdata", ".git", "node_modules"]
  exclude_regex = ["_test.go"]
-  exclude_unchanged = false
+  exclude_unchanged = true
  follow_symlink = false
-  run_after_build = "chmod +x main"
+  run_after_build = "chmod 755 ground-control-dev"
  include_ext = ["go"]
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
[build]
bin = "main"
dir = "."
cmd = "go build -o main ."
exclude_dir = ["assets", "tmp", "vendor", "testdata"]
exclude_regex = ["_test.go"]
exclude_unchanged = false
follow_symlink = false
run_after_build = "chmod +x main"
include_ext = ["go"]
[build]
bin = "ground-control-dev"
dir = "."
cmd = "go build -o ground-control-dev ."
exclude_dir = ["assets", "tmp", "vendor", "testdata", ".git", "node_modules"]
exclude_regex = ["_test.go"]
exclude_unchanged = true
follow_symlink = false
run_after_build = "chmod 755 ground-control-dev"
include_ext = ["go"]


[log]
level = "debug"