-
Notifications
You must be signed in to change notification settings - Fork 8
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Quickstart #60
base: main
Are you sure you want to change the base?
Add Quickstart #60
Changes from 2 commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,13 @@ | ||
[build] | ||
bin = "main" | ||
dir = "." | ||
cmd = "go build -o main ." | ||
exclude_dir = ["assets", "tmp", "vendor", "testdata"] | ||
exclude_regex = ["_test.go"] | ||
exclude_unchanged = false | ||
follow_symlink = false | ||
run_after_build = "chmod +x main" | ||
include_ext = ["go"] | ||
|
||
[log] | ||
level = "debug" |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,100 @@ | ||
## Quickstart: Local Setup of Harbor Satellite from Source | ||
|
||
To set up Harbor Satellite locally, follow these steps: | ||
|
||
### 1. Prerequisites | ||
Ensure you have: | ||
- A Harbor registry instance (or similar OCI-compliant registry). | ||
- Credentials with permission to create robot accounts in the registry. | ||
- The latest version of Dagger installed. | ||
|
||
### 2. Set Up the Registry | ||
In this guide, we'll use a Harbor registry instance. | ||
|
||
- **Registry Login**: Obtain the username and password for your registry, ensuring it has appropriate permissions. | ||
|
||
### 3. Configure Ground Control | ||
Navigate to the `ground-control` directory and set up the following environment variables: | ||
|
||
```bash | ||
HARBOR_USERNAME=admin | ||
HARBOR_PASSWORD=Harbor12345 | ||
HARBOR_URL=https://demo.goharbor.io | ||
|
||
PORT=8080 | ||
APP_ENV=local | ||
|
||
DB_HOST=localhost | ||
DB_PORT=5432 | ||
DB_DATABASE=groundcontrol | ||
DB_USERNAME=postgres # Customize based on your DB config | ||
DB_PASSWORD=password # Customize based on your DB config | ||
``` | ||
|
||
### 4. Run Ground Control | ||
To start the Ground Control service, execute the following Dagger command: | ||
|
||
```bash | ||
dagger call build-dev --component ground-control up | ||
``` | ||
|
||
> **Note:** Ensure you have set up Dagger with the latest version before running this command. Ground Control will run on port 8080. | ||
|
||
### 5. Configure Satellite | ||
Return to the root project directory: | ||
|
||
```bash | ||
cd .. | ||
``` | ||
|
||
Then navigate to the `satellite` directory and verify that `config.toml` is set up correctly: | ||
|
||
```toml | ||
# Whether to use the built-in Zot registry or not | ||
bring_own_registry = false | ||
|
||
# IP address and port of the registry | ||
own_registry_adr = "127.0.0.1" | ||
own_registry_port = "8585" | ||
|
||
# URL of remote registry or local file path | ||
url_or_file = "https://demo.goharbor.io/v2/myproject/album-server" | ||
|
||
|
||
|
||
# Default path for Zot registry config.json | ||
zotConfigPath = "./registry/config.json" | ||
|
||
# Set logging level | ||
log_level = "info" | ||
``` | ||
Comment on lines
+62
to
+82
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🛠️ Refactor suggestion Enhance configuration documentation with explanations and validation steps. The configuration section needs:
Apply this diff: # Whether to use the built-in Zot registry or not
bring_own_registry = false
+# Set to true if you want to use an external registry instead of the built-in Zot registry
# IP address and port of the registry
own_registry_adr = "127.0.0.1"
own_registry_port = "8585"
+# These settings are used when bring_own_registry = true
# URL of remote registry or local file path
-url_or_file = "https://demo.goharbor.io/v2/myproject/album-server"
+url_or_file = "<your_registry_url>/v2/<project>/<repository>"
+# Format: https://registry.example.com/v2/project/repository
# Default path for Zot registry config.json
zotConfigPath = "./registry/config.json"
+# This path must exist and be readable
# Set logging level
log_level = "info"
+# Available levels: debug, info, warn, error Add validation steps: To validate your configuration:
1. Verify file permissions for zotConfigPath
2. Test registry connectivity
3. Confirm port availability |
||
|
||
### 6. Register the Satellite with Ground Control | ||
Using `curl` or Postman, make a `POST` request to register the Satellite with Ground Control: | ||
|
||
```bash | ||
curl -X POST http://localhost:8080/satellites/register -H "Content-Type: application/json" -d '{ "name": "<satellite_name_here>" }' | ||
``` | ||
|
||
The response will include a token string. Set this token in the Satellite `.env` file: | ||
|
||
```console | ||
TOKEN=<string_from_ground_control> | ||
``` | ||
|
||
### 7. Start the Satellite | ||
Run the following Dagger command to start the Satellite service: | ||
|
||
```bash | ||
dagger call build-dev --component satellite up | ||
``` | ||
|
||
The Satellite service will start on port 9090. Ensure that the `ground_control_url` is correctly set in the Satellite configuration before launching. | ||
|
||
|
||
### 8. Finalize and Run | ||
After setting the token, you can now run the Satellite. This setup will launch the Satellite in a container with the following exposed ports: | ||
- **9090** for the Satellite service. | ||
- **8585** for the Zot registry (if configured). | ||
|
||
With this setup, your Harbor Satellite should be up and running! | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🛠️ Refactor suggestion Add error handling and verification steps to the setup process. The setup instructions should include:
Add the following sections: ### Verification Steps
1. Check Ground Control health:
```bash
curl http://localhost:8080/health
TroubleshootingCommon issues and solutions:
Health MonitoringMonitor these endpoints:
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -14,6 +14,9 @@ Harbor Satellite will synchronize with the central Harbor registry, when Interne | |
|
||
Harbor Satellite will also include a toolset enabling the monitoring and management of local decentralized registries. | ||
|
||
## QuickStart | ||
Please refer to the latest setup instructions in QUICKSTART.md file for detailed steps on setting up Harbor Satellite locally. | ||
|
||
## Non-Goals | ||
|
||
T.B.D. | ||
|
@@ -51,29 +54,29 @@ Harbor Satellite, at its most basic, will run in a single container and will be | |
|
||
Harbor Satellite may be implemented following 1 or several of 3 different architectures depending on its use cases : | ||
|
||
1. **Replicating from a remote registry to a local registry.** | ||
1. **Replicating from a remote registry to a local registry.** | ||
In this basic use case, the stateless satellite component will handle pulling images from a remote registry and then pushing them to the local OCI compliant registry. This local registry will then be accessible to other local edge devices who can pull required images directly from it. | ||
_(A direct access from edge device to the remote registry is still possible when network conditions permit it)._ | ||
The satellite component may also handle updating container runtime configurations and fetching image lists from Ground Control, a part of Harbor. | ||
The stateful local regsitry will also need to handle storing and managing data on local volumes. | ||
A typical use case would work as follows : | ||
_(A direct access from edge device to the remote registry is still possible when network conditions permit it)._ | ||
The satellite component may also handle updating container runtime configurations and fetching image lists from Ground Control, a part of Harbor. | ||
The stateful local regsitry will also need to handle storing and managing data on local volumes. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fix typo in "registry" spelling. There are instances where "registry" is misspelled as "regsitry". Apply these corrections: -regsitry
+registry Also applies to: 68-68 |
||
A typical use case would work as follows : | ||
_In an edge computing environment where IoT devices are deployed to a location with limited or no internet connnectivity, these devices need to run containerised images but cannot pull from a central Harbor registry. A local Harbor Satellite instance can be deployed and take up this role while Internet connectivity is unreliable and distribute all required images. Once a reliable connection is re-established, the Harbor Satellite instance will be able to pull required images from its central Harbor registry and thus store up to date images locally._ | ||
|
||
![Use Case #1](docs/images/satellite_use_case_1.svg) | ||
<p align="center"><em>Use case #1</em></p> | ||
|
||
2. **Replicating from a remote regsitry to a local Spegel Registry** | ||
The stateless satellite component send pull instructions to Spegel instances running with each node of a Kubernetes cluster. The node will then directly pull images from a remote registry and share it with other local nodes, removing the need for each of them to individually pull an image from a remote registry. | ||
The network interfaces (boundaries) represented in this use case should and will be the same as those represented in use case #1 | ||
A typical use case would work as follows : | ||
2. **Replicating from a remote regsitry to a local Spegel Registry** | ||
The stateless satellite component send pull instructions to Spegel instances running with each node of a Kubernetes cluster. The node will then directly pull images from a remote registry and share it with other local nodes, removing the need for each of them to individually pull an image from a remote registry. | ||
The network interfaces (boundaries) represented in this use case should and will be the same as those represented in use case #1 | ||
A typical use case would work as follows : | ||
_In a larger scale edge computing environment with a significant amount of IoT devices needing to run containerised applications, a single local registry in might not be able to handle the increased amount of demands from edge devices. The solution is to deploy several registries to several nodes who are able to automatically replicate images across each other thanks to Spegel instances running together with each node. The Satellite component will use the same interface to instruct each node when, where and how to pull new images that need to be replicated across the cluster._ | ||
|
||
![Use Case #2](docs/images/satellite_use_case_2.svg) | ||
<p align="center"><em>Use case #2</em></p> | ||
|
||
3. **Proxying from a remote regsitry over the local registry** | ||
The stateless satellite component will be in charge of configuring the local OCI compliant registry, which will be running in proxy mode only. This local registry will then handle pulling necessary images from the remote registry and serving them up for use by local edge devices. | ||
A typical use case would work as follows : | ||
3. **Proxying from a remote regsitry over the local registry** | ||
The stateless satellite component will be in charge of configuring the local OCI compliant registry, which will be running in proxy mode only. This local registry will then handle pulling necessary images from the remote registry and serving them up for use by local edge devices. | ||
A typical use case would work as follows : | ||
_When, for a number of possible different reasons, the remote registry side of the diagram would not be able to produce a list of images to push down to the Harbor Satellite, the Satellite would then act as a proxy and forward all requests from edge devices to the remote registry. This ensures the availability of necessary images without the need for a pre-compiled list of images_ | ||
|
||
![Use Case #3](docs/images/satellite_use_case_3.svg) | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -20,6 +20,42 @@ const ( | |
|
||
type HarborSatellite struct{} | ||
|
||
// build-dev function would start the dev server for the component provided. | ||
func (m *HarborSatellite) BuildDev( | ||
ctx context.Context, | ||
// +optional | ||
// +defaultPath="./ground-control" | ||
source *dagger.Directory, | ||
component string, | ||
) (*dagger.Service, error) { | ||
if component == "satellite" || component == "ground-control" { | ||
golang := dag.Container(). | ||
From("golang:latest"). | ||
WithMountedCache("/go/pkg/mod", dag.CacheVolume("go-mod")). | ||
WithEnvVariable("GOMODCACHE", "/go/pkg/mod"). | ||
WithMountedCache("/go/build-cache", dag.CacheVolume("go-build")). | ||
WithEnvVariable("GOCACHE", "/go/build-cache"). | ||
WithMountedDirectory(PROJ_MOUNT, source). | ||
WithWorkdir(PROJ_MOUNT). | ||
WithExec([]string{"go", "install", "github.com/air-verse/air@latest"}) | ||
|
||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Improve container setup reliability and consistency. Several issues need attention:
Apply these changes: - golang := dag.Container().
- From("golang:latest").
+ golang := dag.Container().
+ From(DEFAULT_GO).
+ WithExec([]string{"go", "install", "github.com/air-verse/[email protected]"}).
+ WithExec([]string{"which", "air"}). // Verify installation
|
||
if component == "ground-control" { | ||
golang = golang. | ||
WithExec([]string{"ls", "-la"}). | ||
// WithWorkdir("./ground-control/"). | ||
// WithExec([]string{"go", "mod", "download"}). | ||
WithExec([]string{"ls", "-la"}). | ||
WithExec([]string{"air", "-c", ".air.toml"}). | ||
WithExposedPort(8080) | ||
} | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🛠️ Refactor suggestion Clean up development setup and improve error handling. The ground-control setup needs cleanup and improvements:
if component == "ground-control" {
golang = golang.
- WithExec([]string{"ls", "-la"}).
- // WithWorkdir("./ground-control/").
- // WithExec([]string{"go", "mod", "download"}).
- WithExec([]string{"ls", "-la"}).
+ WithExec([]string{"test", "-f", ".air.toml"}). // Validate config
WithExec([]string{"air", "-c", ".air.toml"}).
- WithExposedPort(8080)
+ WithExposedPort(8080) // Consider adding a constant for this
|
||
// to-do: build else part for the satellite | ||
|
||
return golang.AsService(), nil | ||
} | ||
|
||
return nil, fmt.Errorf("error: please provide component as either satellite or ground-control") | ||
} | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 💡 Codebase verification Missing Air configuration file and potential setup issues The verification revealed several concerns:
Required actions:
🔗 Analysis chainVerify Air configuration and method usage. Let's verify the integration points: 🏁 Scripts executedThe following scripts were executed for the analysis: Script: #!/bin/bash
# Verify Air configuration and BuildDev usage
# Check if Air config exists
echo "Checking Air configuration files..."
fd -t f ".air.toml$"
# Look for BuildDev method usage
echo -e "\nChecking BuildDev method usage..."
rg "BuildDev.*ground-control" -t go
Length of output: 224 Script: #!/bin/bash
# Let's broaden our search to get more context
# Check for any Air-related configuration files with different extensions
echo "Checking for Air configuration files with various extensions..."
fd -t f "air\.(toml|conf|config|yaml|yml)$"
# Look for BuildDev method usage with a broader search
echo -e "\nChecking BuildDev method usage (broader search)..."
rg "BuildDev" -A 3 -B 1 -t go
# Check the ground-control directory structure
echo -e "\nChecking ground-control directory structure..."
fd . ground-control
# Look for go.mod in ground-control
echo -e "\nChecking for go.mod in ground-control..."
fd "go.mod$" ground-control
Length of output: 2711 |
||
|
||
// Build function would start the build process for the name provided. Source should be the path to the main.go file. | ||
func (m *HarborSatellite) Build( | ||
ctx context.Context, | ||
|
Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
@@ -0,0 +1,13 @@ | ||||||||||||||||||||||||||||||||||||||||||
[build] | ||||||||||||||||||||||||||||||||||||||||||
bin = "main" | ||||||||||||||||||||||||||||||||||||||||||
dir = "." | ||||||||||||||||||||||||||||||||||||||||||
cmd = "go build -o main ." | ||||||||||||||||||||||||||||||||||||||||||
exclude_dir = ["assets", "tmp", "vendor", "testdata"] | ||||||||||||||||||||||||||||||||||||||||||
exclude_regex = ["_test.go"] | ||||||||||||||||||||||||||||||||||||||||||
exclude_unchanged = false | ||||||||||||||||||||||||||||||||||||||||||
follow_symlink = false | ||||||||||||||||||||||||||||||||||||||||||
run_after_build = "chmod +x main" | ||||||||||||||||||||||||||||||||||||||||||
include_ext = ["go"] | ||||||||||||||||||||||||||||||||||||||||||
Comment on lines
+1
to
+10
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🛠️ Refactor suggestion Several improvements needed in build configuration
Apply these improvements: [build]
- bin = "main"
+ bin = "ground-control-dev"
dir = "."
- cmd = "go build -o main ."
+ cmd = "go build -o ground-control-dev ."
- exclude_dir = ["assets", "tmp", "vendor", "testdata"]
+ exclude_dir = ["assets", "tmp", "vendor", "testdata", ".git", "node_modules"]
exclude_regex = ["_test.go"]
- exclude_unchanged = false
+ exclude_unchanged = true
follow_symlink = false
- run_after_build = "chmod +x main"
+ run_after_build = "chmod 755 ground-control-dev"
include_ext = ["go"] 📝 Committable suggestion
Suggested change
|
||||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||||
[log] | ||||||||||||||||||||||||||||||||||||||||||
level = "debug" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Security: Replace sensitive credentials with placeholders.
The example contains actual credentials and URLs which could be security risks if copied directly. Consider:
Apply this diff:
📝 Committable suggestion