Skip to content

Commit

Permalink
Initial installer version (#110)
Browse files Browse the repository at this point in the history
Initial installer version
  • Loading branch information
pablomartinezbernardo authored Sep 25, 2024
1 parent e3b7fbe commit ea95f3f
Show file tree
Hide file tree
Showing 13 changed files with 973 additions and 4 deletions.
38 changes: 37 additions & 1 deletion .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,34 @@ jobs:
MAKE_JOB_COUNT: 8
WAF: "<< parameters.waf >>"
NGINX_VERSION: "<< parameters.nginx-version >>"
build_installer_arm64:
steps:
- checkout
- run: go -C ./installer/configurator build -o nginx-configurator
- persist_to_workspace:
root: "./installer/configurator"
paths:
- "nginx-configurator"
- store_artifacts:
path: "./installer/configurator/nginx-configurator"
destination: nginx-configurator
machine:
image: ubuntu-2204:current
resource_class: arm.medium
build_installer_amd64:
steps:
- checkout
- run: go -C ./installer/configurator build -o nginx-configurator
- persist_to_workspace:
root: "./installer/configurator"
paths:
- "nginx-configurator"
- store_artifacts:
path: "./installer/configurator/nginx-configurator"
destination: nginx-configurator
machine:
image: ubuntu-2204:current
resource_class: medium
coverage:
environment:
DOCKER_BUILDKIT: 1
Expand Down Expand Up @@ -206,7 +234,7 @@ jobs:
entrypoint: "/bin/sh"
steps:
- checkout
- run: find bin/ test/ example/ -type f -executable | xargs shellcheck --exclude
- run: find bin/ test/ example/ installer/ -type f -executable | xargs shellcheck --exclude
SC1071,SC1091,SC2317
workflows:
build-and-test:
Expand Down Expand Up @@ -241,6 +269,10 @@ workflows:
- 'ON'
- 'OFF'
name: build << matrix.nginx-version >> on arm64 WAF << matrix.waf >>
- build_installer_amd64:
name: build installer on amd64
- build_installer_arm64:
name: build installer on arm64
- coverage:
name: Coverage on 1.27.0 with WAF ON
- test:
Expand Down Expand Up @@ -359,6 +391,10 @@ workflows:
- 'ON'
- 'OFF'
name: build << matrix.nginx-version >> on arm64 WAF << matrix.waf >>
- build_installer_amd64:
name: build installer on amd64
- build_installer_arm64:
name: build installer on arm64
- test:
matrix:
parameters:
Expand Down
41 changes: 38 additions & 3 deletions bin/release.py
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,29 @@ def sign_package(package_path: str) -> None:
run(command, check=True)


def prepare_installer_release_artifact(work_dir, build_job_number, arch):
artifacts = send_ci_request_paged(
f"/project/{PROJECT_SLUG}/{build_job_number}/artifacts")
module_url = None
for artifact in artifacts:
name = artifact["path"]
if name == "nginx-configurator":
module_url = artifact["url"]

if module_url is None:
raise Exception(
f"Job number {build_job_number} doesn't have an 'nginx-configurator' build artifact."
)

module_path = work_dir / "nginx-configurator"
download_file(module_url, module_path)

# Package and sign
tarball_path = (work_dir / f"nginx-configurator-{arch}.tgz")
package(module_path, out=tarball_path)
sign_package(tarball_path)


def prepare_release_artifact(work_dir, build_job_number, version, arch, waf):
waf_suffix = "-appsec" if waf else ""
artifacts = send_ci_request_paged(
Expand Down Expand Up @@ -185,10 +208,19 @@ def prepare_release_artifact(work_dir, build_job_number, version, arch, waf):
sign_package(debug_tarball_path)


def handle_job(job, work_dir):
def handle_job(job, work_dir, installer):

# See the response schema for a list of statuses:
# https://circleci.com/docs/api/v2/index.html#operation/listWorkflowJobs
if job["name"].startswith("build "):
if installer and job["name"].startswith("build installer "):
# name should be something like "build installer on arm64"
match = re.match(r"build installer on (amd64|arm64)", job["name"])
if match is None:
raise Exception(f'Job name does not match regex "{re}": {job}')
arch = match.groups()[0]
prepare_installer_release_artifact(work_dir, job["job_number"], arch)
if not installer and job["name"].startswith(
"build ") and not job["name"].startswith("build installer "):
# name should be something like "build 1.25.4 on arm64 WAF ON"
match = re.match(r"build ([\d.]+) on (amd64|arm64) WAF (ON|OFF)",
job["name"])
Expand All @@ -213,6 +245,9 @@ def handle_job(job, work_dir):
help=
"ID of the release workflow. Find in job url. Example: https://app.circleci.com/pipelines/github/DataDog/nginx-datadog/542/workflows/<WORKFLOW_ID>",
)
parser.add_argument("--installer",
help="Release the NGINX installer",
action=argparse.BooleanOptionalAction)
options = parser.parse_args()

ci_api_token = options.ci_token
Expand Down Expand Up @@ -247,7 +282,7 @@ def handle_job(job, work_dir):
sys.exit(1)

for job in jobs:
result = handle_job(job, Path(work_dir))
result = handle_job(job, Path(work_dir), options.installer)
if result != "done":
sys.exit(1)

Expand Down
40 changes: 40 additions & 0 deletions installer/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
NGINX INSTALLER
===============
This folder contains the necessary components to install the NGINX module.
It consists of an sh script, plus a go binary (configurator) that performs the actual installation.
Running this installer results in the NGINX module being installed locally.

Shell script
------------
Verifies connectivity to the Datadog Agent. Detects the architecture, downloads
the Configurator for the specific architecture and invokes it forwarding all the
parameters plus the architecture.

Configurator
------------
Does the bulk of the installation.
1. Validates parameters
2. Validates NGINX installation and version
3. Downloads the NGINX module
4. Makes neccessary config modifications
5. Validates the modified configurations.

Local Testing
-------------
1. Compile the go binary
```bash
env GOOS=linux go -C configurator build -o ../nginx-configurator
```
2. Start docker compose with the docker-compose file provided. It boots up an NGINX
instance and a Datadog Agent instance with connectivity to each other.
```bash
DD_API_KEY=<YOUR_API_KEY> docker compose -f test/docker-compose.yml up -d
```
3. Run the installer
```bash
docker compose -f test/docker-compose.yml exec nginx bash -c "cd /installer && sh install-nginx-datadog.sh --appId 123 --site datadoghq.com --clientToken abcdef --sessionSampleRate 50 --sessionReplaySampleRate 50 --agentUri http://datadog-agent:8126"
```

Currently, the latest release doesn't yet support rum injection, so expect a message
showing an error when validating the final NGINX configuration `unknown directive
"datadog_rum" in /etc/nginx/nginx.conf`
98 changes: 98 additions & 0 deletions installer/configurator/configurator.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
package main

import (
"flag"
"fmt"
"os"

log "github.com/sirupsen/logrus"
)

var InstallerVersion = "0.1.0"

func validateInput(appID, site, clientToken, arch string, sessionSampleRate, sessionReplaySampleRate int) error {

log.Debug("Validating input arguments")

if appID == "" {
return NewInstallerError(ArgumentError, fmt.Errorf("--appId is required"))
}
if site == "" {
return NewInstallerError(ArgumentError, fmt.Errorf("--site is required"))
}
if clientToken == "" {
return NewInstallerError(ArgumentError, fmt.Errorf("--clientToken is required"))
}
if sessionSampleRate < 0 || sessionSampleRate > 100 {
return NewInstallerError(ArgumentError, fmt.Errorf("sessionSampleRate is required and must be between 0 and 100"))
}
if sessionReplaySampleRate < 0 || sessionReplaySampleRate > 100 {
return NewInstallerError(ArgumentError, fmt.Errorf("sessionReplaySampleRate is required and must be between 0 and 100"))
}
if arch != "amd64" && arch != "arm64" {
return NewInstallerError(ArgumentError, fmt.Errorf("arch must be either 'amd64' or 'arm64'"))
}
return nil
}

func handleError(err error) {
// TODO: Send telemetry

log.Error(err)

os.Exit(1)
}

func main() {
appID := flag.String("appId", "", "Application ID")
site := flag.String("site", "", "Site")
clientToken := flag.String("clientToken", "", "Client Token")
sessionSampleRate := flag.Int("sessionSampleRate", -1, "Session Sample Rate (0-100)")
sessionReplaySampleRate := flag.Int("sessionReplaySampleRate", -1, "Session Replay Sample Rate (0-100)")
arch := flag.String("arch", "", "Architecture (amd64 or arm64)")
agentUri := flag.String("agentUri", "http://localhost:8126", "Datadog Agent URI")
skipVerify := flag.Bool("skipVerify", false, "Skip verifying downloads")
verbose := flag.Bool("verbose", false, "Verbose output")
dryRun := flag.Bool("dryRun", false, "Dry run (no changes made)")

flag.Parse()

if *verbose {
log.SetLevel(log.DebugLevel)
log.Debug("Verbose output enabled")
} else {
log.SetLevel(log.InfoLevel)
}

log.Info("Starting installer version ", InstallerVersion)

if *dryRun {
log.Info("Dry run enabled. No changes will be made.")
}

if err := validateInput(*appID, *site, *clientToken, *arch, *sessionSampleRate, *sessionReplaySampleRate); err != nil {
handleError(err)
}

var configurator ProxyConfigurator = &NginxConfigurator{}

if err := configurator.VerifyRequirements(); err != nil {
handleError(err)
}

if err := configurator.DownloadAndInstallModule(*arch, *skipVerify); err != nil {
handleError(err)
}

if err := configurator.ModifyConfig(*appID, *site, *clientToken, *agentUri, *sessionSampleRate, *sessionReplaySampleRate, *dryRun); err != nil {
handleError(err)
}

if !*dryRun {
if err := configurator.ValidateConfig(); err != nil {
handleError(err)
}

log.Info("Datadog NGINX module has been successfully installed and configured. Please restart NGINX for the changes to take effect")
}
}
46 changes: 46 additions & 0 deletions installer/configurator/error.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
package main

import "fmt"

type ErrorType int

const (
UnexpectedError ErrorType = iota
ArgumentError
InternalError
NginxError
TelemetryError
)

func (e ErrorType) String() string {
switch e {
case UnexpectedError:
return "UnexpectedError"
case ArgumentError:
return "ArgumentError"
case InternalError:
return "InternalError"
case NginxError:
return "NginxError"
case TelemetryError:
return "TelemetryError"
default:
return fmt.Sprintf("%d", int(e))
}
}

func NewInstallerError(errorType ErrorType, err error) *InstallerError {
return &InstallerError{
ErrorType: errorType,
Err: err,
}
}

type InstallerError struct {
ErrorType ErrorType
Err error
}

func (m *InstallerError) Error() string {
return fmt.Sprintf("%s: %v", m.ErrorType, m.Err)
}
15 changes: 15 additions & 0 deletions installer/configurator/go.mod
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
module github.com/DataDog/nginx-datadog/installer

go 1.21.0

require (
github.com/google/go-github v17.0.0+incompatible
github.com/google/uuid v1.6.0
github.com/sirupsen/logrus v1.9.3
)

require (
github.com/google/go-querystring v1.1.0 // indirect
github.com/stretchr/testify v1.9.0 // indirect
golang.org/x/sys v0.22.0 // indirect
)
27 changes: 27 additions & 0 deletions installer/configurator/go.sum
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/google/go-cmp v0.5.2 h1:X2ev0eStA3AbceY54o37/0PQ/UWqKEiiO2dKL5OPaFM=
github.com/google/go-cmp v0.5.2/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-github v17.0.0+incompatible h1:N0LgJ1j65A7kfXrZnUDaYCs/Sf4rEjNlfyDHW9dolSY=
github.com/google/go-github v17.0.0+incompatible/go.mod h1:zLgOLi98H3fifZn+44m+umXrS52loVEgC2AApnigrVQ=
github.com/google/go-querystring v1.1.0 h1:AnCroh3fv4ZBgVIf1Iwtovgjaw/GiKJo8M8yD/fhyJ8=
github.com/google/go-querystring v1.1.0/go.mod h1:Kcdr2DB4koayq7X8pmAG4sNG59So17icRSOU623lUBU=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/sirupsen/logrus v1.9.3 h1:dueUQJ1C2q9oE3F7wvmSGAaVtTmUizReu6fjN8uqzbQ=
github.com/sirupsen/logrus v1.9.3/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.9.0 h1:HtqpIVDClZ4nwg75+f6Lvsy/wHu+3BoSGCbBAcpTsTg=
github.com/stretchr/testify v1.9.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.22.0 h1:RI27ohtqKCnwULzJLqkv897zojh5/DwS/ENaMzUOaWI=
golang.org/x/sys v0.22.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
Loading

0 comments on commit ea95f3f

Please sign in to comment.