diff --git a/README.md b/README.md index 260d1d7..b5d32c5 100644 --- a/README.md +++ b/README.md @@ -19,6 +19,7 @@ Following features and tools are supported: * 🐟 Fish Shell * 📷 AzCopy * 🪪 Certificates +* 📨 cmctl * ⚙️ Direnv * ⛵️ Helm * 🛠 JQ @@ -30,6 +31,7 @@ Following features and tools are supported: * 📦 Packages * 📦 Packer * 👟 Run +* 🔑 sops * 📜 Stern * 🌏 Terraform * 🐗 Terragrunt @@ -53,6 +55,7 @@ Following features and tools are supported: * [Fish Shell](#_fish) * [AzCopy](#azcopy) * [Certificates](#certificates) + * [cmctl](#cmctl) * [Direnv](#direnv) * [Helm](#helm) * [JQ](#jq) @@ -64,6 +67,7 @@ Following features and tools are supported: * [Packages](#packages) * [Packer](#packer) * [Run](#run) + * [sops](#sops) * [Stern](#stern) * [Terraform](#terraform) * [Terragrunt](#terragrunt) @@ -393,6 +397,17 @@ Adds specified trusted certificate authorities into the container (optional). Defaults to `/certificates`. If something different than the default is used, the volume-target needs to be adapted to the same directory +### cmctl + +Installs the cert-manager Command Line Tool + +#### Configuration + +* USE_cmctl: Enable this feature +* DEBUG_cmctl: Debug this feature +* Environment CMCTL_VERSION: Version of cmctl to install (optional) + Defaults to `latest` + ### Direnv Installs [Direnv](https://direnv.net/) @@ -540,6 +555,17 @@ Runs commands inside the shell when entering the cloud control container * DEBUG_run: Debug this feature * Environment RUN_COMMANDS: Valid shell commands to run +### sops + +Installs [sops](https://github.com/getsops/sops) + +#### Configuration + +* USE_sops: Enable this feature +* DEBUG_sops: Debug this feature +* Environment SOPS_VERSION (required): Valid sops version (e.g. 3.8.1) +* Environment specific for the key you use, see [sops documentation](https://github.com/getsops/sops?tab=readme-ov-file#22encrypting-using-age) + ### Stern Installs [stern](https://github.com/stern/stern), a multi pod and container log tailing for Kubernetes @@ -630,14 +656,24 @@ Installs the [YAML parser and processor yq](https://github.com/mikefarah/yq) ## Development -*CloudControl* supports a decoupled development of features and flavours. If you're missing something, just fork this -repository, create a subfolder for your new feature under "features" and add these files: +*CloudControl* supports a decoupled development of features and flavours. + +### Features + +If you're missing a feature, just fork this repository, copy the feature template from features/.template into a +new subfolder, check out the comments in the example files, and modify them to your needs. + +These files make up a feature: * `feature.yaml`: A descriptor for your feature with a title, a description and configuration notes * `install.sh`: A shell script that is run by CloudControlCenter and should install everything you need for your new feature * `motd.sh`: (optional) If you want to show some information to the users upon login, put them here. +And optional, but recommended [integration tests](integration-testing) in a `.goss` folder. + +### Flavours + If you need another flavour (aka cloud provider), add a new subdirectory under "flavour" and add a flavour.yaml describing your flavour the same way as a feature. For the rest of the files, please check out existing flavours for details. Please, include a sample configuration for your flavour to make it easier for other people to work with it. @@ -775,7 +811,7 @@ you can inspect the failing container as well. To rebuild this documentation, first compile the documentation maker: - docker run --rm -e GOOS=[os, e.g. darwin, linux, windows] -e GOARCH=[architecture, e.g. arm64, amd64] -v "$PWD":/usr/src/myapp -w /usr/src/myapp golang:1.19-alpine go run cmd/doc/mkdoc + docker run --rm -e GOOS=[os, e.g. darwin, linux, windows] -e GOARCH=[architecture, e.g. arm64, amd64] -v "$PWD":/usr/src/myapp -w /usr/src/myapp golang:1.19-alpine go build cmd/doc/mkdoc.go Then run it to rebuild README.md based on README.md.gotmpl: diff --git a/README.md.gotmpl b/README.md.gotmpl index 07fb2ee..7ba0d0d 100644 --- a/README.md.gotmpl +++ b/README.md.gotmpl @@ -283,14 +283,24 @@ environment variable in the docker-compose file. Then you can debug with the run ## Development -*CloudControl* supports a decoupled development of features and flavours. If you're missing something, just fork this -repository, create a subfolder for your new feature under "features" and add these files: +*CloudControl* supports a decoupled development of features and flavours. + +### Features + +If you're missing a feature, just fork this repository, copy the feature template from features/.template into a +new subfolder, check out the comments in the example files, and modify them to your needs. + +These files make up a feature: * `feature.yaml`: A descriptor for your feature with a title, a description and configuration notes * `install.sh`: A shell script that is run by CloudControlCenter and should install everything you need for your new feature * `motd.sh`: (optional) If you want to show some information to the users upon login, put them here. +And optional, but recommended [integration tests](integration-testing) in a `.goss` folder. + +### Flavours + If you need another flavour (aka cloud provider), add a new subdirectory under "flavour" and add a flavour.yaml describing your flavour the same way as a feature. For the rest of the files, please check out existing flavours for details. Please, include a sample configuration for your flavour to make it easier for other people to work with it. @@ -428,7 +438,7 @@ you can inspect the failing container as well. To rebuild this documentation, first compile the documentation maker: - docker run --rm -e GOOS=[os, e.g. darwin, linux, windows] -e GOARCH=[architecture, e.g. arm64, amd64] -v "$PWD":/usr/src/myapp -w /usr/src/myapp golang:1.19-alpine go run cmd/doc/mkdoc + docker run --rm -e GOOS=[os, e.g. darwin, linux, windows] -e GOARCH=[architecture, e.g. arm64, amd64] -v "$PWD":/usr/src/myapp -w /usr/src/myapp golang:1.19-alpine go build cmd/doc/mkdoc.go Then run it to rebuild README.md based on README.md.gotmpl: diff --git a/assets/feature-installer-utils.sh b/assets/feature-installer-utils.sh index 5eb02f6..d0f2763 100644 --- a/assets/feature-installer-utils.sh +++ b/assets/feature-installer-utils.sh @@ -1,4 +1,29 @@ -# Execute a command and if it fails, return its output and exit +# Set some informative variables + +# The flavour we're running on +FLAVOUR="$(cat /home/cloudcontrol/flavour)" +export FLAVOUR + +# The path to install software binaries to +export BINPATH="/home/cloudcontrol/bin" + +export TEMPDIR="" + +# Prepare feature installation. Will create a temporary directory and change to it +function prepare { + TEMPDIR=$(mktemp -d) + cd "${TEMPDIR}" || exit +} + +# Cleanup the previously generated temporary directory +function cleanup { + cd - &>/dev/null || exit + rm -rf "${TEMPDIR}" +} + +# Usage: execHandle MESSAGE COMMAND... +# +# Output MESSAGE and then execute COMMAND. If it fails, return its output and exit function execHandle { TITLE=$1 shift @@ -38,6 +63,7 @@ function waitForMfaCode { echo "[VALID_CODE] Valid code entered. Thank you." } +# Get the hardware platform we're on and translate it to the usual platform names used in most software function getPlatform { if [ "$(uname -m)" == 'aarch64' ] then @@ -50,18 +76,32 @@ function getPlatform { fi } +# Usage: checkAndCleanVersion VERSION +# +# Includes checks for version numbers and removes the "v" prefix from it to have a homogeneous version scheme +# throughout CloudControl. function checkAndCleanVersion { VERSION=$1 if [ "${VERSION:0:1}" == "v" ] then - echo "[DEPRECATION WARNING] Versions with a \"v\" prefix are deprecated and will be removed in CloudControl 4.0. Please only use versions without the \"v\" prefix. (Got \"${VERSION}\")" >&2 + echo "[DEPRECATION WARNING] Versions with a \"v\" prefix are deprecated and will be removed in CloudControl 6.0.0 Please only use versions without the \"v\" prefix. (Got \"${VERSION}\")" >&2 echo "${VERSION/#v/}" else echo "${VERSION}" fi } +# Usage: download URL FILENAME +# +# Downloads the given URL into the provided FILENAME +function download { + URL=$1 + FILENAME=$2 + execHandle "Downloading ${FILENAME} from ${URL}" curl -f -s -L "${URL}" -o "${FILENAME}" +} + # Usage: downloadFromGithub USER REPO VERSION PACKAGE_PREFIX PACKAGE_SUFFIX TARGET +# # Downloads a release package from github using the common architecture names # The package will be downloaded from github.com/USER/REPO/releases/VERSION/download/PACKAGE to the given TARGET file # where PACKAGE consists of PACKAGE_PREFIXARCHITECTURE.PACKAGE_SUFFIX. @@ -77,4 +117,4 @@ function downloadFromGithub { ARCH="$(uname -m | sed -e 's/x86_64/amd64/' -e 's/\(arm\)\(64\)\?.*/\1\2/' -e 's/aarch64$/arm64/')" PACKAGE="${PACKAGE_PREFIX}${ARCH}.${PACKAGE_SUFFIX}" execHandle "Downloading ${USER}/${REPO}@${VERSION}" curl -f -s -L "https://github.com/${USER}/${REPO}/releases/${VERSION}/download/${PACKAGE}" --output "${TARGET}" -} \ No newline at end of file +} diff --git a/build.sh b/build.sh index d171681..53b513d 100644 --- a/build.sh +++ b/build.sh @@ -1,5 +1,7 @@ #!/usr/bin/env bash +set -euo pipefail + # CloudControl build script # Usage: # @@ -27,8 +29,8 @@ for FLAVOUR in ${FLAVOURS} do cat build/Dockerfile.prefix > Dockerfile cat "flavour/${FLAVOUR}/Dockerfile.flavour" >> Dockerfile - cat build/Dockerfile.suffix.mo | docker run --rm -i -e FLAVOUR=${FLAVOUR} -e BUILD_DATE=$(date -Iseconds) metal3d/mo >> Dockerfile - docker build --pull . -t "ghcr.io/dodevops/cloudcontrol-${FLAVOUR}:${TAG}" + cat build/Dockerfile.suffix.mo | docker run --rm -i -e FLAVOUR=${FLAVOUR} -e BUILD_DATE=$(date -Iseconds) ghcr.io/tests-always-included/mo:3.0.5 >> Dockerfile + docker build --pull . --no-cache -t "ghcr.io/dodevops/cloudcontrol-${FLAVOUR}:${TAG}" done if [ -e Dockerfile.sav ] ; then diff --git a/cmd/doc/mkdoc.go b/cmd/doc/mkdoc.go index cdc8d98..86cf6c2 100644 --- a/cmd/doc/mkdoc.go +++ b/cmd/doc/mkdoc.go @@ -23,6 +23,9 @@ func fetchDocData(basePath string, filenamePattern string) (map[string]internal. return docDatas, err } else { for _, dir := range subDirs { + if filepath.Base(dir) == ".template" { + continue + } if _, err := os.Stat(filepath.Join(dir, filenamePattern)); err == nil { if yamlFile, err := os.ReadFile(filepath.Join(dir, filenamePattern)); err != nil { return docDatas, err diff --git a/cmd/tests/test-features.go b/cmd/tests/test-features.go index 7250e7a..1f9f93a 100644 --- a/cmd/tests/test-features.go +++ b/cmd/tests/test-features.go @@ -77,7 +77,7 @@ func getFeatures(flavour string) []lib.Feature { funk.FilterString( featuresGlob, func(value string) bool { - if correctPathRegexp.Match([]byte(filepath.Dir(value))) { + if correctPathRegexp.Match([]byte(filepath.Base(value))) { if yamlFile, err := os.ReadFile(filepath.Join(value, "feature.yaml")); err != nil { return false } else { @@ -225,8 +225,10 @@ func main() { os.Exit(1) } + features := getFeatures(*flavour) + for _, includeFeature := range *includeFeatures { - if !funk.Contains(getFeatures(*flavour), func(feature lib.Feature) bool { + if !funk.Contains(features, func(feature lib.Feature) bool { return feature.Name == includeFeature }) { logrus.Errorf("%s is not a known feature", includeFeature) @@ -236,7 +238,7 @@ func main() { } for _, excludeFeature := range *excludeFeatures { - if !funk.Contains(getFeatures(*flavour), func(feature lib.Feature) bool { + if !funk.Contains(features, func(feature lib.Feature) bool { return feature.Name == excludeFeature }) { logrus.Errorf("%s is not a known feature", excludeFeature) diff --git a/feature/.template/feature.yaml b/feature/.template/feature.yaml new file mode 100644 index 0000000..cfc1f24 --- /dev/null +++ b/feature/.template/feature.yaml @@ -0,0 +1,18 @@ +# This is the metadata manifest for a feature. Configure it for the feature to be included in CloudControl +# In your new feature, please remove all descriptive comments from the template. + +# Please use an emoji character as a meaningful logo for your feature +icon: "🫥" + +# Set the title for this feature +title: "Template" + +# Set the description of the feature in Markdown. Be sure to include links to websites of the software the feature +# installs or configures +description: "Does some awesome stuff" + +# Use the configuration key to list additional configuration options that you're providing in your feature +configuration: + #- | + # Environment EXAMPLE_ENVIRONMENT: This environment variable is used in the feature to do xyz (optional) + # Defaults to `abc` diff --git a/feature/.template/goss/.envrc b/feature/.template/goss/.envrc new file mode 100644 index 0000000..ed8b5fe --- /dev/null +++ b/feature/.template/goss/.envrc @@ -0,0 +1,2 @@ +# Setup required environment for test +export EXAMPLE_ENVIRONMENT=something diff --git a/feature/.template/goss/goss.yaml b/feature/.template/goss/goss.yaml new file mode 100644 index 0000000..b803253 --- /dev/null +++ b/feature/.template/goss/goss.yaml @@ -0,0 +1,5 @@ +command: + test: + # Run ls / after installing the feature and if that returns successfully, the test is successful + exec: "ls /" + exit-status: 0 diff --git a/feature/.template/install.sh b/feature/.template/install.sh new file mode 100644 index 0000000..eccc716 --- /dev/null +++ b/feature/.template/install.sh @@ -0,0 +1,27 @@ +# The installer script for your feature, which is called in the init phase of CloudControl + +# Include the feature installer utils which include helpers for the installation of your feature +# See /assets/feature-installer-utils.sh + +. /feature-installer-utils.sh + +# Prepare a workspace for installing the feature +prepare + +if [ "X${FLAVOUR}X" == "XsimpleX" ] +then + # Do specific things for the simple flavour + download "https://example.com/my-tool-for-simple.tar.gz" tool.tar.gz +elif [[ "X${FLAVOR}X" =~ X(azure|gcloud)X ]] +then + # Do specific things for other flavours + download "https://example.com/my-tool-for-others.tar.gz" tool.tar.gz +fi + +# Do other things +execHandle "Extracting tool" tar xzf tool.tar.gz +execHandle "Making tool executable" chmod +x tool +execHandle "Copying tool to ${BINPATH}" cp tool "${BINPATH}" + +# Cleanup what we did +cleanup \ No newline at end of file diff --git a/feature/cmctl/feature.yaml b/feature/cmctl/feature.yaml new file mode 100644 index 0000000..765eca6 --- /dev/null +++ b/feature/cmctl/feature.yaml @@ -0,0 +1,7 @@ +icon: "📨" +title: "cmctl" +description: "Installs the cert-manager Command Line Tool" +configuration: + - | + Environment CMCTL_VERSION: Version of cmctl to install (optional) + Defaults to `latest` diff --git a/feature/cmctl/goss/goss.yaml b/feature/cmctl/goss/goss.yaml new file mode 100644 index 0000000..8a02ad4 --- /dev/null +++ b/feature/cmctl/goss/goss.yaml @@ -0,0 +1,6 @@ +command: + cmctl: + exec: "/home/cloudcontrol/bin/cmctl version --client" + stdout: + - "Client Version" + exit-status: 0 diff --git a/feature/cmctl/install.sh b/feature/cmctl/install.sh new file mode 100644 index 0000000..6d6e489 --- /dev/null +++ b/feature/cmctl/install.sh @@ -0,0 +1,9 @@ +. /feature-installer-utils.sh + +# Prepare a workspace for installing the feature +prepare + +download "https://github.com/cert-manager/cmctl/releases/${CMCTL_VERSION:-latest}/download/cmctl_linux_$(getPlatform)" cmctl +execHandle "Making cmctl executable" chmod +x cmctl +execHandle "Copying tool to ${BINPATH}" cp cmctl "${BINPATH}" +cleanup \ No newline at end of file diff --git a/feature/sops/feature.yaml b/feature/sops/feature.yaml new file mode 100644 index 0000000..5fe5405 --- /dev/null +++ b/feature/sops/feature.yaml @@ -0,0 +1,6 @@ +icon: "🔑" +title: "sops" +description: "Installs [sops](https://github.com/getsops/sops)" +configuration: + - "Environment SOPS_VERSION (required): Valid sops version (e.g. 3.8.1)" + - "Environment specific for the key you use, see [sops documentation](https://github.com/getsops/sops?tab=readme-ov-file#22encrypting-using-age)" diff --git a/feature/sops/goss/.env b/feature/sops/goss/.env new file mode 100644 index 0000000..4a9e6df --- /dev/null +++ b/feature/sops/goss/.env @@ -0,0 +1 @@ +SOPS_VERSION=3.8.1 \ No newline at end of file diff --git a/feature/sops/goss/goss.yaml b/feature/sops/goss/goss.yaml new file mode 100644 index 0000000..6ecf167 --- /dev/null +++ b/feature/sops/goss/goss.yaml @@ -0,0 +1,6 @@ +command: + sops: + exec: "/home/cloudcontrol/bin/sops --version" + exit-status: 0 + stdout: + - "sops" diff --git a/feature/sops/install.sh b/feature/sops/install.sh new file mode 100644 index 0000000..471cc21 --- /dev/null +++ b/feature/sops/install.sh @@ -0,0 +1,23 @@ +. /feature-installer-utils.sh + +if [ -z "${SOPS_VERSION}" ] +then + echo "The sops feature requires a version set using SOPS_VERSION. See https://github.com/getsops/sops/releases/ for valid versions" + exit 1 +fi + +SOPS_VERSION=$(checkAndCleanVersion "${SOPS_VERSION}") + +TEMPDIR=$(mktemp -d) +cd "${TEMPDIR}" || exit + +execHandle "Downloading sops" curl -f -s -L "https://github.com/getsops/sops/releases/download/v${SOPS_VERSION}/sops-v${SOPS_VERSION}.linux.$(getPlatform)" --output sops +execHandle "Installing sops" mv sops /home/cloudcontrol/bin +execHandle "Making sops executable" chmod +x /home/cloudcontrol/bin/sops + +cd - &>/dev/null || exit +rm -rf "${TEMPDIR}" + + + +