Release PR

README.md

@@ -19,6 +19,7 @@ Following features and tools are supported:
 * 🐟 Fish Shell
 * 📷 AzCopy
 * 🪪 Certificates
+* 📨 cmctl
 * ⚙️ Direnv
 * ⛵️ Helm
 * 🛠 JQ
@@ -30,6 +31,7 @@ Following features and tools are supported:
 * 📦 Packages
 * 📦 Packer
 * 👟 Run
+* 🔑 sops
 * 📜 Stern
 * 🌏 Terraform
 * 🐗 Terragrunt
@@ -53,6 +55,7 @@ Following features and tools are supported:
     * [Fish Shell](#_fish)
     * [AzCopy](#azcopy)
     * [Certificates](#certificates)
+    * [cmctl](#cmctl)
     * [Direnv](#direnv)
     * [Helm](#helm)
     * [JQ](#jq)
@@ -64,6 +67,7 @@ Following features and tools are supported:
     * [Packages](#packages)
     * [Packer](#packer)
     * [Run](#run)
+    * [sops](#sops)
     * [Stern](#stern)
     * [Terraform](#terraform)
     * [Terragrunt](#terragrunt)
@@ -393,6 +397,17 @@ Adds specified trusted certificate authorities into the container
   (optional). Defaults to `/certificates`. If something different than the default is used, the volume-target needs to be adapted to 
   the same directory
 
+### <a id="cmctl"></a> cmctl
+
+Installs the cert-manager Command Line Tool
+
+#### Configuration
+
+* USE_cmctl: Enable this feature
+* DEBUG_cmctl: Debug this feature
+* Environment CMCTL_VERSION: Version of cmctl to install (optional)
+  Defaults to `latest`
+
 ### <a id="direnv"></a> Direnv
 
 Installs [Direnv](https://direnv.net/)
@@ -540,6 +555,17 @@ Runs commands inside the shell when entering the cloud control container
 * DEBUG_run: Debug this feature
 * Environment RUN_COMMANDS: Valid shell commands to run
 
+### <a id="sops"></a> sops
+
+Installs [sops](https://github.com/getsops/sops)
+
+#### Configuration
+
+* USE_sops: Enable this feature
+* DEBUG_sops: Debug this feature
+* Environment SOPS_VERSION (required): Valid sops version (e.g. 3.8.1)
+* Environment specific for the key you use, see [sops documentation](https://github.com/getsops/sops?tab=readme-ov-file#22encrypting-using-age)
+
 ### <a id="stern"></a> Stern
 
 Installs [stern](https://github.com/stern/stern), a multi pod and container log tailing for Kubernetes
@@ -630,14 +656,24 @@ Installs the [YAML parser and processor yq](https://github.com/mikefarah/yq)
 
 ## Development
 
-*CloudControl* supports a decoupled development of features and flavours. If you're missing something, just fork this
-repository, create a subfolder for your new feature under "features" and add these files:
+*CloudControl* supports a decoupled development of features and flavours.
+
+### Features
+
+If you're missing a feature, just fork this repository, copy the feature template from features/.template into a
+new subfolder, check out the comments in the example files, and modify them to your needs.
+
+These files make up a feature:
 
 * `feature.yaml`: A descriptor for your feature with a title, a description and configuration notes
 * `install.sh`: A shell script that is run by CloudControlCenter and should install everything you need
   for your new feature
 * `motd.sh`: (optional) If you want to show some information to the users upon login, put them here.
 
+And optional, but recommended [integration tests](integration-testing) in a `.goss` folder.
+
+### Flavours
+
 If you need another flavour (aka cloud provider), add a new subdirectory under "flavour" and add a flavour.yaml describing
 your flavour the same way as a feature. For the rest of the files, please check out existing flavours for details. Please,
 include a sample configuration for your flavour to make it easier for other people to work with it.
@@ -775,7 +811,7 @@ you can inspect the failing container as well.
 
 To rebuild this documentation, first compile the documentation maker:
 
-    docker run --rm -e GOOS=[os, e.g. darwin, linux, windows] -e GOARCH=[architecture, e.g. arm64, amd64] -v "$PWD":/usr/src/myapp -w /usr/src/myapp golang:1.19-alpine go run cmd/doc/mkdoc
+    docker run --rm -e GOOS=[os, e.g. darwin, linux, windows] -e GOARCH=[architecture, e.g. arm64, amd64] -v "$PWD":/usr/src/myapp -w /usr/src/myapp golang:1.19-alpine go build cmd/doc/mkdoc.go
 
 Then run it to rebuild README.md based on README.md.gotmpl:
 

README.md.gotmpl

@@ -283,14 +283,24 @@ environment variable in the docker-compose file. Then you can debug with the run
 
 ## Development
 
-*CloudControl* supports a decoupled development of features and flavours. If you're missing something, just fork this
-repository, create a subfolder for your new feature under "features" and add these files:
+*CloudControl* supports a decoupled development of features and flavours.
+
+### Features
+
+If you're missing a feature, just fork this repository, copy the feature template from features/.template into a
+new subfolder, check out the comments in the example files, and modify them to your needs.
+
+These files make up a feature:
 
 * `feature.yaml`: A descriptor for your feature with a title, a description and configuration notes
 * `install.sh`: A shell script that is run by CloudControlCenter and should install everything you need
   for your new feature
 * `motd.sh`: (optional) If you want to show some information to the users upon login, put them here.
 
+And optional, but recommended [integration tests](integration-testing) in a `.goss` folder.
+
+### Flavours
+
 If you need another flavour (aka cloud provider), add a new subdirectory under "flavour" and add a flavour.yaml describing
 your flavour the same way as a feature. For the rest of the files, please check out existing flavours for details. Please,
 include a sample configuration for your flavour to make it easier for other people to work with it.
@@ -428,7 +438,7 @@ you can inspect the failing container as well.
 
 To rebuild this documentation, first compile the documentation maker:
 
-    docker run --rm -e GOOS=[os, e.g. darwin, linux, windows] -e GOARCH=[architecture, e.g. arm64, amd64] -v "$PWD":/usr/src/myapp -w /usr/src/myapp golang:1.19-alpine go run cmd/doc/mkdoc
+    docker run --rm -e GOOS=[os, e.g. darwin, linux, windows] -e GOARCH=[architecture, e.g. arm64, amd64] -v "$PWD":/usr/src/myapp -w /usr/src/myapp golang:1.19-alpine go build cmd/doc/mkdoc.go
 
 Then run it to rebuild README.md based on README.md.gotmpl:
 

assets/feature-installer-utils.sh

@@ -1,4 +1,29 @@
-# Execute a command and if it fails, return its output and exit
+# Set some informative variables
+
+# The flavour we're running on
+FLAVOUR="$(cat /home/cloudcontrol/flavour)"
+export FLAVOUR
+
+# The path to install software binaries to
+export BINPATH="/home/cloudcontrol/bin"
+
+export TEMPDIR=""
+
+# Prepare feature installation. Will create a temporary directory and change to it
+function prepare {
+  TEMPDIR=$(mktemp -d)
+  cd "${TEMPDIR}" || exit
+}
+
+# Cleanup the previously generated temporary directory
+function cleanup {
+  cd - &>/dev/null || exit
+  rm -rf "${TEMPDIR}"
+}
+
+# Usage: execHandle MESSAGE COMMAND...
+#
+# Output MESSAGE and then execute COMMAND. If it fails, return its output and exit
 function execHandle {
   TITLE=$1
   shift
@@ -38,6 +63,7 @@ function waitForMfaCode {
   echo "[VALID_CODE] Valid code entered. Thank you."
 }
 
+# Get the hardware platform we're on and translate it to the usual platform names used in most software
 function getPlatform {
   if [ "$(uname -m)" == 'aarch64' ]
   then
@@ -50,18 +76,32 @@ function getPlatform {
   fi
 }
 
+# Usage: checkAndCleanVersion VERSION
+#
+# Includes checks for version numbers and removes the "v" prefix from it to have a homogeneous version scheme
+# throughout CloudControl.
 function checkAndCleanVersion {
   VERSION=$1
   if [ "${VERSION:0:1}" == "v" ]
   then
-    echo "[DEPRECATION WARNING] Versions with a \"v\" prefix are deprecated and will be removed in CloudControl 4.0. Please only use versions without the \"v\" prefix. (Got \"${VERSION}\")" >&2
+    echo "[DEPRECATION WARNING] Versions with a \"v\" prefix are deprecated and will be removed in CloudControl 6.0.0 Please only use versions without the \"v\" prefix. (Got \"${VERSION}\")" >&2
     echo "${VERSION/#v/}"
   else
     echo "${VERSION}"
   fi
 }
 
+# Usage: download URL FILENAME
+#
+# Downloads the given URL into the provided FILENAME
+function download {
+  URL=$1
+  FILENAME=$2
+  execHandle "Downloading ${FILENAME} from ${URL}" curl -f -s -L "${URL}" -o "${FILENAME}"
+}
+
 # Usage: downloadFromGithub USER REPO VERSION PACKAGE_PREFIX PACKAGE_SUFFIX TARGET
+#
 # Downloads a release package from github using the common architecture names
 # The package will be downloaded from github.com/USER/REPO/releases/VERSION/download/PACKAGE to the given TARGET file
 # where PACKAGE consists of PACKAGE_PREFIXARCHITECTURE.PACKAGE_SUFFIX.
@@ -77,4 +117,4 @@ function downloadFromGithub {
   ARCH="$(uname -m | sed -e 's/x86_64/amd64/' -e 's/\(arm\)\(64\)\?.*/\1\2/' -e 's/aarch64$/arm64/')"
   PACKAGE="${PACKAGE_PREFIX}${ARCH}.${PACKAGE_SUFFIX}"
   execHandle "Downloading ${USER}/${REPO}@${VERSION}" curl -f -s -L "https://github.com/${USER}/${REPO}/releases/${VERSION}/download/${PACKAGE}" --output "${TARGET}"
-}
+}

build.sh

@@ -1,5 +1,7 @@
 #!/usr/bin/env bash
 
+set -euo pipefail
+
 # CloudControl build script
 # Usage:
 #
@@ -27,8 +29,8 @@ for FLAVOUR in ${FLAVOURS}
 do
   cat build/Dockerfile.prefix > Dockerfile
   cat "flavour/${FLAVOUR}/Dockerfile.flavour" >> Dockerfile
-  cat build/Dockerfile.suffix.mo | docker run --rm -i -e FLAVOUR=${FLAVOUR} -e BUILD_DATE=$(date -Iseconds) metal3d/mo >> Dockerfile
-  docker build --pull . -t "ghcr.io/dodevops/cloudcontrol-${FLAVOUR}:${TAG}"
+  cat build/Dockerfile.suffix.mo | docker run --rm -i -e FLAVOUR=${FLAVOUR} -e BUILD_DATE=$(date -Iseconds) ghcr.io/tests-always-included/mo:3.0.5 >> Dockerfile
+  docker build --pull . --no-cache -t "ghcr.io/dodevops/cloudcontrol-${FLAVOUR}:${TAG}"
 done
 
 if [ -e Dockerfile.sav ] ; then

cmd/doc/mkdoc.go

@@ -23,6 +23,9 @@ func fetchDocData(basePath string, filenamePattern string) (map[string]internal.
 		return docDatas, err
 	} else {
 		for _, dir := range subDirs {
+			if filepath.Base(dir) == ".template" {
+				continue
+			}
 			if _, err := os.Stat(filepath.Join(dir, filenamePattern)); err == nil {
 				if yamlFile, err := os.ReadFile(filepath.Join(dir, filenamePattern)); err != nil {
 					return docDatas, err

cmd/tests/test-features.go

@@ -77,7 +77,7 @@ func getFeatures(flavour string) []lib.Feature {
 			funk.FilterString(
 				featuresGlob,
 				func(value string) bool {
-					if correctPathRegexp.Match([]byte(filepath.Dir(value))) {
+					if correctPathRegexp.Match([]byte(filepath.Base(value))) {
 						if yamlFile, err := os.ReadFile(filepath.Join(value, "feature.yaml")); err != nil {
 							return false
 						} else {
@@ -225,8 +225,10 @@ func main() {
 		os.Exit(1)
 	}
 
+	features := getFeatures(*flavour)
+
 	for _, includeFeature := range *includeFeatures {
-		if !funk.Contains(getFeatures(*flavour), func(feature lib.Feature) bool {
+		if !funk.Contains(features, func(feature lib.Feature) bool {
 			return feature.Name == includeFeature
 		}) {
 			logrus.Errorf("%s is not a known feature", includeFeature)
@@ -236,7 +238,7 @@ func main() {
 	}
 
 	for _, excludeFeature := range *excludeFeatures {
-		if !funk.Contains(getFeatures(*flavour), func(feature lib.Feature) bool {
+		if !funk.Contains(features, func(feature lib.Feature) bool {
 			return feature.Name == excludeFeature
 		}) {
 			logrus.Errorf("%s is not a known feature", excludeFeature)

feature/.template/feature.yaml

@@ -0,0 +1,18 @@
+# This is the metadata manifest for a feature. Configure it for the feature to be included in CloudControl
+# In your new feature, please remove all descriptive comments from the template.
+
+# Please use an emoji character as a meaningful logo for your feature
+icon: "🫥"
+
+# Set the title for this feature
+title: "Template"
+
+# Set the description of the feature in Markdown. Be sure to include links to websites of the software the feature
+# installs or configures
+description: "Does some awesome stuff"
+
+# Use the configuration key to list additional configuration options that you're providing in your feature
+configuration:
+    #- |
+    #    Environment EXAMPLE_ENVIRONMENT: This environment variable is used in the feature to do xyz (optional)
+    #    Defaults to `abc`

feature/.template/goss/.envrc

@@ -0,0 +1,2 @@
+# Setup required environment for test
+export EXAMPLE_ENVIRONMENT=something

feature/.template/goss/goss.yaml

@@ -0,0 +1,5 @@
+command:
+    test:
+        # Run ls / after installing the feature and if that returns successfully, the test is successful
+        exec: "ls /"
+        exit-status: 0

feature/.template/install.sh

@@ -0,0 +1,27 @@
+# The installer script for your feature, which is called in the init phase of CloudControl
+
+# Include the feature installer utils which include helpers for the installation of your feature
+# See /assets/feature-installer-utils.sh
+
+. /feature-installer-utils.sh
+
+# Prepare a workspace for installing the feature
+prepare
+
+if [ "X${FLAVOUR}X" == "XsimpleX" ]
+then
+  # Do specific things for the simple flavour
+  download "https://example.com/my-tool-for-simple.tar.gz" tool.tar.gz
+elif [[ "X${FLAVOR}X" =~ X(azure|gcloud)X ]]
+then
+  # Do specific things for other flavours
+  download "https://example.com/my-tool-for-others.tar.gz" tool.tar.gz
+fi
+
+# Do other things
+execHandle "Extracting tool" tar xzf tool.tar.gz
+execHandle "Making tool executable" chmod +x tool
+execHandle "Copying tool to ${BINPATH}" cp tool "${BINPATH}"
+
+# Cleanup what we did
+cleanup

feature/cmctl/feature.yaml

@@ -0,0 +1,7 @@
+icon: "📨"
+title: "cmctl"
+description: "Installs the cert-manager Command Line Tool"
+configuration:
+    - |
+        Environment CMCTL_VERSION: Version of cmctl to install (optional)
+        Defaults to `latest`

feature/cmctl/goss/goss.yaml

@@ -0,0 +1,6 @@
+command:
+    cmctl:
+        exec: "/home/cloudcontrol/bin/cmctl version --client"
+        stdout:
+            - "Client Version"
+        exit-status: 0

feature/cmctl/install.sh

@@ -0,0 +1,9 @@
+. /feature-installer-utils.sh
+
+# Prepare a workspace for installing the feature
+prepare
+
+download "https://github.com/cert-manager/cmctl/releases/${CMCTL_VERSION:-latest}/download/cmctl_linux_$(getPlatform)" cmctl
+execHandle "Making cmctl executable" chmod +x cmctl
+execHandle "Copying tool to ${BINPATH}" cp cmctl "${BINPATH}"
+cleanup

feature/sops/feature.yaml

@@ -0,0 +1,6 @@
+icon: "🔑"
+title: "sops"
+description: "Installs [sops](https://github.com/getsops/sops)"
+configuration:
+  - "Environment SOPS_VERSION (required): Valid sops version (e.g. 3.8.1)"
+  - "Environment specific for the key you use, see [sops documentation](https://github.com/getsops/sops?tab=readme-ov-file#22encrypting-using-age)"

feature/sops/goss/.env

@@ -0,0 +1 @@
+SOPS_VERSION=3.8.1

feature/sops/goss/goss.yaml

@@ -0,0 +1,6 @@
+command:
+  sops:
+    exec: "/home/cloudcontrol/bin/sops --version"
+    exit-status: 0
+    stdout:
+      - "sops"