Skip to content

Commit

Permalink
Merge pull request #297 from cisagov/v24.02.0_merge_cisagov
Browse files Browse the repository at this point in the history
Malcolm v24.02.0 contains new features, improvements, bug fixes and component version updates.

v24.01.0...v24.02.0

* Features and enhancements
    - [Hedgehog Linux SD card image for Raspberry Pi](https://cisagov.github.io/Malcolm/docs/hedgehog-raspi-build.html#HedgehogRaspiBuild) (idaholab#250; special thanks to @aut0exec for his work on this)
    - allow configuration of Arkime's ILM/ISM settings (idaholab#300)
    - add option for customizing which log types get NetBox enrichment (idaholab#316)
    - improve the extracted_files download page (idaholab#329)
    - include missing aggregations in API bucket queries (idaholab#386)
    - more intelligent .env file checking on startup (idaholab#387)
    - Malcolm report to itself on capture statistics (idaholab#395)
    - link to Dashboards/Arkime from NetBox devices view (idaholab#410)
    - changed default PCAP storage format to zstd(3) for new installations
    - various documentation updates and improvements
    - changed back to using official Zeek .deb files rather than building from source to reduce build times
* Component version updates
    - Arkime to [v5.0.0](https://github.com/arkime/arkime/blob/6914792d86ecba0009f9b49dabb1aa987e46ad26/CHANGELOG#L33-L130)
    - Capa to [v7.0.1](https://github.com/mandiant/capa/releases)
    - YARA to [v4.5.0](https://github.com/VirusTotal/yara/releases)
    - Beats to [v8.12.1](https://www.elastic.co/guide/en/beats/libbeat/current/release-notes-8.12.1.html)
    - Logstash to [v8.12.1](https://www.elastic.co/guide/en/logstash/current/logstash-8-12-1.html)
    - Zeek to [v6.1.1](https://github.com/zeek/zeek/releases/tag/v6.1.1)
* Bug fixes
    - pivot links from Arkime to Kibana in external elasticsearch are not working (idaholab#335)
    - redirect /dashboards/ link to Kibana in NGINX proxy in elasticsearch/kibana-based deployment (idaholab#403)
    - allow netbox-restore and netbox-backup to specify container name (idaholab#337)
    - fuzzy matching for manufacturers based on OUI to NetBox list is not very good (idaholab#393) (and [updated documentation](https://cisagov.github.io/Malcolm/docs/asset-interaction-analysis.html#NetBoxPopPassiveOUIMatch))
    - source.ip and destination.ip not set for parsed files.log entries for uploaded PCAP (idaholab#401)
    - event.severity_tags is not being assigned correctly based on rule.category (idaholab#402)
    - basic authentication breaks with special characters (idaholab#404)
    - changed some Logstash Ruby variables from global (`$`) to instance (`@`) (see ["avoiding concurrency issues"](https://www.elastic.co/guide/en/logstash/current/plugins-filters-ruby.html#plugins-filters-ruby-concurrency))
* Configuration changes (in [environment variables](https://cisagov.github.io/Malcolm/docs/malcolm-config.html#MalcolmConfigEnvVars) in [`./config/`](https://github.com/cisagov/Malcolm/blob/v24.02.0/config))
    * these variables in [`arkime.env`](https://github.com/cisagov/Malcolm/blob/main/config/arkime.env.example) to allow configuration of Arkime's ILM/ISM settings (idaholab#300)
    ```
    # These variables manage setting for Arkime's ILM/ISM features (https://arkime.com/faq#ilm)
    # Whether or not Arkime should perform index management
    INDEX_MANAGEMENT_ENABLED=false
    # Time in hours/days before moving to warm and force merge (number followed by h or d)
    INDEX_MANAGEMENT_OPTIMIZATION_PERIOD=30d
    # Time in hours/days before deleting index (number followed by h or d)
    INDEX_MANAGEMENT_RETENTION_TIME=90d
    # Number of replicas for older sessions indices
    INDEX_MANAGEMENT_OLDER_SESSION_REPLICAS=0
    # Number of weeks of history to retain
    INDEX_MANAGEMENT_HISTORY_RETENTION_WEEKS=13
    # Number of segments to optimize sessions for
    INDEX_MANAGEMENT_SEGMENTS=1
    # Whether or not Arkime should use a hot/warm design (storing non-session data in a warm index)
    INDEX_MANAGEMENT_HOT_WARM_ENABLED=false
    ```
    * these variables in [`dashboards.env`](https://github.com/cisagov/Malcolm/blob/main/config/dashboards.env.example) to override the values automatically configured for pivot links (idaholab#335) and `/dashboard/` redirect (idaholab#403) for Elasticsearch backend
    ```
    # These values are used to handle the Arkime value actions to pivot from Arkime
    #   to Dashboards. The nginx-proxy container's entrypoint will try to formulate
    #   them automatically, but they may be specified explicitly here.
    NGINX_DASHBOARDS_PREFIX=
    NGINX_DASHBOARDS_PROXY_PASS=
    ```
    * these variables in [`logstash.env`](https://github.com/cisagov/Malcolm/blob/main/config/logstash.env.example) for customizing which log types get NetBox enrichment (idaholab#316) and customizing which types of Zeek logs will be ignored (dropped) by LogStash
    ```
    # Which types of logs will be enriched via NetBox (comma-separated list of provider.dataset, or the string all to enrich all logs)
    LOGSTASH_NETBOX_ENRICHMENT_DATASETS=suricata.alert,zeek.conn,zeek.known_hosts,zeek.known_services,zeek.notice,zeek.signatures,zeek.software,zeek.weird
    ```
    ```
    # Zeek log types that will be ignored (dropped) by LogStash
    LOGSTASH_ZEEK_IGNORED_LOGS=analyzer,broker,bsap_ip_unknown,bsap_serial_unknown,capture_loss,cluster,config,ecat_arp_info,loaded_scripts,packet_filter,png,print,prof,reporter,stats,stderr,stdout
    ```
    * these variables in [`netbox-common.env`](https://github.com/cisagov/Malcolm/blob/main/config/netbox-common.env.example) for adjusting [matching device manufacturers to OUIs](https://cisagov.github.io/Malcolm/docs/asset-interaction-analysis.html#NetBoxPopPassiveOUIMatch) in NetBox autopopulation
    ```
    # Customize manufacturer matching/creation with LOGSTASH_NETBOX_AUTO_POPULATE (see logstash.env)
    NETBOX_DEFAULT_AUTOCREATE_MANUFACTURER=true
    NETBOX_DEFAULT_FUZZY_THRESHOLD=0.95
    ```
    * these variables in [suricata-live.env](https://github.com/cisagov/Malcolm/blob/main/config/suricata-live.env.example) and [zeek-live.env](https://github.com/cisagov/Malcolm/blob/main/config/zeek-live.env.example) that can be used to configure Malcolm reporting to itself on its Zeek and Suricata live capture statistics (idaholab#395)
    ```
    # Whether or not enable capture statistics and include them in eve.json
    SURICATA_STATS_ENABLED=false
    SURICATA_STATS_EVE_ENABLED=false
    SURICATA_STATS_INTERVAL=30
    SURICATA_STATS_DECODER_EVENTS=false
    ```
    ```
    # Set ZEEK_DISABLE_STATS to blank to generate stats.log and capture_loss.log
    ZEEK_DISABLE_STATS=true
    ```
    * this variable in [zeek.env](https://github.com/cisagov/Malcolm/blob/main/config/zeek.env.example) related to the improvements to the extracted_files download page (idaholab#329)
    ```
    # Whether or not to use libmagic to show MIME types for Zeek-extracted files served
    EXTRACTED_FILE_HTTP_SERVER_MAGIC=false
    ```
  • Loading branch information
mmguero authored Feb 15, 2024
2 parents 2862d43 + 80b7f16 commit 98ac2fb
Show file tree
Hide file tree
Showing 270 changed files with 5,925 additions and 2,247 deletions.
1 change: 1 addition & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ arkime-raw
kubernetes
malcolm-iso
sensor-iso
sensor-raspi
nginx/nginx_ldap*.conf
pcap
_site
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/api-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ jobs:
-
name: Extract Malcolm version
shell: bash
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
id: extract_malcolm_version
-
name: Set up QEMU
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/arkime-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ jobs:
-
name: Extract Malcolm version
shell: bash
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
id: extract_malcolm_version
-
name: Set up QEMU
Expand Down
42 changes: 0 additions & 42 deletions .github/workflows/codeql.yml

This file was deleted.

2 changes: 1 addition & 1 deletion .github/workflows/dashboards-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ jobs:
-
name: Extract Malcolm version
shell: bash
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
id: extract_malcolm_version
-
name: Set up QEMU
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ jobs:
-
name: Extract Malcolm version
shell: bash
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
id: extract_malcolm_version
-
name: Set up QEMU
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/dirinit-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ jobs:
-
name: Extract Malcolm version
shell: bash
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
id: extract_malcolm_version
-
name: Set up QEMU
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/file-monitor-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ jobs:
-
name: Extract Malcolm version
shell: bash
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
id: extract_malcolm_version
-
name: Set up QEMU
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/file-upload-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ jobs:
-
name: Extract Malcolm version
shell: bash
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
id: extract_malcolm_version
-
name: Set up QEMU
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/filebeat-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ jobs:
-
name: Extract Malcolm version
shell: bash
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
id: extract_malcolm_version
-
name: Set up QEMU
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/freq-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ jobs:
-
name: Extract Malcolm version
shell: bash
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
id: extract_malcolm_version
-
name: Set up QEMU
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/htadmin-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ jobs:
-
name: Extract Malcolm version
shell: bash
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
id: extract_malcolm_version
-
name: Set up QEMU
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/logstash-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ jobs:
-
name: Extract Malcolm version
shell: bash
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
id: extract_malcolm_version
-
name: Set up QEMU
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/malcolm-iso-build-docker-wrap-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -92,12 +92,12 @@ jobs:
-
name: Extract Malcolm version
shell: bash
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
id: extract_malcolm_version
-
name: Build image
run: |
IMAGES=( $(grep image: docker-compose.yml | awk '{print $2}' | sort -u) )
IMAGES=( $(grep image: docker-compose-dev.yml | awk '{print $2}' | sort -u) )
for IMAGE in "${IMAGES[@]}"; do
REPO_IMAGE="$(echo "$IMAGE" | sed "s@^.*\(malcolm\)@ghcr.io/${{ github.repository_owner }}/\1@" | sed "s/:.*/:${{ steps.extract_branch.outputs.branch }}/")"
docker pull "$REPO_IMAGE" && \
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/netbox-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ jobs:
-
name: Extract Malcolm version
shell: bash
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
id: extract_malcolm_version
-
name: Set up QEMU
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/nginx-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ jobs:
-
name: Extract Malcolm version
shell: bash
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
id: extract_malcolm_version
-
name: Set up QEMU
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/opensearch-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ jobs:
-
name: Extract Malcolm version
shell: bash
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
id: extract_malcolm_version
-
name: Set up QEMU
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/pcap-capture-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ jobs:
-
name: Extract Malcolm version
shell: bash
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
id: extract_malcolm_version
-
name: Set up QEMU
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/pcap-monitor-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ jobs:
-
name: Extract Malcolm version
shell: bash
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
id: extract_malcolm_version
-
name: Set up QEMU
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/postgresql-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ jobs:
-
name: Extract Malcolm version
shell: bash
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
id: extract_malcolm_version
-
name: Set up QEMU
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/redis-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ jobs:
-
name: Extract Malcolm version
shell: bash
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
id: extract_malcolm_version
-
name: Set up QEMU
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ jobs:
-
name: Extract Malcolm version
shell: bash
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
id: extract_malcolm_version
-
name: Build image
Expand Down
102 changes: 102 additions & 0 deletions .github/workflows/sensor-raspi-build-docker-wrap-push-ghcr.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
name: sensor-raspi-build-docker-wrap-push-ghcr

on:
# push:
# branches:
# - main
# - development
# paths:
# - '.trigger_raspi_workflow_build'
workflow_dispatch:
# repository_dispatch:

jobs:
build:
runs-on: ubuntu-22.04
permissions:
actions: write
packages: write
contents: read
security-events: write
defaults:
run:
shell: bash
steps:
-
name: Cancel previous run in progress
uses: styfle/[email protected]
with:
ignore_sha: true
all_but_latest: true
access_token: ${{ secrets.GITHUB_TOKEN }}
-
name: Set up QEMU
uses: docker/setup-qemu-action@v3
-
name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
with:
driver-opts: |
image=moby/buildkit:master
-
name: Build environment setup
run: |
sudo apt-get -q update
sudo env DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -y -q \
binfmt-support \
bmap-tools \
ca-certificates \
debootstrap \
dosfstools \
kpartx \
python3 \
qemu-user-static \
qemu-utils \
time \
vmdb2 \
zerofree
-
name: Checkout
uses: actions/checkout@v4
-
name: Extract branch name
shell: bash
run: echo "branch=$(echo ${GITHUB_REF#refs/heads/})" >> $GITHUB_OUTPUT
id: extract_branch
-
name: Extract commit SHA
shell: bash
run: echo "sha=$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT
id: extract_commit_sha
-
name: Extract Malcolm version
shell: bash
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
id: extract_malcolm_version
-
name: Build image
run: |
pushd ./sensor-raspi
mkdir -p ./shared
echo "${{ steps.extract_malcolm_version.outputs.mversion }}" > ./shared/version.txt
echo "${{ secrets.MAXMIND_GEOIP_DB_LICENSE_KEY }}" > ./shared/maxmind_license.txt
echo "GITHUB_TOKEN=${{ secrets.GITHUB_TOKEN }}" > ./shared/environment.chroot
echo "VCS_REVSION=${{ steps.extract_commit_sha.outputs.sha }}" > ./shared/environment.chroot
echo "BUILD_JOBS=2" > ./shared/environment.chroot
sudo make raspi_4_bookworm.img
sudo chmod 644 ./raspi_4_bookworm*.*
popd
-
name: ghcr.io login
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
password: ${{ secrets.GITHUB_TOKEN }}
-
name: Build and push IMG image
uses: docker/build-push-action@v5
with:
context: ./sensor-raspi
push: true
tags: ghcr.io/${{ github.repository_owner }}/malcolm/hedgehog-raspi:${{ steps.extract_branch.outputs.branch }}
2 changes: 1 addition & 1 deletion .github/workflows/suricata-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ jobs:
-
name: Extract Malcolm version
shell: bash
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
id: extract_malcolm_version
-
name: Set up QEMU
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/zeek-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ jobs:
-
name: Extract Malcolm version
shell: bash
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose-dev.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
id: extract_malcolm_version
-
name: Set up QEMU
Expand Down
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -28,10 +28,12 @@ config.*/
.envrc
.direnv
.vagrant
.fuse_*
malcolm_*images.tar.gz
malcolm_*images.tar.xz
malcolm_netbox_backup_*.gz
*.iso
*.img
*-build.log
Gemfile.lock
_site
Expand Down
2 changes: 1 addition & 1 deletion .trigger_iso_workflow_build
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
# this file exists solely for the purpose of being updated and seen by github to trigger a commit build action
2
3
Loading

0 comments on commit 98ac2fb

Please sign in to comment.