Skip to content

Commit

Permalink
Merge pull request #274 from cisagov/v23.08.0_merge_cisagov
Browse files Browse the repository at this point in the history
Malcolm v23.08.0 is a minor release with a few improvements, bug fixes and component updates.

v23.07.1...v23.08.0

* Features and enhancements
    + Rewrote the [Network Traffic Artifact Upload](https://cisagov.github.io/Malcolm/docs/upload.html#Upload) interface and backend, replacing the defunct [jQuery-File-Upload](https://github.com/blueimp/jQuery-File-Upload) with [FilePond](https://pqina.nl/filepond/). This was mainly due to jQuery-File-Upload no longer receiving security fixes and having some known vulnerabilities. see idaholab#235
    + Use [netbox-initializers](https://github.com/tobiasge/netbox-initializers) plugin, adding the ability to drop YAML files for various NetBox obects to be [preloaded](https://cisagov.github.io/Malcolm/docs/asset-interaction-analysis.html#NetBoxPreload) at startup. see idaholab#228
    + handle changes to ICSNPP parsers with source_ip/destination_ip fields (idaholab#233 and idaholab#226)

* Bug fixes
    + Fixed extracting Malcolm version during ISO build
    + Workaround for wireshark no longer publishing raw manuf (OUI) list (idaholab#230)
    + Remove news feed from default NetBox dashboard (as it would try to reach out to the web for RSS updates)

* Component version updates
    + Rebased Docker and ISO images to Debian 12 (bookworm)
    + live-build tool for building ISO images to debian/1%20230131
    + Arkime to [v4.4.0](https://github.com/arkime/arkime/blob/6f667600596e8a2252555640933f424730c258d5/CHANGELOG#L33-L55)
    + supercronic to [v0.2.26](https://github.com/aptible/supercronic/releases/tag/v0.2.26)
    + FileBeat to [v8.9.0](https://www.elastic.co/guide/en/beats/libbeat/current/release-notes-8.9.0.html)
    + LogStash to [v8.9.0](https://www.elastic.co/guide/en/logstash/8.9/logstash-8-9-0.html) (idaholab#234)
    + NetBox to [v3.5.7](https://github.com/netbox-community/netbox/releases/tag/v3.5.7)
    + PostgreSQL (used by NetBox) to [v15](https://www.postgresql.org/docs/release/15.0/)
    + opensearch-py to [v2.3.0](https://github.com/opensearch-project/opensearch-py/releases/tag/v2.3.0)
    + PHP (as used by Upload interface) to [v8.2](https://www.php.net/ChangeLog-8.php#PHP_8_2)
    + Fluent Bit to [v2.1.8](https://github.com/fluent/fluent-bit/releases/tag/v2.1.8)
    + certifi to [v2023.7.22](https://github.com/certifi/python-certifi/releases/tag/2023.07.22) (idaholab#229)
  • Loading branch information
mmguero authored Aug 15, 2023
2 parents 06b9562 + bfd945c commit d5ee03e
Show file tree
Hide file tree
Showing 157 changed files with 1,880 additions and 1,905 deletions.
6 changes: 3 additions & 3 deletions .github/workflows/malcolm-iso-build-docker-wrap-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -70,12 +70,12 @@ jobs:
virt-what \
xorriso \
xz-utils
git clone --depth=1 --single-branch --recurse-submodules --shallow-submodules --branch='debian/1%20210407' 'https://salsa.debian.org/live-team/live-build.git' /tmp/live-build
git clone --depth=1 --single-branch --recurse-submodules --shallow-submodules --branch='debian/1%20230131' 'https://salsa.debian.org/live-team/live-build.git' /tmp/live-build
cd /tmp/live-build
dpkg-buildpackage -b -uc -us
cd /tmp
sudo dpkg -i /tmp/live-build*.deb
rm -rf /tmp/live-build
sudo rm -rf /tmp/live-build /tmp/live-build*.deb
-
name: Checkout
uses: actions/checkout@v3
Expand All @@ -92,7 +92,7 @@ jobs:
-
name: Extract Malcolm version
shell: bash
run: echo "mversion=$(grep -P "^\s+image:\s*malcolm" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
id: extract_malcolm_version
-
name: Build image
Expand Down
12 changes: 7 additions & 5 deletions .github/workflows/sensor-iso-build-docker-wrap-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -66,10 +66,12 @@ jobs:
squashfs-tools \
virt-what \
xorriso
git clone --depth=1 --single-branch --recurse-submodules --shallow-submodules --branch='debian/1%20210407' 'https://salsa.debian.org/live-team/live-build.git' /tmp/live-build
cd /tmp/live-build
dpkg-buildpackage -b -uc -us
sudo dpkg -i /tmp/live-build*.deb
git clone --depth=1 --single-branch --recurse-submodules --shallow-submodules --branch='debian/1%20230131' 'https://salsa.debian.org/live-team/live-build.git' /tmp/live-build
cd /tmp/live-build
dpkg-buildpackage -b -uc -us
cd /tmp
sudo dpkg -i /tmp/live-build*.deb
sudo rm -rf /tmp/live-build /tmp/live-build*.deb
-
name: Checkout
uses: actions/checkout@v3
Expand All @@ -86,7 +88,7 @@ jobs:
-
name: Extract Malcolm version
shell: bash
run: echo "mversion=$(grep -P "^\s+image:\s*malcolm" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
run: echo "mversion=$(grep -P "^\s+image:.*/malcolm/" docker-compose.yml | awk '{print $2}' | cut -d':' -f2 | uniq -c | sort -nr | awk '{print $2}' | head -n 1)" >> $GITHUB_OUTPUT
id: extract_malcolm_version
-
name: Build image
Expand Down
10 changes: 5 additions & 5 deletions Dockerfiles/api.Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM python:3-slim-bullseye as builder
FROM python:3-slim-bookworm as builder

ENV DEBIAN_FRONTEND noninteractive
ENV TERM xterm
Expand All @@ -8,8 +8,8 @@ ENV PYTHONUNBUFFERED 1
RUN apt-get update -q \
&& apt-get -y -q upgrade \
&& apt-get install -y gcc \
&& python3 -m pip install --upgrade pip \
&& python3 -m pip install flake8
&& python3 -m pip install --break-system-packages --no-cache-dir --upgrade pip \
&& python3 -m pip install --break-system-packages --no-cache-dir flake8

COPY ./api /usr/src/app/
COPY scripts/malcolm_utils.py /usr/src/app/
Expand All @@ -18,7 +18,7 @@ WORKDIR /usr/src/app
RUN python3 -m pip wheel --no-cache-dir --no-deps --wheel-dir /usr/src/app/wheels -r requirements.txt \
&& flake8 --ignore=E203,E501,F401,W503

FROM python:3-slim-bullseye
FROM python:3-slim-bookworm

# Copyright (c) 2023 Battelle Energy Alliance, LLC. All rights reserved.
LABEL maintainer="[email protected]"
Expand Down Expand Up @@ -79,7 +79,7 @@ COPY --from=ghcr.io/mmguero-dev/gostatic --chmod=755 /goStatic /usr/bin/goStatic

RUN apt-get -q update \
&& apt-get -y -q --no-install-recommends upgrade \
&& apt-get -y -q --no-install-recommends install curl netcat rsync tini \
&& apt-get -y -q --no-install-recommends install curl netcat-openbsd rsync tini \
&& python3 -m pip install --upgrade pip \
&& python3 -m pip install --no-cache /wheels/* \
&& groupadd --gid ${DEFAULT_GID} ${PGROUP} \
Expand Down
25 changes: 13 additions & 12 deletions Dockerfiles/arkime.Dockerfile
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
FROM debian:11-slim AS build
FROM debian:12-slim AS build

# Copyright (c) 2023 Battelle Energy Alliance, LLC. All rights reserved.

ENV DEBIAN_FRONTEND noninteractive

ENV ARKIME_VERSION "v4.3.2"
ENV ARKIME_VERSION "v4.4.0"
ENV ARKIME_DIR "/opt/arkime"
ENV ARKIME_URL "https://github.com/arkime/arkime.git"
ENV ARKIME_LOCALELASTICSEARCH no
Expand Down Expand Up @@ -33,6 +33,7 @@ RUN apt-get -q update && \
libkrb5-dev \
libmaxminddb-dev \
libpcap0.8-dev \
libpcre3-dev \
libssl-dev \
libtool \
libwww-perl \
Expand All @@ -49,7 +50,7 @@ RUN apt-get -q update && \
swig \
wget \
zlib1g-dev && \
pip3 install --no-cache-dir beautifulsoup4 && \
python3 -m pip install --break-system-packages --no-cache-dir beautifulsoup4 meson && \
cd /opt && \
git clone --recurse-submodules --branch="$ARKIME_VERSION" "$ARKIME_URL" "./arkime-"$ARKIME_VERSION && \
cd "./arkime-"$ARKIME_VERSION && \
Expand All @@ -61,7 +62,7 @@ RUN apt-get -q update && \
python3 /opt/bs4_remove_div.py -i ./viewer/vueapp/src/components/users/Users.vue -o ./viewer/vueapp/src/components/users/Users.new -c "new-user-form" && \
mv -vf ./viewer/vueapp/src/components/users/Users.new ./viewer/vueapp/src/components/users/Users.vue && \
sed -i 's/v-if.*password.*"/v-if="false"/g' ./viewer/vueapp/src/components/settings/Settings.vue && \
rm -rf ./viewer/vueapp/src/components/upload ./capture/plugins/suricata* && \
rm -rf ./capture/plugins/suricata* && \
sed -i "s/^\(ARKIME_LOCALELASTICSEARCH=\).*/\1"$ARKIME_LOCALELASTICSEARCH"/" ./release/Configure && \
sed -i "s/^\(ARKIME_INET=\).*/\1"$ARKIME_INET"/" ./release/Configure && \
./easybutton-build.sh && \
Expand All @@ -71,7 +72,7 @@ RUN apt-get -q update && \
rm -f ${ARKIME_DIR}/wiseService/source.* ${ARKIME_DIR}/etc/*.systemd.service && \
bash -c "file ${ARKIME_DIR}/bin/* ${ARKIME_DIR}/node-v*/bin/* | grep 'ELF 64-bit' | sed 's/:.*//' | xargs -l -r strip -v --strip-unneeded"

FROM debian:11-slim
FROM debian:12-slim

LABEL maintainer="[email protected]"
LABEL org.opencontainers.image.authors='[email protected]'
Expand Down Expand Up @@ -133,7 +134,7 @@ ENV PCAP_MONITOR_HOST $PCAP_MONITOR_HOST

COPY --from=build $ARKIME_DIR $ARKIME_DIR

RUN sed -i "s/bullseye main/bullseye main contrib non-free/g" /etc/apt/sources.list && \
RUN sed -i "s/main$/main contrib non-free/g" /etc/apt/sources.list.d/debian.sources && \
apt-get -q update && \
apt-get -y -q --no-install-recommends upgrade && \
apt-get install -q -y --no-install-recommends \
Expand All @@ -148,14 +149,14 @@ RUN sed -i "s/bullseye main/bullseye main contrib non-free/g" /etc/apt/sources.l
libkrb5-3 \
libmaxminddb0 \
libpcap0.8 \
libssl1.0 \
libpcre3 \
libssl3 \
libtool \
libwww-perl \
libyaml-0-2 \
libzmq5 \
procps \
psmisc \
python \
python3 \
python3-pip \
python3-setuptools \
Expand All @@ -164,15 +165,15 @@ RUN sed -i "s/bullseye main/bullseye main contrib non-free/g" /etc/apt/sources.l
rsync \
sudo \
supervisor \
tini \
vim-tiny \
wget \
tini \
tar gzip unzip cpio bzip2 lzma xz-utils p7zip-full unrar zlib1g && \
pip3 install --no-cache-dir beautifulsoup4 pyzmq watchdog && \
python3 -m pip install --break-system-packages --no-cache-dir beautifulsoup4 pyzmq watchdog && \
ln -sfr $ARKIME_DIR/bin/npm /usr/local/bin/npm && \
ln -sfr $ARKIME_DIR/bin/node /usr/local/bin/node && \
ln -sfr $ARKIME_DIR/bin/npx /usr/local/bin/npx && \
apt-get -q -y --purge remove gcc gcc-10 cpp cpp-10 libssl-dev && \
apt-get -q -y --purge remove gcc gcc-12 cpp cpp-12 libssl-dev && \
apt-get -q -y autoremove && \
apt-get clean && \
rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
Expand Down Expand Up @@ -206,7 +207,7 @@ RUN [ ${#MAXMIND_GEOIP_DB_LICENSE_KEY} -gt 1 ] && for DB in ASN Country City; do
rm -f "GeoLite2-$DB*"; \
done; \
curl -s -S -L -o $ARKIME_DIR/etc/ipv4-address-space.csv "https://www.iana.org/assignments/ipv4-address-space/ipv4-address-space.csv" && \
curl -s -S -L -o $ARKIME_DIR/etc/oui.txt "https://raw.githubusercontent.com/wireshark/wireshark/master/manuf"
curl -s -S -L -o $ARKIME_DIR/etc/oui.txt "https://gitlab.com/wireshark/wireshark/raw/release-4.0/manuf"

RUN groupadd --gid $DEFAULT_GID $PGROUP && \
useradd -M --uid $DEFAULT_UID --gid $DEFAULT_GID --home $ARKIME_DIR $PUSER && \
Expand Down
4 changes: 2 additions & 2 deletions Dockerfiles/dashboards-helper.Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -47,10 +47,10 @@ ENV DASHBOARDS_URL $DASHBOARDS_URL
ENV DASHBOARDS_DARKMODE $DASHBOARDS_DARKMODE
ENV PATH="/data:${PATH}"

ENV SUPERCRONIC_VERSION "0.2.25"
ENV SUPERCRONIC_VERSION "0.2.26"
ENV SUPERCRONIC_URL "https://github.com/aptible/supercronic/releases/download/v$SUPERCRONIC_VERSION/supercronic-linux-amd64"
ENV SUPERCRONIC "supercronic-linux-amd64"
ENV SUPERCRONIC_SHA1SUM "642f4f5a2b67f3400b5ea71ff24f18c0a7d77d49"
ENV SUPERCRONIC_SHA1SUM "7a79496cf8ad899b99a719355d4db27422396735"
ENV SUPERCRONIC_CRONTAB "/etc/crontab"

ENV ECS_RELEASES_URL "https://api.github.com/repos/elastic/ecs/releases/latest"
Expand Down
4 changes: 2 additions & 2 deletions Dockerfiles/dashboards.Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -128,8 +128,8 @@ RUN yum upgrade -y && \
/usr/share/opensearch-dashboards/bin/opensearch-dashboards-plugin install file:///tmp/kbnSankeyVis.zip --allow-root && \
cd /tmp && \
# unzip transformVis.zip opensearch-dashboards/transformVis/opensearch_dashboards.json opensearch-dashboards/transformVis/package.json && \
# sed -i "s/2\.7\.0/2\.8\.0/g" opensearch-dashboards/transformVis/opensearch_dashboards.json && \
# sed -i "s/2\.7\.0/2\.8\.0/g" opensearch-dashboards/transformVis/package.json && \
# sed -i "s/2\.9\.0/2\.9\.0/g" opensearch-dashboards/transformVis/opensearch_dashboards.json && \
# sed -i "s/2\.9\.0/2\.9\.0/g" opensearch-dashboards/transformVis/package.json && \
# zip transformVis.zip opensearch-dashboards/transformVis/opensearch_dashboards.json opensearch-dashboards/transformVis/package.json && \
cd /usr/share/opensearch-dashboards/plugins && \
/usr/share/opensearch-dashboards/bin/opensearch-dashboards-plugin install file:///tmp/transformVis.zip --allow-root && \
Expand Down
18 changes: 9 additions & 9 deletions Dockerfiles/file-monitor.Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM debian:11-slim
FROM debian:12-slim

# Copyright (c) 2023 Battelle Energy Alliance, LLC. All rights reserved.
LABEL maintainer="[email protected]"
Expand Down Expand Up @@ -91,15 +91,15 @@ ENV EXTRACTED_FILE_HTTP_SERVER_ENCRYPT $EXTRACTED_FILE_HTTP_SERVER_ENCRYPT
ENV EXTRACTED_FILE_HTTP_SERVER_KEY $EXTRACTED_FILE_HTTP_SERVER_KEY
ENV EXTRACTED_FILE_HTTP_SERVER_PORT $EXTRACTED_FILE_HTTP_SERVER_PORT

ENV SUPERCRONIC_VERSION "0.2.25"
ENV SUPERCRONIC_VERSION "0.2.26"
ENV SUPERCRONIC_URL "https://github.com/aptible/supercronic/releases/download/v$SUPERCRONIC_VERSION/supercronic-linux-amd64"
ENV SUPERCRONIC "supercronic-linux-amd64"
ENV SUPERCRONIC_SHA1SUM "642f4f5a2b67f3400b5ea71ff24f18c0a7d77d49"
ENV SUPERCRONIC_SHA1SUM "7a79496cf8ad899b99a719355d4db27422396735"
ENV SUPERCRONIC_CRONTAB "/etc/crontab"

COPY --chmod=755 shared/bin/yara_rules_setup.sh /usr/local/bin/

RUN sed -i "s/bullseye main/bullseye main contrib non-free/g" /etc/apt/sources.list && \
RUN sed -i "s/main$/main contrib non-free/g" /etc/apt/sources.list.d/debian.sources && \
apt-get -q update && \
apt-get -y -q --no-install-recommends upgrade && \
apt-get install --no-install-recommends -y -q \
Expand All @@ -112,13 +112,13 @@ RUN sed -i "s/bullseye main/bullseye main contrib non-free/g" /etc/apt/sources.l
gcc \
git \
jq \
libclamunrar9 \
libclamunrar11 \
libjansson-dev \
libjansson4 \
libmagic-dev \
libmagic1 \
libssl-dev \
libssl1.1 \
libssl3 \
libtool \
make \
pkg-config \
Expand All @@ -135,7 +135,7 @@ RUN sed -i "s/bullseye main/bullseye main contrib non-free/g" /etc/apt/sources.l
python3-requests \
python3-zmq \
rsync && \
pip3 install clamd supervisor yara-python python-magic psutil pycryptodome watchdog && \
python3 -m pip install --break-system-packages --no-cache-dir clamd supervisor yara-python python-magic psutil pycryptodome watchdog && \
curl -fsSLO "$SUPERCRONIC_URL" && \
echo "${SUPERCRONIC_SHA1SUM} ${SUPERCRONIC}" | sha1sum -c - && \
chmod +x "$SUPERCRONIC" && \
Expand Down Expand Up @@ -167,9 +167,9 @@ RUN sed -i "s/bullseye main/bullseye main contrib non-free/g" /etc/apt/sources.l
automake \
build-essential \
gcc \
gcc-8 \
gcc-12 \
libc6-dev \
libgcc-8-dev \
libgcc-12-dev \
libjansson-dev \
libmagic-dev \
libssl-dev \
Expand Down
Loading

0 comments on commit d5ee03e

Please sign in to comment.