Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Malcolm v24.09.0 #336

Merged
merged 69 commits into from
Sep 19, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
4cb7644
idaholab/Malcolm#487, no option to go backwards in installer
mmguero Aug 26, 2024
d61fdd3
idaholab/Malcolm#553, include netbox-topology plugin by default
mmguero Aug 28, 2024
b4d4278
Bump to v24.09.0 for development
mmguero Aug 28, 2024
ebc061b
Merge branch 'development' of https://github.com/mmguero-dev/Malcolm …
mmguero Aug 28, 2024
6aa9378
idaholab/Malcolm#487; no 'back' option in config.py
mmguero Aug 28, 2024
490eccb
idaholab/Malcolm#487; no 'back' option in config.py
mmguero Aug 28, 2024
e4d49f4
fix idaholab/Malcolm#554, filtering on Hunt ID in Arkime not working
mmguero Aug 28, 2024
b035061
idaholab/Malcolm#450, allow splitting out indexes by other field values
mmguero Aug 28, 2024
d64970a
should not affect anyone that does't explicitly specify it, but allow…
mmguero Aug 29, 2024
d0a8b05
bump various component versions
mmguero Sep 5, 2024
2363779
incorporate changes for enip packet correlation ID (see cisagov/icsnp…
mmguero Sep 5, 2024
f666788
bump beats and logstash to 8.15.1
mmguero Sep 9, 2024
abaadbd
fix logstash syntax error
mmguero Sep 9, 2024
f5e37c5
update enip/cip dashboard for packet correlation id
mmguero Sep 9, 2024
f1124c6
Update Watchdog package to v5.0.2 (idaholab/Malcolm#550)
mmguero Sep 9, 2024
6065eb9
allow users to use the Arkime Lua plugin without having to create new…
mmguero Sep 9, 2024
f81becc
bump Yara, OSD_Transform, and elasticsearch-py
mmguero Sep 10, 2024
d6d77e6
slides update
mmguero Sep 10, 2024
800f3f5
fix slides
mmguero Sep 10, 2024
456ee50
Fix formatting for doc
mmguero Sep 11, 2024
6c37b75
remove some samples that aren't needed
mmguero Sep 11, 2024
3452580
documentation
mmguero Sep 11, 2024
ef5e672
Set ARKIME_NODE_HOST from OS_HOST interface rather than default route
divinehawk Sep 11, 2024
ba4a7e9
for idaholab/Malcolm#557, install zeek-iec104 parser (logging not ena…
mmguero Sep 11, 2024
c6f66a8
for idaholab/Malcolm#560 (see also idaholab/Malcolm#559), slight twea…
mmguero Sep 11, 2024
f7f9d02
Revert "for idaholab/Malcolm#557, install zeek-iec104 parser (logging…
mmguero Sep 11, 2024
8af3b0f
bump postgres to 16-alpine for netbox
mmguero Sep 11, 2024
22de09f
added user-ns for podman idaholab/Malcolm#407
mmguero Sep 11, 2024
eddac68
github_image_helper for podman idaholab/Malcolm#407
mmguero Sep 11, 2024
4505774
work in progress for podman idaholab/Malcolm#407
mmguero Sep 11, 2024
cfd629a
for podman, make sure initial USER is 'root' in docker containers so …
mmguero Sep 11, 2024
d24f7dd
for podman, some config/install/control script stuff; idaholab/Malcol…
mmguero Sep 11, 2024
bc00264
for podman, some config/install/control script stuff; idaholab/Malcol…
mmguero Sep 11, 2024
ac37cb5
for podman, some config/install/control script stuff; idaholab/Malcol…
mmguero Sep 11, 2024
0e05e26
for podman, documentation stuff; idaholab/Malcolm#407
mmguero Sep 11, 2024
9252f09
for podman, some config/install/control script stuff; idaholab/Malcol…
mmguero Sep 12, 2024
73f1625
for podman, some documentation stuff; idaholab/Malcolm#407
mmguero Sep 12, 2024
24539fb
fix hedgehog run profile by removing ports from docker-compose
mmguero Sep 12, 2024
03e8c00
capture needs wise
mmguero Sep 12, 2024
fe91ce4
for podman, some documentation stuff; idaholab/Malcolm#407
mmguero Sep 12, 2024
a4b1f0c
for podman, some documentation stuff; idaholab/Malcolm#407
mmguero Sep 12, 2024
2dbe5a1
for podman, some documentation stuff; idaholab/Malcolm#407
mmguero Sep 12, 2024
10358be
bump supercronic to v0.2.32
mmguero Sep 12, 2024
f9067e5
change default logging for podman to json-file
mmguero Sep 16, 2024
87086cd
for idaholab/Malcolm#446, allow specifying OPENSEARCH_INDEX_SIZE_PRUN…
mmguero Sep 16, 2024
05c2439
for idaholab/Malcolm#446, allow specifying OPENSEARCH_INDEX_SIZE_PRUN…
mmguero Sep 16, 2024
0f97828
script update
mmguero Sep 16, 2024
54e887c
improvements to documentation and install.py for Linux performance tw…
mmguero Sep 16, 2024
d619840
fixes to remove redundant targetplatform from docker images; also sta…
mmguero Sep 16, 2024
e9c88d9
work in progress for hart ip integration, idaholab/Malcolm#561
mmguero Sep 16, 2024
e76a7a4
make option for *not* overriding tmp directory
mmguero Sep 17, 2024
6c69a92
Fixes for template for hartip (see idaholab/Malcolm#561)
mmguero Sep 17, 2024
4e245e1
fix offline suricata Docker container does not initialize suricata co…
mmguero Sep 17, 2024
969c8b1
work in progress hartip (see idaholab/Malcolm#561)
mmguero Sep 17, 2024
5faca03
idaholab/Malcolm#565, allow specifying alternate download location fo…
mmguero Sep 17, 2024
af2ccd9
idaholab/Malcolm#565, allow specifying alternate download location fo…
mmguero Sep 17, 2024
8a7622e
fix install.py to restore -h/--help functionality
mmguero Sep 17, 2024
57bdb84
work in progress hartip (see idaholab/Malcolm#561)
mmguero Sep 17, 2024
296043c
Add opensearch api call to create indices
melaniepierce Sep 17, 2024
0cb0711
update opensearch and dashboards to v2.17.0
mmguero Sep 17, 2024
75790e2
bump fluent-bit script for v3.1.8
mmguero Sep 18, 2024
4f527ad
work in progress hartip (see idaholab/Malcolm#561)
mmguero Sep 18, 2024
9029733
script fix
mmguero Sep 18, 2024
b7594d9
work in progress for hart ip integration, idaholab/Malcolm#561
mmguero Sep 18, 2024
51f7587
initial dashboard for hart ip integration, idaholab/Malcolm#561
mmguero Sep 18, 2024
2f0e259
added hart-ip dashboard
mmguero Sep 18, 2024
ab28355
Merge branch 'development' of https://github.com/mmguero-dev/Malcolm …
mmguero Sep 18, 2024
7af48a0
bump elasticsearch-dsl
mmguero Sep 18, 2024
150674d
Merge branch 'development' of https://github.com/mmguero-dev/Malcolm …
mmguero Sep 18, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/arkime-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,7 @@ jobs:
BUILD_DATE=${{ steps.generate_build_timestamp.outputs.btimestamp }}
VCS_REVISION=${{ steps.extract_commit_sha.outputs.sha }}
MAXMIND_GEOIP_DB_LICENSE_KEY=${{ secrets.MAXMIND_GEOIP_DB_LICENSE_KEY }}
MAXMIND_GEOIP_DB_ALTERNATE_DOWNLOAD_URL=${{ secrets.MAXMIND_GEOIP_DB_ALTERNATE_DOWNLOAD_URL }}
push: true
provenance: false
platforms: ${{ matrix.platform }}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,7 @@ jobs:
pushd ./hedgehog-iso
echo "${{ steps.extract_malcolm_version.outputs.mversion }}" > ./shared/version.txt
echo "${{ secrets.MAXMIND_GEOIP_DB_LICENSE_KEY }}" > ./shared/maxmind_license.txt
echo "${{ secrets.MAXMIND_GEOIP_DB_ALTERNATE_DOWNLOAD_URL }}" > ./shared/maxmind_url.txt
echo "GITHUB_TOKEN=${{ secrets.GITHUB_TOKEN }}" > ./shared/environment.chroot
echo "VCS_REVSION=${{ steps.extract_commit_sha.outputs.sha }}" > ./shared/environment.chroot
echo "BUILD_JOBS=2" > ./shared/environment.chroot
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,7 @@ jobs:
mkdir -p ./shared
echo "${{ steps.extract_malcolm_version.outputs.mversion }}" > ./shared/version.txt
echo "${{ secrets.MAXMIND_GEOIP_DB_LICENSE_KEY }}" > ./shared/maxmind_license.txt
echo "${{ secrets.MAXMIND_GEOIP_DB_ALTERNATE_DOWNLOAD_URL }}" > ./shared/maxmind_url.txt
echo "GITHUB_TOKEN=${{ secrets.GITHUB_TOKEN }}" > ./shared/environment.chroot
echo "VCS_REVSION=${{ steps.extract_commit_sha.outputs.sha }}" > ./shared/environment.chroot
echo "BUILD_JOBS=2" > ./shared/environment.chroot
Expand Down
7 changes: 3 additions & 4 deletions Dockerfiles/api.Dockerfile
Original file line number Diff line number Diff line change
@@ -1,6 +1,4 @@
ARG TARGETPLATFORM=linux/amd64

FROM --platform=${TARGETPLATFORM} python:3-slim-bookworm as builder
FROM python:3-slim-bookworm as builder

ENV DEBIAN_FRONTEND noninteractive
ENV TERM xterm
Expand All @@ -20,7 +18,7 @@ WORKDIR /usr/src/app
RUN python3 -m pip wheel --no-cache-dir --no-deps --wheel-dir /usr/src/app/wheels -r requirements.txt \
&& flake8 --ignore=E203,E501,F401,W503

FROM --platform=${TARGETPLATFORM} python:3-slim-bookworm
FROM python:3-slim-bookworm

# Copyright (c) 2024 Battelle Energy Alliance, LLC. All rights reserved.
LABEL maintainer="[email protected]"
Expand All @@ -39,6 +37,7 @@ ENV DEFAULT_GID $DEFAULT_GID
ENV PUSER "yeflask"
ENV PGROUP "yeflask"
ENV PUSER_PRIV_DROP true
USER root

ENV DEBIAN_FRONTEND noninteractive
ENV TERM xterm
Expand Down
26 changes: 11 additions & 15 deletions Dockerfiles/arkime.Dockerfile
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
ARG TARGETPLATFORM=linux/amd64

# Copyright (c) 2024 Battelle Energy Alliance, LLC. All rights reserved.

FROM --platform=${TARGETPLATFORM} debian:12-slim
FROM debian:12-slim

LABEL maintainer="[email protected]"
LABEL org.opencontainers.image.authors='[email protected]'
Expand All @@ -27,6 +25,7 @@ ENV PGROUP "arkime"
# a final check in docker_entrypoint.sh before startup
ENV PUSER_PRIV_DROP false
ENV PUSER_RLIMIT_UNLOCK true
USER root

ENV DEBIAN_FRONTEND noninteractive
ENV TERM xterm
Expand Down Expand Up @@ -65,6 +64,7 @@ ARG PCAP_PIPELINE_VERBOSITY=""
ARG PCAP_MONITOR_HOST=pcap-monitor
ARG PCAP_NODE_NAME=malcolm
ARG MAXMIND_GEOIP_DB_LICENSE_KEY=""
ARG MAXMIND_GEOIP_DB_ALTERNATE_DOWNLOAD_URL=""

# Declare envs vars for each arg
ENV MALCOLM_USERNAME $MALCOLM_USERNAME
Expand Down Expand Up @@ -123,6 +123,7 @@ RUN export DEBARCH=$(dpkg --print-architecture) && \
libyaml-dev \
libyara9 \
libzmq5 \
lua5.4 \
lzma \
p7zip-full \
procps \
Expand All @@ -148,7 +149,7 @@ RUN export DEBARCH=$(dpkg --print-architecture) && \
mkdir -p "${ARKIME_DIR}"/plugins && \
curl -fsSL -o "${ARKIME_DIR}/plugins/ja4plus.${DEBARCH}.so" "$(echo "${ARKIME_JA4_SO_URL}" | sed "s/XXX/${DEBARCH}/g")" && \
chmod 755 "${ARKIME_DIR}/plugins/ja4plus.${DEBARCH}.so" && \
python3 -m pip install --break-system-packages --no-compile --no-cache-dir beautifulsoup4 pyzmq watchdog==4.0.2 && \
python3 -m pip install --break-system-packages --no-compile --no-cache-dir beautifulsoup4 pyzmq watchdog==5.0.2 && \
ln -sfr $ARKIME_DIR/bin/npm /usr/local/bin/npm && \
ln -sfr $ARKIME_DIR/bin/node /usr/local/bin/node && \
ln -sfr $ARKIME_DIR/bin/npx /usr/local/bin/npx && \
Expand All @@ -161,6 +162,7 @@ RUN export DEBARCH=$(dpkg --print-architecture) && \
COPY --chmod=755 shared/bin/docker-uid-gid-setup.sh /usr/local/bin/
COPY --chmod=755 shared/bin/service_check_passthrough.sh /usr/local/bin/
COPY --chmod=755 shared/bin/self_signed_key_gen.sh /usr/local/bin/
COPY --chmod=755 shared/bin/maxmind-mmdb-download.sh /usr/local/bin/
COPY --chmod=755 shared/bin/nic-capture-setup.sh /usr/local/bin/
COPY --chmod=755 shared/bin/opensearch_status.sh /opt
COPY --chmod=755 shared/bin/pcap_processor.py /opt/
Expand All @@ -179,15 +181,9 @@ COPY --from=ghcr.io/mmguero-dev/gostatic --chmod=755 /goStatic /usr/bin/goStatic
# see https://dev.maxmind.com/geoip/geoipupdate/#Direct_Downloads
# see https://github.com/arkime/arkime/issues/1350
# see https://github.com/arkime/arkime/issues/1352
RUN [ ${#MAXMIND_GEOIP_DB_LICENSE_KEY} -gt 1 ] && for DB in ASN Country City; do \
cd /tmp && \
curl -s -S -L -o "GeoLite2-$DB.mmdb.tar.gz" "https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-$DB&license_key=$MAXMIND_GEOIP_DB_LICENSE_KEY&suffix=tar.gz" && \
tar xf "GeoLite2-$DB.mmdb.tar.gz" --wildcards --no-anchored '*.mmdb' --strip=1 && \
mv -v "GeoLite2-$DB.mmdb" $ARKIME_DIR/etc/; \
rm -f "GeoLite2-$DB*"; \
done; \
curl -s -S -L -o $ARKIME_DIR/etc/ipv4-address-space.csv "https://www.iana.org/assignments/ipv4-address-space/ipv4-address-space.csv" && \
curl -s -S -L -o $ARKIME_DIR/etc/oui.txt "https://www.wireshark.org/download/automated/data/manuf"
RUN ( /usr/local/bin/maxmind-mmdb-download.sh -o $ARKIME_DIR/etc || true ) && \
curl -s -S -L -o $ARKIME_DIR/etc/ipv4-address-space.csv "https://www.iana.org/assignments/ipv4-address-space/ipv4-address-space.csv" && \
curl -s -S -L -o $ARKIME_DIR/etc/oui.txt "https://www.wireshark.org/download/automated/data/manuf"

RUN groupadd --gid $DEFAULT_GID $PGROUP && \
useradd -M --uid $DEFAULT_UID --gid $DEFAULT_GID --home $ARKIME_DIR $PUSER && \
Expand All @@ -201,8 +197,8 @@ RUN groupadd --gid $DEFAULT_GID $PGROUP && \
setcap 'CAP_NET_RAW+eip CAP_NET_ADMIN+eip CAP_IPC_LOCK+eip' $ARKIME_DIR/bin/capture && \
chown root:${PGROUP} /sbin/ethtool && \
setcap 'CAP_NET_RAW+eip CAP_NET_ADMIN+eip' /sbin/ethtool && \
mkdir -p /var/run/arkime $ARKIME_DIR/logs && \
chown -R $PUSER:$PGROUP $ARKIME_DIR/etc $ARKIME_DIR/rules $ARKIME_DIR/logs /var/run/arkime
mkdir -p /var/run/arkime $ARKIME_DIR/logs $ARKIME_DIR/lua && \
chown -R $PUSER:$PGROUP $ARKIME_DIR/etc $ARKIME_DIR/lua $ARKIME_DIR/rules $ARKIME_DIR/logs /var/run/arkime
#Update Path
ENV PATH="/opt:$ARKIME_DIR/bin:${PATH}"

Expand Down
14 changes: 10 additions & 4 deletions Dockerfiles/dashboards-helper.Dockerfile
Original file line number Diff line number Diff line change
@@ -1,6 +1,4 @@
ARG TARGETPLATFORM=linux/amd64

FROM --platform=${TARGETPLATFORM} debian:12-slim
FROM debian:12-slim

# Copyright (c) 2020 Battelle Energy Alliance, LLC. All rights reserved.
LABEL maintainer="[email protected]"
Expand All @@ -18,7 +16,14 @@ ENV DEFAULT_UID $DEFAULT_UID
ENV DEFAULT_GID $DEFAULT_GID
ENV PUSER "helper"
ENV PGROUP "helper"
# This is to handle an issue when running with rootless podman and
# "userns_mode: keep-id". It seems that anything defined as a VOLUME
# in the Dockerfile is getting set with an ownership of 999:999.
# This is to override that, although I'm not yet sure if there are
# other implications. See containers/podman#23347.
ENV PUSER_CHOWN "/data/init"
ENV PUSER_PRIV_DROP true
USER root

ENV TERM xterm

Expand All @@ -37,7 +42,7 @@ ENV OPENSEARCH_DEFAULT_DASHBOARD $OPENSEARCH_DEFAULT_DASHBOARD
ENV DASHBOARDS_DARKMODE $DASHBOARDS_DARKMODE
ENV PATH="/data:${PATH}"

ENV SUPERCRONIC_VERSION "0.2.30"
ENV SUPERCRONIC_VERSION "0.2.32"
ENV SUPERCRONIC_URL "https://github.com/aptible/supercronic/releases/download/v$SUPERCRONIC_VERSION/supercronic-linux-"
ENV SUPERCRONIC_CRONTAB "/etc/crontab"

Expand Down Expand Up @@ -114,6 +119,7 @@ ENTRYPOINT ["/usr/bin/tini", \

CMD ["/usr/local/bin/supervisord", "-c", "/etc/supervisord.conf", "-n"]

# see PUSER_CHOWN comment above
VOLUME ["/data/init"]

# to be populated at build-time:
Expand Down
11 changes: 5 additions & 6 deletions Dockerfiles/dashboards.Dockerfile
Original file line number Diff line number Diff line change
@@ -1,6 +1,4 @@
ARG TARGETPLATFORM=linux/amd64

FROM --platform=${TARGETPLATFORM} opensearchproject/opensearch-dashboards:2.16.0
FROM opensearchproject/opensearch-dashboards:2.17.0

LABEL maintainer="[email protected]"
LABEL org.opencontainers.image.authors='[email protected]'
Expand All @@ -18,13 +16,14 @@ ENV DEFAULT_GID $DEFAULT_GID
ENV PUSER "opensearch-dashboards"
ENV PGROUP "opensearch-dashboards"
ENV PUSER_PRIV_DROP true
USER root

ENV TERM xterm

ENV TINI_VERSION v0.19.0
ENV TINI_URL https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini

ENV OSD_TRANSFORM_VIS_VERSION 2.15.0
ENV OSD_TRANSFORM_VIS_VERSION 2.16.0

ARG NODE_OPTIONS="--max_old_space_size=4096"
ENV NODE_OPTIONS $NODE_OPTIONS
Expand All @@ -44,8 +43,8 @@ RUN export BINARCH=$(uname -m | sed 's/x86_64/amd64/' | sed 's/aarch64/arm64/')
/usr/share/opensearch-dashboards/bin/opensearch-dashboards-plugin remove securityDashboards --allow-root && \
cd /tmp && \
unzip transformVis.zip opensearch-dashboards/transformVis/opensearch_dashboards.json opensearch-dashboards/transformVis/package.json && \
sed -i "s/2\.15\.0/2\.16\.0/g" opensearch-dashboards/transformVis/opensearch_dashboards.json && \
sed -i "s/2\.15\.0/2\.16\.0/g" opensearch-dashboards/transformVis/package.json && \
sed -i "s/2\.16\.0/2\.17\.0/g" opensearch-dashboards/transformVis/opensearch_dashboards.json && \
sed -i "s/2\.16\.0/2\.17\.0/g" opensearch-dashboards/transformVis/package.json && \
zip transformVis.zip opensearch-dashboards/transformVis/opensearch_dashboards.json opensearch-dashboards/transformVis/package.json && \
cd /usr/share/opensearch-dashboards/plugins && \
/usr/share/opensearch-dashboards/bin/opensearch-dashboards-plugin install file:///tmp/transformVis.zip --allow-root && \
Expand Down
5 changes: 2 additions & 3 deletions Dockerfiles/dirinit.Dockerfile
Original file line number Diff line number Diff line change
@@ -1,6 +1,4 @@
ARG TARGETPLATFORM=linux/amd64

FROM --platform=${TARGETPLATFORM} alpine:3.20
FROM alpine:3.20

# Copyright (c) 2024 Battelle Energy Alliance, LLC. All rights reserved.
LABEL maintainer="[email protected]"
Expand All @@ -19,6 +17,7 @@ ENV DEFAULT_GID $DEFAULT_GID
ENV PUSER "dirinit"
ENV PGROUP "dirinit"
ENV PUSER_PRIV_DROP true
USER root

ENV TERM xterm

Expand Down
21 changes: 14 additions & 7 deletions Dockerfiles/file-monitor.Dockerfile
Original file line number Diff line number Diff line change
@@ -1,6 +1,4 @@
ARG TARGETPLATFORM=linux/amd64

FROM --platform=${TARGETPLATFORM} debian:12-slim
FROM debian:12-slim

# Copyright (c) 2024 Battelle Energy Alliance, LLC. All rights reserved.
LABEL maintainer="[email protected]"
Expand All @@ -19,6 +17,8 @@ ENV DEFAULT_GID $DEFAULT_GID
ENV PUSER "monitor"
ENV PGROUP "monitor"
ENV PUSER_PRIV_DROP true
# see PUSER_CHOWN at the bottom of the file (after the other environment variables it references)
USER root

ENV DEBIAN_FRONTEND noninteractive
ENV TERM xterm
Expand Down Expand Up @@ -88,7 +88,7 @@ ENV EXTRACTED_FILE_ENABLE_CAPA $EXTRACTED_FILE_ENABLE_CAPA
ENV EXTRACTED_FILE_CAPA_VERBOSE $EXTRACTED_FILE_CAPA_VERBOSE
ENV SRC_BASE_DIR "/usr/local/src"
ENV CLAMAV_RULES_DIR "/var/lib/clamav"
ENV YARA_VERSION "4.5.1"
ENV YARA_VERSION "4.5.2"
ENV YARA_URL "https://github.com/VirusTotal/yara/archive/v${YARA_VERSION}.tar.gz"
ENV YARA_RULES_SRC_DIR "/yara-rules-src"
ENV YARA_RULES_DIR "/yara-rules"
Expand All @@ -100,7 +100,7 @@ ENV EXTRACTED_FILE_HTTP_SERVER_KEY $EXTRACTED_FILE_HTTP_SERVER_KEY
ENV EXTRACTED_FILE_HTTP_SERVER_RECURSIVE $EXTRACTED_FILE_HTTP_SERVER_RECURSIVE
ENV EXTRACTED_FILE_HTTP_SERVER_PORT $EXTRACTED_FILE_HTTP_SERVER_PORT

ENV SUPERCRONIC_VERSION "0.2.30"
ENV SUPERCRONIC_VERSION "0.2.32"
ENV SUPERCRONIC_URL "https://github.com/aptible/supercronic/releases/download/v$SUPERCRONIC_VERSION/supercronic-linux-"
ENV SUPERCRONIC_CRONTAB "/etc/crontab"

Expand Down Expand Up @@ -159,7 +159,7 @@ RUN export BINARCH=$(uname -m | sed 's/x86_64/amd64/' | sed 's/aarch64/arm64/')
python-magic \
stream-zip \
supervisor \
watchdog==4.0.2 \
watchdog==5.0.2 \
yara-python && \
curl -fsSL -o /usr/local/bin/supercronic "${SUPERCRONIC_URL}${BINARCH}" && \
chmod +x /usr/local/bin/supercronic && \
Expand Down Expand Up @@ -237,11 +237,18 @@ COPY --chmod=644 shared/bin/watch_common.py /usr/local/bin/
COPY --chmod=644 scripts/malcolm_utils.py /usr/local/bin/
COPY --chmod=644 file-monitor/supervisord.conf /etc/supervisord.conf
COPY --chmod=755 file-monitor/docker-entrypoint.sh /docker-entrypoint.sh
COPY --chmod=755 file-monitor/*update.sh /usr/local/bin/
COPY --from=ghcr.io/mmguero-dev/gostatic --chmod=755 /goStatic /usr/bin/goStatic

WORKDIR /zeek/extract_files

# This is to handle an issue when running with rootless podman and
# "userns_mode: keep-id". It seems that anything defined as a VOLUME
# in the Dockerfile is getting set with an ownership of 999:999.
# This is to override that, although I'm not yet sure if there are
# other implications. See containers/podman#23347.
ENV PUSER_CHOWN "$CLAMAV_RULES_DIR;$YARA_RULES_DIR;$YARA_RULES_SRC_DIR"

# see PUSER_CHOWN comment above
VOLUME ["$CLAMAV_RULES_DIR"]
VOLUME ["$YARA_RULES_DIR"]
VOLUME ["$YARA_RULES_SRC_DIR"]
Expand Down
17 changes: 12 additions & 5 deletions Dockerfiles/file-upload.Dockerfile
Original file line number Diff line number Diff line change
@@ -1,6 +1,4 @@
ARG TARGETPLATFORM=linux/amd64

FROM --platform=${TARGETPLATFORM} debian:12-slim AS npmget
FROM debian:12-slim AS npmget

# Copyright (c) 2024 Battelle Energy Alliance, LLC. All rights reserved.

Expand All @@ -17,7 +15,7 @@ RUN apt-get -q update && \
filepond-plugin-file-rename \
@jcubic/tagger

FROM --platform=${TARGETPLATFORM} debian:12-slim AS runtime
FROM debian:12-slim AS runtime

LABEL maintainer="[email protected]"
LABEL org.opencontainers.image.authors='[email protected]'
Expand All @@ -34,10 +32,17 @@ ENV DEFAULT_UID $DEFAULT_UID
ENV DEFAULT_GID $DEFAULT_GID
ENV PUSER "www-data"
ENV PGROUP "www-data"
# This is to handle an issue when running with rootless podman and
# "userns_mode: keep-id". It seems that anything defined as a VOLUME
# in the Dockerfile is getting set with an ownership of 999:999.
# This is to override that, although I'm not yet sure if there are
# other implications. See containers/podman#23347.
ENV PUSER_CHOWN "/var/www/upload/server/php/chroot/files"
# not dropping privileges globally in this container as required to run SFTP server. this can
# be handled by supervisord instead on an as-needed basis, and/or php-fpm/nginx itself
# will drop privileges to www-data as well.
ENV PUSER_PRIV_DROP false
USER root

ENV DEBIAN_FRONTEND noninteractive
ENV TERM xterm
Expand All @@ -51,7 +56,7 @@ ENV FILEPOND_SERVER_BRANCH $FILEPOND_SERVER_BRANCH
ARG STALE_UPLOAD_DELETE_MIN=360
ENV STALE_UPLOAD_DELETE_MIN $STALE_UPLOAD_DELETE_MIN

ENV SUPERCRONIC_VERSION "0.2.30"
ENV SUPERCRONIC_VERSION "0.2.32"
ENV SUPERCRONIC_URL "https://github.com/aptible/supercronic/releases/download/v$SUPERCRONIC_VERSION/supercronic-linux-"
ENV SUPERCRONIC_CRONTAB "/etc/crontab"

Expand Down Expand Up @@ -122,7 +127,9 @@ RUN mkdir -p /run/php \
>/var/www/upload/server/php/chroot/README.txt && \
rm -rf /var/lib/apt/lists/* /var/cache/* /tmp/* /var/tmp/*

# see PUSER_CHOWN comment above
VOLUME [ "/var/www/upload/server/php/chroot/files" ]

EXPOSE 22 80

ENTRYPOINT ["/usr/bin/tini", \
Expand Down
18 changes: 13 additions & 5 deletions Dockerfiles/filebeat.Dockerfile
Original file line number Diff line number Diff line change
@@ -1,6 +1,4 @@
ARG TARGETPLATFORM=linux/amd64

FROM --platform=${TARGETPLATFORM} docker.elastic.co/beats/filebeat-oss:8.15.0
FROM docker.elastic.co/beats/filebeat-oss:8.15.1

# Copyright (c) 2024 Battelle Energy Alliance, LLC. All rights reserved.
LABEL maintainer="[email protected]"
Expand All @@ -18,10 +16,17 @@ ENV DEFAULT_UID $DEFAULT_UID
ENV DEFAULT_GID $DEFAULT_GID
ENV PUSER "filebeat"
ENV PGROUP "filebeat"
# This is to handle an issue when running with rootless podman and
# "userns_mode: keep-id". It seems that anything defined as a VOLUME
# in the Dockerfile is getting set with an ownership of 999:999.
# This is to override that, although I'm not yet sure if there are
# other implications. See containers/podman#23347.
ENV PUSER_CHOWN "/usr/share/filebeat-logs/data;/usr/share/filebeat-nginx/data;/usr/share/filebeat-tcp/data"
# not dropping privileges globally: supervisord will take care of it
# on a case-by-case basis so that one script (filebeat-watch-zeeklogs-uploads-folder.py)
# can chown uploaded files
ENV PUSER_PRIV_DROP false
USER root

ENV DEBIAN_FRONTEND noninteractive
ENV TERM xterm
Expand Down Expand Up @@ -62,7 +67,7 @@ ARG FILEBEAT_TCP_PARSE_DROP_FIELD=""
ARG FILEBEAT_TCP_TAG="_malcolm_beats"
ARG PCAP_NODE_NAME=malcolm

ENV SUPERCRONIC_VERSION "0.2.30"
ENV SUPERCRONIC_VERSION "0.2.32"
ENV SUPERCRONIC_URL "https://github.com/aptible/supercronic/releases/download/v$SUPERCRONIC_VERSION/supercronic-linux-"
ENV SUPERCRONIC_CRONTAB "/etc/crontab"

Expand Down Expand Up @@ -95,13 +100,15 @@ RUN export EVTXARCH=$(uname -m | sed 's/arm64/aarch64/') && \
psmisc \
python3-pip \
python3-setuptools \
python3.9 \
rsync \
tar \
tini \
unar \
unzip \
xz-utils && \
python3 -m pip install --no-compile --no-cache-dir patool entrypoint2 pyunpack python-magic ordered-set supervisor watchdog==4.0.2 && \
ln -s -f -r /usr/bin/python3.9 /usr/bin/python3 && \
python3.9 -m pip install --no-compile --no-cache-dir patool entrypoint2 pyunpack python-magic ordered-set supervisor watchdog==5.0.2 && \
curl -fsSL -o /usr/local/bin/supercronic "${SUPERCRONIC_URL}${BINARCH}" && \
chmod +x /usr/local/bin/supercronic && \
curl -fsSL -o /usr/local/bin/yq "${YQ_URL}${BINARCH}" && \
Expand Down Expand Up @@ -170,6 +177,7 @@ ENV FILEBEAT_REGISTRY_FILE "/usr/share/filebeat-logs/data/registry/filebeat/log.
ENV FILEBEAT_ZEEK_DIR "/zeek/"
ENV PCAP_NODE_NAME $PCAP_NODE_NAME

# see PUSER_CHOWN comment above
VOLUME ["/usr/share/filebeat-logs/data", "/usr/share/filebeat-nginx/data", "/usr/share/filebeat-tcp/data"]

ENTRYPOINT ["/usr/bin/tini", \
Expand Down
Loading