-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Create Docker installation files #7
Open
timwoj
wants to merge
17
commits into
master
Choose a base branch
from
topic/timw/docker
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
17 commits
Select commit
Hold shift + click to select a range
63f14e6
Add pre-commit configuration for various linters
timwoj 6e72e09
Add docker configuration for running packages site
timwoj 9b4d336
Upgrade package versions in composer.json slightly
timwoj c8e0416
Minor changes required for cake 3.7
timwoj 01e574d
Remove unused settings from secrets.env
timwoj 355d5b7
Minor fixes to the setup for nginx/php
timwoj 3cb1866
Add disabled service for mariadb (for testing)
timwoj 7f70fe6
Add cron to php image to run the update job
timwoj ab28b39
Update README.md with new setup process
timwoj b009858
Add certbot for managing a Lets Encrypt cert
timwoj c413099
Cleanup the zkg scratch directory while refreshing packages
timwoj 84d1b49
Extend timeout for zkg info to 120s
timwoj 2091fbb
Lint docker files
bbannier 4058189
Add shellcheck pre-commit, fix reported issues
timwoj fcc4453
Move mariadb configuration to a separate docker compose file
timwoj 606fb82
fixup! Add certbot for managing a Lets Encrypt cert
timwoj a03a8b5
Minor changes to ssl-update.sh
timwoj File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
name: pre-commit | ||
|
||
on: | ||
pull_request: | ||
push: | ||
branches: [master] | ||
|
||
jobs: | ||
pre-commit: | ||
runs-on: ubuntu-22.04 | ||
steps: | ||
- uses: actions/checkout@v4 | ||
- uses: actions/setup-python@v5 | ||
- uses: pre-commit/[email protected] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
data |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,31 @@ | ||
# See https://pre-commit.com for more information | ||
# See https://pre-commit.com/hooks.html for more hooks | ||
# | ||
repos: | ||
- repo: https://github.com/pre-commit/pre-commit-hooks | ||
rev: v5.0.0 | ||
hooks: | ||
- id: check-yaml | ||
- id: end-of-file-fixer | ||
- id: trailing-whitespace | ||
|
||
- repo: https://github.com/maxwinterstein/shfmt-py | ||
rev: v3.7.0.1 | ||
hooks: | ||
- id: shfmt | ||
args: ["-w", "-i", "4", "-ci"] | ||
|
||
- repo: https://github.com/crate-ci/typos | ||
rev: v1.29.5 | ||
hooks: | ||
- id: typos | ||
|
||
- repo: https://github.com/hadolint/hadolint | ||
rev: v2.12.0 | ||
hooks: | ||
- id: hadolint-docker | ||
|
||
- repo: https://github.com/koalaman/shellcheck-precommit | ||
rev: v0.10.0 | ||
hooks: | ||
- id: shellcheck |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change | ||||||||
---|---|---|---|---|---|---|---|---|---|---|
|
@@ -8,104 +8,48 @@ git clone [email protected]:zeek/zeek-pkg-web.git | |||||||||
cd zeek-pkg-web | ||||||||||
``` | ||||||||||
|
||||||||||
## Copy files/directories | ||||||||||
``` | ||||||||||
sudo cp -a bropkg /var/www | ||||||||||
### Modify salt, database password, and CILogon client id/key in | ||||||||||
### secrets/.env to something suitable for your installation | ||||||||||
cp secrets/.env /var/www/bropkg/config/ | ||||||||||
chmod 640 /var/www/bropkg/config/.env | ||||||||||
cd /var/www/bropkg | ||||||||||
composer update | ||||||||||
sudo chgrp -R apache /var/www/bropkg | ||||||||||
``` | ||||||||||
|
||||||||||
## Set up HTTPD | ||||||||||
Edit /etc/httpd/conf.d/ssl.conf : | ||||||||||
``` | ||||||||||
<VirtualHost _default_:443> | ||||||||||
DocumentRoot "/var/www/bropkg" | ||||||||||
## Edit secrets/.env | ||||||||||
|
||||||||||
... | ||||||||||
`secrets/.env` has a set of variables for passwords and such that PHP will need | ||||||||||
to connect to the database and update the packages list from GitHub. | ||||||||||
|
||||||||||
<Directory /> | ||||||||||
Options FollowSymLinks | ||||||||||
AllowOverride All | ||||||||||
</Directory> | ||||||||||
## Initialize an SSL certificate | ||||||||||
|
||||||||||
</VirtualHost> | ||||||||||
- Edit `cert_setup/ssl-update.sh` and set the `DOMAINS` and `EMAIL` values to | ||||||||||
be sane for your installation. | ||||||||||
- Run the `cert_setup/init-certs.sh` script. This will generate a Let's Encrypt | ||||||||||
certificate, store it in the location that nginx container will use, and add | ||||||||||
a cron task to automatically update it. | ||||||||||
- Edit `docker/nginx-default.conf` and set the hostname in the `ssl_certificate` | ||||||||||
and `ssl_certificate_key` values to match the `DOMAINS` setting from earlier. | ||||||||||
|
||||||||||
``` | ||||||||||
|
||||||||||
Edit /etc/httpd/conf.d/virthost.conf : | ||||||||||
``` | ||||||||||
<VirtualHost *:80> | ||||||||||
DocumentRoot /var/www/bropkg | ||||||||||
## (For development only) Enable the database container | ||||||||||
|
||||||||||
``` | ||||||||||
- Edit `secrets/database.sql` and change the `BRO_USER_PASSWORD` value to match | ||||||||||
what is set in `secrets.env`. | ||||||||||
- Edit `secrets/.env` and change the `DB_HOST` value to `db` to map to the | ||||||||||
internal hostname for the docker database service. | ||||||||||
- Run the following: | ||||||||||
|
||||||||||
Restart httpd process: | ||||||||||
``` | ||||||||||
sudo service httpd restart | ||||||||||
docker compose -f docker-compose-dev.yml build | ||||||||||
docker compose -f docker-compose-dev.yml up -d | ||||||||||
Comment on lines
+35
to
+36
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This only starts the services in the dev config (the database), users also need to start the remaining services.
Suggested change
|
||||||||||
``` | ||||||||||
|
||||||||||
## Initialize database | ||||||||||
## Run `docker-compose` | ||||||||||
|
||||||||||
``` | ||||||||||
mysql_secure_installation # only needed once | ||||||||||
|
||||||||||
Enter current password for root (enter for none): <none> | ||||||||||
OK, successfully used password, moving on... | ||||||||||
Set root password? [Y/n] y | ||||||||||
New password: | ||||||||||
Re-enter new password: | ||||||||||
Password updated successfully! | ||||||||||
Reloading privilege tables.. | ||||||||||
... Success! | ||||||||||
Remove anonymous users? [Y/n] y | ||||||||||
... Success! | ||||||||||
Disallow root login remotely? [Y/n] y | ||||||||||
... Success! | ||||||||||
Remove test database and access to it? [Y/n] y | ||||||||||
- Dropping test database... | ||||||||||
... Success! | ||||||||||
- Removing privileges on test database... | ||||||||||
... Success! | ||||||||||
Reload privilege tables now? [Y/n] y | ||||||||||
... Success! | ||||||||||
Cleaning up... | ||||||||||
All done! | ||||||||||
docker compose build | ||||||||||
docker compose up -d | ||||||||||
``` | ||||||||||
|
||||||||||
Load bropkg user and associated tables. Change BRO_USER_PASSWORD to | ||||||||||
something suitable for the installation. | ||||||||||
|
||||||||||
``` | ||||||||||
mysql -u root -p < secrets/database.sql | ||||||||||
``` | ||||||||||
This will create the images needed for nginx and PHP and start them running. The | ||||||||||
Dockerfiles for these images are stored in the `docker` directory. This will | ||||||||||
also create a Let's Encrypt cert based on the hostname set in the | ||||||||||
|
||||||||||
## Set up cronjob to read Zeek pkg info | ||||||||||
Note: Change [email protected] to the email that should receive emails | ||||||||||
about the output of the bro-pkg-web-updater script. Use of new Python 3 | ||||||||||
versions is also possible, this example just happens to use 3.4. | ||||||||||
## (Optional) Run an update of the packages database | ||||||||||
|
||||||||||
``` | ||||||||||
sudo su | ||||||||||
yum install python34-pip | ||||||||||
pip-3.4 install zkg | ||||||||||
python3 -m pip install https://github.com/zeek/zeek-package-ci/archive/master.zip | ||||||||||
cp cronjob/bro-pkg-web-updater.php /usr/local/sbin/ | ||||||||||
chmod 700 /usr/local/sbin/bro-pkg-web-updater.php | ||||||||||
echo '[email protected] | ||||||||||
# Update to latest zkg | ||||||||||
0 4 * * * root pip-3.4 install --upgrade zkg | ||||||||||
# Read the list of Bro packages and update database at 4am daily | ||||||||||
0 4 * * * root /usr/local/sbin/bro-pkg-web-updater.php' > \ | ||||||||||
/etc/cron.d/bro-pkg-web.cron | ||||||||||
exit | ||||||||||
docker exec -it zeek-pkg-web-php-1 /bin/bash | ||||||||||
/etc/cron.daily/bro-pkg-web-cron.sh | ||||||||||
``` | ||||||||||
Run the script at least once! | ||||||||||
``` | ||||||||||
sudo php /usr/local/sbin/bro-pkg-web-updater.php | ||||||||||
``` | ||||||||||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have no idea how I should set this up for development since LE seems to expect me to own a domain where I can put a challenge file; for all my attempts this part failed. Can you please add a dummy cert at the location expected for the default
DOMAIN
so the dev workflow just works?I suspect all of this works for you since you have some cert in
data/certbot/letsencrypt/live/
; if you do a new checkout somewhere else and follow these instructions you should run into similar issues (do not assume that you have a host set up with a challenge since this is neither documented nor needed for frontend or backend development).