Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create Docker installation files #7

Open
wants to merge 17 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions .github/workflows/pre-commit.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
name: pre-commit

on:
pull_request:
push:
branches: [master]

jobs:
pre-commit:
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
- uses: pre-commit/[email protected]
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
data
31 changes: 31 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# See https://pre-commit.com for more information
# See https://pre-commit.com/hooks.html for more hooks
#
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v5.0.0
hooks:
- id: check-yaml
- id: end-of-file-fixer
- id: trailing-whitespace

- repo: https://github.com/maxwinterstein/shfmt-py
rev: v3.7.0.1
hooks:
- id: shfmt
args: ["-w", "-i", "4", "-ci"]

- repo: https://github.com/crate-ci/typos
rev: v1.29.5
hooks:
- id: typos

- repo: https://github.com/hadolint/hadolint
rev: v2.12.0
hooks:
- id: hadolint-docker

- repo: https://github.com/koalaman/shellcheck-precommit
rev: v0.10.0
hooks:
- id: shellcheck
112 changes: 28 additions & 84 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,104 +8,48 @@ git clone [email protected]:zeek/zeek-pkg-web.git
cd zeek-pkg-web
```

## Copy files/directories
```
sudo cp -a bropkg /var/www
### Modify salt, database password, and CILogon client id/key in
### secrets/.env to something suitable for your installation
cp secrets/.env /var/www/bropkg/config/
chmod 640 /var/www/bropkg/config/.env
cd /var/www/bropkg
composer update
sudo chgrp -R apache /var/www/bropkg
```

## Set up HTTPD
Edit /etc/httpd/conf.d/ssl.conf :
```
<VirtualHost _default_:443>
DocumentRoot "/var/www/bropkg"
## Edit secrets/.env

...
`secrets/.env` has a set of variables for passwords and such that PHP will need
to connect to the database and update the packages list from GitHub.

<Directory />
Options FollowSymLinks
AllowOverride All
</Directory>
## Initialize an SSL certificate

</VirtualHost>
- Edit `cert_setup/ssl-update.sh` and set the `DOMAINS` and `EMAIL` values to
be sane for your installation.
- Run the `cert_setup/init-certs.sh` script. This will generate a Let's Encrypt
certificate, store it in the location that nginx container will use, and add
a cron task to automatically update it.
Comment on lines +20 to +22
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have no idea how I should set this up for development since LE seems to expect me to own a domain where I can put a challenge file; for all my attempts this part failed. Can you please add a dummy cert at the location expected for the default DOMAIN so the dev workflow just works?

I suspect all of this works for you since you have some cert in data/certbot/letsencrypt/live/; if you do a new checkout somewhere else and follow these instructions you should run into similar issues (do not assume that you have a host set up with a challenge since this is neither documented nor needed for frontend or backend development).

- Edit `docker/nginx-default.conf` and set the hostname in the `ssl_certificate`
and `ssl_certificate_key` values to match the `DOMAINS` setting from earlier.

```

Edit /etc/httpd/conf.d/virthost.conf :
```
<VirtualHost *:80>
DocumentRoot /var/www/bropkg
## (For development only) Enable the database container

```
- Edit `secrets/database.sql` and change the `BRO_USER_PASSWORD` value to match
what is set in `secrets.env`.
- Edit `secrets/.env` and change the `DB_HOST` value to `db` to map to the
internal hostname for the docker database service.
- Run the following:

Restart httpd process:
```
sudo service httpd restart
docker compose -f docker-compose-dev.yml build
docker compose -f docker-compose-dev.yml up -d
Comment on lines +35 to +36
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This only starts the services in the dev config (the database), users also need to start the remaining services.

Suggested change
docker compose -f docker-compose-dev.yml build
docker compose -f docker-compose-dev.yml up -d
docker compose -f docker-compose-dev.yml -f docker-compose.yml build
docker compose -f docker-compose-dev.yml -f docker-compose.yml up -d

```

## Initialize database
## Run `docker-compose`

```
mysql_secure_installation # only needed once

Enter current password for root (enter for none): <none>
OK, successfully used password, moving on...
Set root password? [Y/n] y
New password:
Re-enter new password:
Password updated successfully!
Reloading privilege tables..
... Success!
Remove anonymous users? [Y/n] y
... Success!
Disallow root login remotely? [Y/n] y
... Success!
Remove test database and access to it? [Y/n] y
- Dropping test database...
... Success!
- Removing privileges on test database...
... Success!
Reload privilege tables now? [Y/n] y
... Success!
Cleaning up...
All done!
docker compose build
docker compose up -d
```

Load bropkg user and associated tables. Change BRO_USER_PASSWORD to
something suitable for the installation.

```
mysql -u root -p < secrets/database.sql
```
This will create the images needed for nginx and PHP and start them running. The
Dockerfiles for these images are stored in the `docker` directory. This will
also create a Let's Encrypt cert based on the hostname set in the

## Set up cronjob to read Zeek pkg info
Note: Change [email protected] to the email that should receive emails
about the output of the bro-pkg-web-updater script. Use of new Python 3
versions is also possible, this example just happens to use 3.4.
## (Optional) Run an update of the packages database

```
sudo su
yum install python34-pip
pip-3.4 install zkg
python3 -m pip install https://github.com/zeek/zeek-package-ci/archive/master.zip
cp cronjob/bro-pkg-web-updater.php /usr/local/sbin/
chmod 700 /usr/local/sbin/bro-pkg-web-updater.php
echo '[email protected]
# Update to latest zkg
0 4 * * * root pip-3.4 install --upgrade zkg
# Read the list of Bro packages and update database at 4am daily
0 4 * * * root /usr/local/sbin/bro-pkg-web-updater.php' > \
/etc/cron.d/bro-pkg-web.cron
exit
docker exec -it zeek-pkg-web-php-1 /bin/bash
/etc/cron.daily/bro-pkg-web-cron.sh
```
Run the script at least once!
```
sudo php /usr/local/sbin/bro-pkg-web-updater.php
```

28 changes: 14 additions & 14 deletions bropkg/bin/cake
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
#!/usr/bin/env sh
# shellcheck disable=all

################################################################################
#
# Cake is a shell script for invoking CakePHP shell commands
Expand All @@ -19,25 +21,23 @@

# Canonicalize by following every symlink of the given name recursively
canonicalize() {
NAME="$1"
if [ -f "$NAME" ]
then
DIR=$(dirname -- "$NAME")
NAME=$(cd -P "$DIR" > /dev/null && pwd -P)/$(basename -- "$NAME")
fi
while [ -h "$NAME" ]; do
DIR=$(dirname -- "$NAME")
SYM=$(readlink "$NAME")
NAME=$(cd "$DIR" > /dev/null && cd $(dirname -- "$SYM") > /dev/null && pwd)/$(basename -- "$SYM")
done
echo "$NAME"
NAME="$1"
if [ -f "$NAME" ]; then
DIR=$(dirname -- "$NAME")
NAME=$(cd -P "$DIR" >/dev/null && pwd -P)/$(basename -- "$NAME")
fi
while [ -h "$NAME" ]; do
DIR=$(dirname -- "$NAME")
SYM=$(readlink "$NAME")
NAME=$(cd "$DIR" >/dev/null && cd $(dirname -- "$SYM") >/dev/null && pwd)/$(basename -- "$SYM")
done
echo "$NAME"
}

CONSOLE=$(dirname -- "$(canonicalize "$0")")
APP=$(dirname "$CONSOLE")

if [ $(basename $0) != 'cake' ]
then
if [ $(basename $0) != 'cake' ]; then
exec php "$CONSOLE"/cake.php $(basename $0) "$@"
else
exec php "$CONSOLE"/cake.php "$@"
Expand Down
20 changes: 13 additions & 7 deletions bropkg/composer.json
Original file line number Diff line number Diff line change
Expand Up @@ -5,16 +5,17 @@
"type": "project",
"license": "MIT",
"require": {
"php": ">=5.6",
"cakephp/cakephp": "3.5.*",
"mobiledetect/mobiledetectlib": "2.*",
"cakephp/migrations": "~1.0",
"php": ">=7.1",
"cakephp/cakephp": "3.7.*",
"mobiledetect/mobiledetectlib": "^2.8",
"cakephp/migrations": "~2.0",
"cakephp/plugin-installer": "~1.0",
"josegonzalez/dotenv": "2.*",
"tanuck/cakephp-markdown": "^0.1.0",
"friendsofcake/search": "^4.2",
"friendsofcake/search": "^5.0",
"holt59/cakephp3-bootstrap-helpers": "dev-master",
"gregwar/rst": "^1.0"
"gregwar/rst": "^1.0",
"psr/http-message": "^1.0"
},
"require-dev": {
"psy/psysh": "@stable",
Expand Down Expand Up @@ -51,5 +52,10 @@
"cs-fix": "phpcbf --colors --standard=vendor/cakephp/cakephp-codesniffer/CakePHP ./src ./tests",
"test": "phpunit --colors=always"
},
"prefer-stable": true
"prefer-stable": true,
"config": {
"allow-plugins": {
"cakephp/plugin-installer": true
}
}
}
46 changes: 2 additions & 44 deletions bropkg/config/bootstrap.php
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@
use Cake\Http\ServerRequest;
use Cake\Log\Log;
use Cake\Mailer\Email;
use Cake\Mailer\TransportFactory;
use Cake\ORM\TableRegistry;
use Cake\Utility\Inflector;
use Cake\Utility\Security;
Expand Down Expand Up @@ -147,7 +148,7 @@

Cache::setConfig(Configure::consume('Cache'));
ConnectionManager::setConfig(Configure::consume('Datasources'));
Email::setConfigTransport(Configure::consume('EmailTransport'));
TransportFactory::setConfig(Configure::consume('EmailTransport'));
Email::setConfig(Configure::consume('Email'));
Log::setConfig(Configure::consume('Log'));
Security::setSalt(Configure::consume('Security.salt'));
Expand Down Expand Up @@ -189,46 +190,3 @@
->useImmutable();
Type::build('timestamp')
->useImmutable();

/*
* Custom Inflector rules, can be set to correctly pluralize or singularize
* table, model, controller names or whatever other string is passed to the
* inflection functions.
*/
//Inflector::rules('plural', ['/^(inflect)or$/i' => '\1ables']);
//Inflector::rules('irregular', ['red' => 'redlings']);
//Inflector::rules('uninflected', ['dontinflectme']);
//Inflector::rules('transliteration', ['/å/' => 'aa']);

/*
* Plugins need to be loaded manually, you can either load them one by one or all of them in a single call
* Uncomment one of the lines below, as you need. make sure you read the documentation on Plugin to use more
* advanced ways of loading plugins
*
* Plugin::loadAll(); // Loads all plugins at once
* Plugin::load('Migrations'); //Loads a single plugin named Migrations
*
*/

/*
* Only try to load DebugKit in development mode
* Debug Kit should not be installed on a production system
*/
if (Configure::read('debug')) {
Plugin::load('DebugKit', ['bootstrap' => true]);
}

/*
* https://github.com/tanuck/cakephp-markdown
*/
Plugin::load('Tanuck/Markdown');

/*
* https://github.com/FriendsOfCake/search
*/
Plugin::load('Search');

/*
* https://github.com/Holt59/cakephp3-bootstrap-helpers
*/
Plugin::load('Bootstrap');
8 changes: 0 additions & 8 deletions bropkg/config/bootstrap_cli.php
Original file line number Diff line number Diff line change
Expand Up @@ -28,11 +28,3 @@
// Set logs to different files so they don't have permission conflicts.
Configure::write('Log.debug.file', 'cli-debug');
Configure::write('Log.error.file', 'cli-error');

try {
Plugin::load('Bake');
} catch (MissingPluginException $e) {
// Do not halt if the plugin is missing
}

Plugin::load('Migrations');
7 changes: 0 additions & 7 deletions bropkg/config/routes.php
Original file line number Diff line number Diff line change
Expand Up @@ -85,10 +85,3 @@ function ($routes) {
*/
$routes->fallbacks(DashedRoute::class);
});

/**
* Load all plugin routes. See the Plugin documentation on
* how to customize the loading of plugin routes.
*/
Plugin::routes();

35 changes: 35 additions & 0 deletions bropkg/src/Application.php
Original file line number Diff line number Diff line change
Expand Up @@ -49,4 +49,39 @@ public function middleware($middlewareQueue)

return $middlewareQueue;
}

public function bootstrap()
{
parent::bootstrap();

if (Configure::read('debug')) {
$this->addPlugin('DebugKit');
}

/*
* https://github.com/tanuck/cakephp-markdown
*/
$this->addPlugin('Tanuck/Markdown');

/*
* https://github.com/FriendsOfCake/search
*/
$this->addPlugin('Search');

/*
* https://github.com/Holt59/cakephp3-bootstrap-helpers
*/
$this->addPlugin('Bootstrap');

if ( PHP_SAPI === 'cli' )
{
try {
$this->addPlugin('Bake');
} catch (MissingPluginException $e) {
// Do not halt if the plugin is missing
}

$this->addPlugin('Migrations');
}
}
}
2 changes: 1 addition & 1 deletion bropkg/src/Controller/PackagesController.php
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ public function index()
*/
public function view($id = null)
{
// If no pakcage specified, simply list all packages.
// If no package specified, simply list all packages.
if (is_null($id)) {
return $this->redirect([
'controller' => 'Packages',
Expand Down
Loading
Loading