Local Docker Setup for Open Government Portal

Table of Contents
Prerequisites
Recommendations
Docker Networking
Project Prebuild
Database and File Backups
Build Containers
Installing Applications
How to Use
Postgres Version Upgrade

Prerequisites

• WSL2 if using Windows
• Docker
• docker-compose
• Git
  • You will also need your git configuration to be accessible to your user, located by default at ~/.gitconfig. This will be attached to the Docker container so that the build scripts can pull the repositories.
  • If using WSL2, you will need to have this config set up inside of WSL2 for your WSL2 user.

Recommendations

It is highly recommended to use the OG CLI tool to manage docker-og projects.

The OG CLI tool will give you an easier command-line interface to view, manage, and run Docker projects that use docker-og.

It will also allow you simpler git tooling and debugging.

Networking

The docker-og scripts use port and IP octet renting from a pool of pre-defined ports and octets.

Ports: 57000 - 57999 IP Addresses: 172.25.1.0/24 - 172.25.254.0/24

You can see the project's important variables in the .env file after running the pre-build script.

Prebuild

Note: If using the og-cli tool, you do not need to run the pre-build.sh script.

Otherwise:

• Run the pre build script with a Project ID: ./pre-build.sh example

Backups

Global

If you do not have any backups in the root backup directory, the installation scripts will look for the backup files in /opt/tbs/docker/backup

1. Create the directory sudo mkdir -p /opt/tbs/docker/backup
2. Set permissions sudo chmod 777 -R /opt/tbs/docker/backup
3. /opt/tbs/docker/backup directory. In it, place the following files:
   • For Drupal:
     1. drupal_db.pgdump (Required) <- the database backup for the Drupal site.
     2. drupal_files.tgz (Required) <- the compressed folder of the public files from the Drupal site.
   • For CKAN:
     1. ckan_portal_db.pgdump (Optional) <- the database backup for the CKAN Portal app.
     2. ckan_portal_ds_db.pgdump (Optional) <- the database backup for the CKAN Portal Datastore.
     3. ckan_registry_db.pgdump (Optional) <- the database backup for the CKAN Registry app.
     4. ckan_registry_ds_db.pgdump (Optional) <- the database backup for the CKAN Registry Datastore.
   • For Solr:
     1. inventory.csv (Optional) <- Open Data Inventory data set csv file.

Per Project

To override the use of the global backups during the installation scripts, place any of the above files into the root backup directory.

Build

1. Build the container: docker-compose build
   • The initial build will take a long time.
   • If you are rebuilding and receive errors such as max depth exceeded, you may need to destroy all of the docker images (docker image prune -a) and then run the above build command. Please note that this will also destroy any other docker images you have on your machine.
   • If your build fails with errors regarding failure of resolving domains, restart your docker service (sudo service docker restart).
2. Bring up the app and detach the shell: docker-compose up -d to make sure that all the containers can start correctly.
   • The initial up may take a long time.
   • To stop all the containers: docker-compose down

Installation

Databases

Though there is an initialization script to create the databases on the initial up of the postgres container, this script only runs once. After you have built the containers once, this script will no longer run. To fix any issues with missing databases, users, or password:

1. Bring up the Drupal or CKAN docker container: docker-compose up -d <drupal or ckan>
2. Run the install script in the docker container: docker-compose exec <drupal or ckan> bash and then ./install.sh
   1. Select Databases (fixes missing databases, privileges, and users)
   2. Select Test Databases (clones existing databases into empty ones) (Optional for destructive, unit tests)

Drupal (D9)

1. Bring up the Drupal docker container: docker-compose up -d drupal
2. Run the install script in the docker container: docker-compose exec drupal bash and then ./install.sh
   1. Select Drupal
   2. Select what you want to install for Drupal:
      • Database: will destroy the current og_drupal_local database and import a fresh one from backup/drupal_db.pgdump
      • Repositories: will destroy all files inside of the drupal directory and pull all required repositories related to Drupal.
      • Local Files: will destroy all Drupal local files and extract the directory from backup/drupal_files.tgz.
      • Set File Permissions (also creates missing directories): will download default settings files, copy drupal-local-settings.php create the private files directory, and set the correct file and directory ownerships and permissions.
      • Create Local User: will create a local admin user for Drupal.
      • All: will execute all of the above, use this for first time install or if you wish to re-install everything.
      • Exit: will exit the installation script.
3. Login here: https://open.local/en/user/login
   1. Username: admin.local
   2. Password: 12345678

CKAN (2.9)

Registry

1. Bring up the CKAN Registry docker container: docker-compose up -d ckan
2. Run the install script in the docker container: docker-compose exec ckan bash and then ./install.sh
   1. Select CKAN (registry)
   2. Select what you want to install for CKAN Registry:
      • Registry Database: will destroy the current og_ckan_registry_local database and import a fresh one from backup/ckan_registry_db.pgdump
        • Importing a database is optional, if the file does not exist, this step will be skipped automatically.
        • The database for CKAN Registry is not too large, however it has a lot of tables so importing the database will take a long time.
        • You may recieve warnings during the pg_restore: out of shared memory, this can be ignored, the import will just take longer.
      • Registry Datastore Database: will destroy the current og_ckan_registry_ds_local database and import a fresh one from backup/ckan_registry_ds_db.pgdump
        • Importing a database is optional, if the file does not exist, this step will be skipped automatically.
        • The database for CKAN Registry is not too large, however it has a lot of tables so importing the database will take a long time.
        • You may recieve warnings during the pg_restore: out of shared memory, this can be ignored, the import will just take longer.
      • Repositories: will destroy all files inside of the ckan/registry directory, and pull & install all required repositories related to CKAN and install them into the Python environment (along with their requirements).
      • Downlod Wet-Boew Files: will destroy all files inside of the ckan/static_files direcotry, and download and extract:
        • wet-boew-cdn into ckan/static_files/wet-boew
        • themes-cdn into ckan/static_files/GCWeb
      • Set File Permissions: will set the correct file and directory ownerships and permissions.
        • This will also set the correct database permissions for CKAN.
      • Create Local User: will create a local admin user for CKAN Registry.
      • Import Organizations: will dump the most recent Organizations from the ckanapi and import them into the database.
      • Import Datasets: will download the most recent dataset file and import them into the datase.
        • This will import all 30k+ resources. This may take a long time.
      • All: will execute all of the above, use this for first time install or if you wish to re-install everything.
      • Exit: will exit the installation script.

Portal

1. Bring up the CKAN Portal docker container: docker-compose up -d ckanapi
2. Run the install script in the docker container: docker-compose exec ckanapi bash and then ./install.sh
   1. Select CKAN (portal)
   2. Select what you want to install for CKAN Portal:
      • Portal Database: will destroy the current og_ckan_portal_local database and import a fresh one from backup/ckan_portal_db.pgdump
        • Importing a database is optional, if the file does not exist, this step will be skipped automatically.
        • The database for CKAN is large, so importing the database will take a long time.
        • You may recieve warnings during the pg_restore: out of shared memory, this can be ignored, the import will just take longer.
      • Portal Datastore Database: will destroy the current og_ckan_portal_ds_local database and import a fresh one from backup/ckan_portal_ds_db.pgdump
        • Importing a database is optional, if the file does not exist, this step will be skipped automatically.
        • The database for CKAN is large, so importing the database will take a long time.
        • You may recieve warnings during the pg_restore: out of shared memory, this can be ignored, the import will just take longer.
      • Repositories: will destroy all files inside of the ckan/portal directory, and pull & install all required repositories related to CKAN and install them into the Python environment (along with their requirements).
      • Downlod Wet-Boew Files: will destroy all files inside of the ckan/static_files direcotry, and download and extract:
        • wet-boew-cdn into ckan/static_files/wet-boew
        • themes-cdn into ckan/static_files/GCWeb
      • Set File Permissions: will set the correct file and directory ownerships and permissions.
        • This will also set the correct database permissions for CKAN.
      • Create Local User: will create a local admin user for CKAN Portal.
      • Import Organizations: will dump the most recent Organizations from the ckanapi and import them into the database.
      • Import Datasets: will download the most recent dataset file and import them into the datase.
        • This will import all 30k+ resources. This may take a long time.
      • All: will execute all of the above, use this for first time install or if you wish to re-install everything.
      • Exit: will exit the installation script.

Django (Search App v1 - DEPRECATED)

1. Bring up the Django docker container: docker-compose up -d django
2. Run the install script in the docker container: docker-compose exec django ./install.sh
   1. Select Django
   2. Select what you want to install for Django:
      • OGC Django Search App (version 1): will destory the current Python virtual environment (if it exists) and install a fresh one, pull the ogc_search repository and copy search-settings.py to the environment.
        • This will also download the required CKAN yaml and json files.
      • Static Files: will download the required files, including the GCWeb theme and GCWeb Static release from the CDTS repository.
        • It will also copy files to the correct locations for serving them correctly.
      • Set File Permissions: will set the correct file and directory ownerships and permissions.
      • All: will execute all of the above, use this for first time install or if you wish to re-install everything.
      • Exit: will exit the installation script.

Usage

Drupal (D9)

1. Bring up the Drupal docker container: docker-compose up -d drupal
2. Open a browser into: http://open.local:<project port>
3. Login here: http://open.local:<project port>/en/user/login
   1. Username: admin.local
   2. Password: 12345678

Multisites are also available at:

• http://blog.open.local:<project port>
• http://guides.open.local:<project port>

Solr (solr:8)

The Solr container will automatically be brought up with the CKAN and Drupal containers.

1. Bring up the Solr docker container: docker-compose up -d solr
2. Open a browser into: http://solr.open.local:<project port>

Postgres (postgres:13.14)

The Postgres container will automatically be brought up with the CKAN and Drupal containers.

1. Bring up the Postgres docker container: docker-compose up -d postgres
2. You can use the pgAdmin software to connect to the Postgres container and query the databases.
3. For the connection info:
   • Hostname/address: 127.0.0.1
   • Port: <project database port>
   • Username: homestead
   • Maintenance database: postgres

CKAN (2.9)

Registry

1. Bring up the CKAN Registry docker container: docker-compose up -d ckan
2. Open a browser into: http://registry.open.local:<project port>
   1. Login here: http://registry.open.local:<project port>/en/user/login
      1. Normal User:
         1. Username: user_local
         2. Password: 12345678
      2. Sys Admin User:
         1. Username: admin_local
         2. Password: 12345678
3. Build the indices (Optional):
   1. Inventory: paster --plugin=ckanext-canada inventory rebuild --lenient -c $REGISTRY_CONFIG -f /srv/app/backup/inventory.csv

Portal

1. Bring up the CKAN Portal docker container: docker-compose up -d ckanapi
2. Open a browser into: http://open.local:<project port>/data/en/dataset

Django (Search App v1 - DEPRECATED)

1. Bring up the Django docker container: docker-compose up -d django
2. Open a browser into: http://search.open.local:<project port>

Upgrading Postgres

If you need to upgrade major versions of PostgreSQL, you can dump all of the databases via the install script, and restore them all after the PostgreSQL major version upgrade.

1. While still on your current version of PostgreSQL (viewable in the Dockerfile at docker/postgres/Dockefile).

   FROM postgres:9.6
   

2. Bring up the Drupal or CKAN docker container: docker-compose up -d <drupal or ckan>

3. Run the install script in the docker container: docker-compose exec <drupal or ckan> bash and then ./install.sh

   1. Select Postgres Upgrade (dump and load existing databases for psql version upgrade)
   2. Select Dump all existing databases (do PRIOR to Postgres version upgrade) (Do this prior to upgrading the Postgres major version)

4. Bring down all the containers: docker-compose down

5. Modify the Dockerfile docker/postgres/Dockefile:

   FROM postgres:13.14
   

6. Rebuild the postgres container: docker-compose build postgres

7. Run the install script in the docker container: docker-compose exec <drupal or ckan> bash and then ./install.sh

   1. Select Postgres Upgrade (dump and load existing databases for psql version upgrade)
   2. Select Restore all databases (do AFTER Postgres version upgrade) (Do this after upgrading the Postgres major version)