Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Individual docker scripts #1329

Open
wants to merge 4 commits into
base: release_February_2025
Choose a base branch
from

Conversation

nedvedba
Copy link
Collaborator

@nedvedba nedvedba commented Feb 24, 2025

PR Description

This PR serves to introduce two scripts that will create the individual container scripts needed to run the application in production.

Tasks

  • - A description of the PR has been provided, and a diagram included if it is a new feature.
  • - Formatter has been run
  • - CHANGELOG comment has been added
  • - Labels have been assigned to the pr
  • - A reviewer has been added
  • - A user has been assigned to work on the pr
  • - If new feature a unit test has been added

Summary by Sourcery

This pull request introduces scripts to generate individual Docker container scripts for metadata and repository services, and enhances the generate_datafed.sh script to support specifying the database port.

New Features:

  • Introduces two new scripts, generate_metadata_container_scripts.sh and generate_repo_container_scripts.sh, to generate individual container scripts for running the application in production.
  • generate_metadata_container_scripts.sh creates scripts for managing ArangoDB, DataFed Core, DataFed Web, and Nginx containers, including creation, stopping, and removal.
  • generate_repo_container_scripts.sh creates scripts for managing DataFed Repo and GCS containers, including creation, stopping, and removal.

Enhancements:

  • The generate_datafed.sh script now includes an option to specify the database port via the DATAFED_DATABASE_PORT environment variable.

@nedvedba nedvedba added the Component: Scripts Helper scripts or admin scripts label Feb 24, 2025
Copy link

sourcery-ai bot commented Feb 24, 2025

Reviewer's Guide by Sourcery

This PR introduces two new scripts, generate_metadata_container_scripts.sh and generate_repo_container_scripts.sh, to create individual container scripts for running the application in production. It also adds DATAFED_DATABASE_PORT to the datafed.sh config file.

No diagrams generated as the changes look simple and do not need a visual representation.

File-Level Changes

Change Details Files
Added a script to generate container scripts for the metadata server.
  • Added a script that takes in a docker tag and backup directory and generates scripts to run, stop, and remove arango, core, web, and nginx containers.
  • The generated scripts are placed in the DATAFED_INSTALL_PATH/scripts directory.
  • The script includes options for setting the docker tag and backup directory via command line arguments or environment variables.
scripts/generate_metadata_container_scripts.sh
Added a script to generate container scripts for the repository server.
  • Added a script that takes in a docker tag, IP address, repo domain, and optional repo volume mounts and generates scripts to run, stop, and remove repo and gcs containers.
  • The generated scripts are placed in the DATAFED_INSTALL_PATH/scripts directory.
  • The script includes options for setting the docker tag, IP address, repo domain, and repo volume mounts via command line arguments or environment variables.
scripts/generate_repo_container_scripts.sh
Added DATAFED_DATABASE_PORT to the datafed.sh config file.
  • Added logic to check if the DATAFED_DATABASE_PORT environment variable is set, and if not, defaults to 8529.
  • Exported the DATAFED_DATABASE_PORT variable.
scripts/generate_datafed.sh

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!
  • Generate a plan of action for an issue: Comment @sourcery-ai plan on
    an issue to generate a plan of action for it.

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

Copy link

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @nedvedba - I've reviewed your changes - here's some feedback:

Overall Comments:

  • Consider adding a shared function to avoid repeating the argument parsing logic in both scripts.
  • It might be helpful to add some comments to the generated scripts to explain the purpose of each command.
Here's what I looked at during the review
  • 🟡 General issues: 3 issues found
  • 🟢 Security: all looks good
  • 🟢 Testing: all looks good
  • 🟢 Complexity: all looks good
  • 🟢 Documentation: all looks good

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

Comment on lines +106 to +109
IFS=',' read -ra local_REPO_VOLUME_MOUNTS <<< "$local_REPO_VOLUME_MOUNTS"
local_REPO_VOLUME_MOUNTS_EXPANDED=""
for volume_mount in "${local_REPO_VOLUME_MOUNTS[@]}"; do
local_REPO_VOLUME_MOUNTS_EXPANDED+="-v \"$volume_mount\" "
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion (bug_risk): Review the method used for quoting volume mount strings.

Assembling the volume mount flags by embedding literal quotes might lead to issues if a mount value contains spaces or special characters. Consider accumulating these in an array and joining them carefully when generating the docker command.

Suggested implementation:

IFS=',' read -ra local_REPO_VOLUME_MOUNTS <<< "$local_REPO_VOLUME_MOUNTS"
local_REPO_VOLUME_MOUNTS_FLAGS=()
for volume_mount in "${local_REPO_VOLUME_MOUNTS[@]}"; do
  local_REPO_VOLUME_MOUNTS_FLAGS+=( "-v" "$volume_mount" )
done

Any subsequent sections of the script that currently reference "local_REPO_VOLUME_MOUNTS_EXPANDED" should be updated to use the new array "local_REPO_VOLUME_MOUNTS_FLAGS". For example, when constructing the docker command, make sure to expand the array using "${local_REPO_VOLUME_MOUNTS_FLAGS[@]}".

-v "\$DATAFED_INSTALL_PATH/logs:/datafed/logs" \\
-v "\$DATAFED_INSTALL_PATH/keys/datafed-core-key.pub:/opt/datafed/keys/datafed-core-key.pub" \\
-v "\$DATAFED_INSTALL_PATH/keys/datafed-core-key.priv:/opt/datafed/keys/datafed-core-key.priv" \\
-t "datafed/core:$local_DOCKER_TAG"
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion (bug_risk): Consider sanitizing the docker tag for container names.

The web container uses a sanitized variable while core, for example, uses the raw docker tag. If DATAFED_DOCKER_TAG contains unexpected characters, it might cause naming issues.

$local_REPO_VOLUME_MOUNTS_EXPANDED \\
-v "\$DATAFED_INSTALL_PATH/keys/datafed-repo-key.pub:/opt/datafed/keys/datafed-repo-key.pub" \\
-v "\$DATAFED_INSTALL_PATH/keys/datafed-repo-key.priv:/opt/datafed/keys/datafed-repo-key.priv" \\
-t "datafed/repo:$local_DOCKER_TAG"
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: Apply consistent sanitization to the docker tag for container naming.

Containers such as the repository container use the raw docker tag. Consistency in sanitizing the tag (as done for the web container) can help avoid potential issues with invalid container names.

@AronPerez
Copy link
Collaborator

#974

Copy link
Collaborator

@AronPerez AronPerez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Questions about configurability and duplicate code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Component: Scripts Helper scripts or admin scripts
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants