Skip to content

Commit

Permalink
Merge pull request #148 from nmfs-opensci/eeholmes-patch-1
Browse files Browse the repository at this point in the history 
Update install-desktop.sh
  • Loading branch information
@eeholmes
eeholmes authored Nov 6, 2024
2 parents afad8e0 + f004457 commit 732c273
Show file tree
Hide file tree
Showing 15 changed files with 235 additions and 149 deletions.
3 changes: 0 additions & 3 deletions appendix
View file
Original file line number Diff line number Diff line change
Expand Up @@ -51,9 +51,6 @@ RUN chown -R root:staff /pyrocket_scripts && \
# Convert NB_USER to ENV (from ARG) so that it passes to the child dockerfile
ENV NB_USER=${NB_USER}

# Copy the child repo files into childimage so they are available to scripts
ONBUILD COPY --chown=${NB_USER}:${NB_USER} . ${REPO_DIR}/childimage

# Revert to default user and home as pwd
USER ${NB_USER}
WORKDIR ${HOME}
27 changes: 21 additions & 6 deletions book/configuration_files.qmd
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,9 @@ The `install-conda-packages.sh` script will install conda packages to the conda
Here is the code for your Docker file. You can name the conda package file to something other than `environment.yml`. Make sure your file has `name:`. The name is arbitrary. It is ignored but required for the script.

```
COPY environment.yml environment.yml
RUN /pyrocket_scripts/install-conda-packages.sh environment.yml
RUN rm environment.yml
```

This is a standard format. You can add version pinning.
Expand All @@ -27,16 +29,21 @@ dependencies:
Instead of a list of conda packages (typically called environment.yml), you can use a conda lock file instead.

Here is the code for your Docker file. You can name your conda lock file something other than `conda-lock.yml`.

```
COPY conda-lock.yml conda-lock.yml
RUN /pyrocket_scripts/install-conda-packages.sh conda-lock.yml
RUN rm conda-lock.yml
```

## requirments.txt
## requirements.txt

The `install-pip-packages.sh` script will install packages using `pip`. Here is the code for your Docker file. You can name your pip package file something other than `requirements.txt`.

```
COPY requirements.txt requirements.txt
RUN /pyrocket_scripts/install-pip-packages.sh requirements.txt
RUN rm requirements.txt
```

requirements.txt
Expand All @@ -53,7 +60,9 @@ The `install-r-packages.sh` script will run the supplied R script which you can
Here is the code for your Docker file. You can name the R script file to something other than `install.R`. Make sure your file is an R script.

```
COPY install.R install.R
RUN /pyrocket_scripts/install-r-packages.sh install.R
RUN rm install.R
```

install.R example
Expand Down Expand Up @@ -85,7 +94,9 @@ The `install-apt-packages.sh` script will install packages with `apt-get`. Here

```
USER root
COPY apt.txt apt.txt
RUN /pyrocket_scripts/install-apt-packages.sh apt.txt
RUN rm apt.txt
USER ${NB_USER}
```

Expand All @@ -100,10 +111,12 @@ cmocean

## postBuild

The `run-postbuild.sh` script can be run as root or jovyan (`${NB_USER}`). This script does not accept a file name. You need to name your postBuild script `postBuild` and put at the base level with your Docker file.
The `run-postbuild.sh` script can be run as root or jovyan (`${NB_USER}`). The script has some extra code to remove leftover files after installing Python extensions.

```
RUN /pyrocket_scripts/run-postbuild.sh
COPY postBuild postBuild
RUN /pyrocket_scripts/run-postbuild.sh postBuild
RUN rm postBuild
```

postBuild
Expand All @@ -116,12 +129,14 @@ set -e

## start

The `start` bash code is run when the image starts. py-rocker-base has a start script at `${REPO_DIR}/start` which loads the Desktop applications. If you change that start file (by copying your start file onto that location), then the Desktop apps will not be loaded properly. Instead, the `setup-start.sh` will add your start file to the end of `${REPO_DIR}/start` so that the Desktop is still setup properly.
The `start` bash code is run when the image starts. py-rocker-base has a start script at `${REPO_DIR}/start` which loads the Desktop applications. If you change that start file (by copying your start file onto that location), then the Desktop apps will not be loaded properly. Instead, the `setup-start.sh` will add your start file to a directory `${REPO_DIR}/childstarts` and will run all those scripts after `${REPO_DIR}/start`.

The `setup-start.sh` script does not accept a file name. You need to name your start script `start` and put at the base level with your Docker file.
The `setup-start.sh` script will move the file you provide into `${REPO_DIR}/childstarts`. As usual you can name your script something other than `start`.

```
RUN /pyrocket_scripts/setup-start.sh
COPY start start
RUN /pyrocket_scripts/setup-start.sh start
RUN rm start
```

## Desktop applications
Expand Down
14 changes: 12 additions & 2 deletions book/customizing.qmd
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,16 @@ py-rocket-base is designed to be used in the FROM line of a Dockerfile similar t

**Calling the scripts**

The format for calling the pyrocket and rocker scripts is the following.
The format for calling the pyrocket and rocker scripts is the following.

pyrocket scripts take files (or a path to a directory with Desktop files) as arguments. The `COPY` command is needed to copy the file into the Docker build context where it can be used in `RUN` commands. Without this you will get a "file not found" error. Removing the file after you are done with it will clean up your image files.
```
COPY environment.yml environment.yml
RUN /pyrocket_scripts/install-conda-packages.sh environment.yml
RUN rm environment.yml
```

Note that PATH must be given for the subshell since rocker installation scripts will fail with conda on the path.
Rocker scripts do not take arguments. Note that PATH must be given since rocker installation scripts will fail with conda on the path. The path specification is only within the specific RUN context.
```
USER root
RUN PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin && \
Expand Down Expand Up @@ -54,7 +58,9 @@ Dockerfile
```
FROM ghcr.io/nmfs-opensci/py-rocket-base:latest
COPY environment.yml environment.yml
RUN /pyrocket_scripts/install-conda-packages.sh environment.yml
RUN rm environment.yml
```

environment.yml
Expand All @@ -81,7 +87,9 @@ Dockerfile
```
FROM ghcr.io/nmfs-opensci/py-rocket-base:latest
COPY install.R install.R
RUN /pyrocket_scripts/install-r-packages.sh install.R
RUN rm install.R
```

install.R
Expand All @@ -108,7 +116,9 @@ Dockerfile
FROM ghcr.io/nmfs-opensci/py-rocket-base:latest
USER root
COPY apt.txt apt.txt
RUN /pyrocket_scripts/install-apt-packages.sh apt.txt
RUN rm apt.txt
USER ${NB_USER}
```

Expand Down
6 changes: 4 additions & 2 deletions book/desktop.qmd
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,13 @@ py-rocket-base puts these `.desktop` files in `/usr/share/Desktop`. Typically th

## Adding an application in your child docker image

Use the pyrocket script, `install-desktop.sh` to set up the desktop and move your files to the proper location. Here is the code for your Docker file. This script must be run as root. It does not take any arguments. Instead you include your desktop files in a directory called `Desktop` at the base level with your Docker file.
Use the pyrocket script `install-desktop.sh` to set up the desktop and move your files to the proper location. Provide a path to a directory with your Desktop files as the argument to the script. Here is the code for your Docker file. This script must be run as root.

```
USER root
RUN /pyrocket_scripts/install-desktop.sh
COPY ./Desktop /tmp/Desktop
RUN /pyrocket_scripts/install-desktop.sh /tmp/Desktop
RUN rm -rf /tmp/Desktop
USER ${NB_USER}
```

Expand Down
25 changes: 15 additions & 10 deletions book/developers.qmd
View file
Original file line number Diff line number Diff line change
Expand Up @@ -166,12 +166,9 @@ RUN chown -R root:staff /pyrocket_scripts && \ # <8>
# Convert NB_USER to ENV (from ARG) so that it passes to the child dockerfile
ENV NB_USER=${NB_USER} # <9>

# Copy the child repo files into childimage so they are available to scripts
ONBUILD COPY --chown=${NB_USER}:${NB_USER} . ${REPO_DIR}/childimage # <10>

# Revert to default user and home as pwd
USER ${NB_USER} # <11>
WORKDIR ${HOME} # <11>
USER ${NB_USER} # <10>
WORKDIR ${HOME} # <10>
```
1. Some commands need to be run as root, such as installing linux packages with `apt-get`
2. Set variables. CONDA_ENV is useful for child builds
Expand All @@ -182,7 +179,6 @@ WORKDIR ${HOME} # <11>
7. `book` and `docs` are the documentation files and are not needed in the image.
8. Copy the pyrocket helper scripts to the `/pyrocket_scripts` directory and set to executable.
9. The `NB_USER` environmental variable is not exported by repo2docker (it is an argument confined to the parent build) but is very useful for child builds. So it is converted to an environmental variable.
10. Copy the child build context (files with the Docker file) into `${REPO_DIR}`. Make sure that jovyan owns the directory. Note, jovyan owns `${REPO_DIR}` (this is set by repo2docker).
11. The parent docker build completes by setting the user to jovyan and the working directory to `${HOME}`. Within a JupyterHub deployment, `${HOME}` will often be re-mapped to the user persistent memory so it is important not to write anything that needs to be persistent to `${HOME}`, for example configuration. You can do this in the `start` script since that runs after the user directory is mapped or you can put configuration files in some place other than `${HOME}`.

## rocker.sh
Expand Down Expand Up @@ -263,13 +259,22 @@ set -euo pipefail
source ${REPO_DIR}/env.txt # <1>
# End - Set any environment variables here # <1>
# Run child start in a subshell to contain its environment
[ -f ${REPO_DIR}/childimage/start ] && ( source ${REPO_DIR}/childimage/start ) # <2>
# Run child start scripts in a subshell to contain its environment
# ${REPO_DIR}/childstart/ is created by setup-start.sh
if [ -d "${REPO_DIR}/childstart/" ]; then # <2>
for script in ${REPO_DIR}/childstart/*; do # <2>
if [ -f "$script" ]; then # <2>
echo "Sourcing script: $script" # <2>
source "$script" || { # <2>
echo "Error: Failed to source $script. Moving on to the next script." # <2>
} # <2>
fi # <2>
done # <2>
fi # <2>
exec "$@"
```
1. In a Docker file so no way to dynamically set environmental variables, so the `env.txt` file with the `export <var>=<value>` are source at start up.
2. Run any child start script in a subshell. Run in a subshell to contain any `set` statements or similar.
2. Run any child start script in a subshell. Run in a subshell to contain any `set` statements or similar. start scripts are moved into `childstarts` by the `setup-start.sh` pyrocket script.

## desktop.sh

Expand Down
38 changes: 26 additions & 12 deletions docs/configuration_files.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -176,7 +176,7 @@ <h2 id="toc-title">Table of contents</h2>
<ul>
<li><a href="#environment.yml" id="toc-environment.yml" class="nav-link active" data-scroll-target="#environment.yml"><span class="header-section-number">2.1</span> environment.yml</a></li>
<li><a href="#conda-lock.yml" id="toc-conda-lock.yml" class="nav-link" data-scroll-target="#conda-lock.yml"><span class="header-section-number">2.2</span> conda-lock.yml</a></li>
<li><a href="#requirments.txt" id="toc-requirments.txt" class="nav-link" data-scroll-target="#requirments.txt"><span class="header-section-number">2.3</span> requirments.txt</a></li>
<li><a href="#requirements.txt" id="toc-requirements.txt" class="nav-link" data-scroll-target="#requirements.txt"><span class="header-section-number">2.3</span> requirements.txt</a></li>
<li><a href="#install.r" id="toc-install.r" class="nav-link" data-scroll-target="#install.r"><span class="header-section-number">2.4</span> install.R</a>
<ul class="collapse">
<li><a href="#add-r-geospatial-packages" id="toc-add-r-geospatial-packages" class="nav-link" data-scroll-target="#add-r-geospatial-packages"><span class="header-section-number">2.4.1</span> Add R geospatial packages</a></li>
Expand Down Expand Up @@ -214,7 +214,9 @@ <h1 class="title"><span class="chapter-number">2</span>&nbsp; <span class="chapt
<h2 data-number="2.1" class="anchored" data-anchor-id="environment.yml"><span class="header-section-number">2.1</span> environment.yml</h2>
<p>The <code>install-conda-packages.sh</code> script will install conda packages to the conda notebook environment, the user environment in the py-rocket-base image (same as for pangeo and repo2docker images).</p>
<p>Here is the code for your Docker file. You can name the conda package file to something other than <code>environment.yml</code>. Make sure your file has <code>name:</code>. The name is arbitrary. It is ignored but required for the script.</p>
<pre><code>RUN /pyrocket_scripts/install-conda-packages.sh environment.yml</code></pre>
<pre><code>COPY environment.yml environment.yml
RUN /pyrocket_scripts/install-conda-packages.sh environment.yml
RUN rm environment.yml</code></pre>
<p>This is a standard format. You can add version pinning.</p>
<p>environment.yml</p>
<pre><code>name: optional
Expand All @@ -228,12 +230,16 @@ <h2 data-number="2.1" class="anchored" data-anchor-id="environment.yml"><span cl
<h2 data-number="2.2" class="anchored" data-anchor-id="conda-lock.yml"><span class="header-section-number">2.2</span> conda-lock.yml</h2>
<p>Instead of a list of conda packages (typically called environment.yml), you can use a conda lock file instead.</p>
<p>Here is the code for your Docker file. You can name your conda lock file something other than <code>conda-lock.yml</code>.</p>
<pre><code>RUN /pyrocket_scripts/install-conda-packages.sh conda-lock.yml</code></pre>
<pre><code>COPY conda-lock.yml conda-lock.yml
RUN /pyrocket_scripts/install-conda-packages.sh conda-lock.yml
RUN rm conda-lock.yml</code></pre>
</section>
<section id="requirments.txt" class="level2" data-number="2.3">
<h2 data-number="2.3" class="anchored" data-anchor-id="requirments.txt"><span class="header-section-number">2.3</span> requirments.txt</h2>
<section id="requirements.txt" class="level2" data-number="2.3">
<h2 data-number="2.3" class="anchored" data-anchor-id="requirements.txt"><span class="header-section-number">2.3</span> requirements.txt</h2>
<p>The <code>install-pip-packages.sh</code> script will install packages using <code>pip</code>. Here is the code for your Docker file. You can name your pip package file something other than <code>requirements.txt</code>.</p>
<pre><code>RUN /pyrocket_scripts/install-pip-packages.sh requirements.txt</code></pre>
<pre><code>COPY requirements.txt requirements.txt
RUN /pyrocket_scripts/install-pip-packages.sh requirements.txt
RUN rm requirements.txt</code></pre>
<p>requirements.txt</p>
<pre><code>#a package
harmony-py
Expand All @@ -243,7 +249,9 @@ <h2 data-number="2.3" class="anchored" data-anchor-id="requirments.txt"><span cl
<h2 data-number="2.4" class="anchored" data-anchor-id="install.r"><span class="header-section-number">2.4</span> install.R</h2>
<p>The <code>install-r-packages.sh</code> script will run the supplied R script which you can use to install R packages to the system library.</p>
<p>Here is the code for your Docker file. You can name the R script file to something other than <code>install.R</code>. Make sure your file is an R script.</p>
<pre><code>RUN /pyrocket_scripts/install-r-packages.sh install.R</code></pre>
<pre><code>COPY install.R install.R
RUN /pyrocket_scripts/install-r-packages.sh install.R
RUN rm install.R</code></pre>
<p>install.R example</p>
<pre><code># to match rocker/verse:4.4 used in py-rocker-base
# look up the date that the Rocker image was created and put that
Expand All @@ -266,7 +274,9 @@ <h3 data-number="2.4.1" class="anchored" data-anchor-id="add-r-geospatial-packag
<h2 data-number="2.5" class="anchored" data-anchor-id="apt.txt"><span class="header-section-number">2.5</span> apt.txt</h2>
<p>The <code>install-apt-packages.sh</code> script will install packages with <code>apt-get</code>. Here is the code for your Docker file. You can name the apt file of packages names to something other than <code>apt.txt</code>. Comments and newlines are allowed. Installation requires root.</p>
<pre><code>USER root
COPY apt.txt apt.txt
RUN /pyrocket_scripts/install-apt-packages.sh apt.txt
RUN rm apt.txt
USER ${NB_USER}</code></pre>
<p>apt.txt example</p>
<pre><code># Some useful stuff
Expand All @@ -277,8 +287,10 @@ <h2 data-number="2.5" class="anchored" data-anchor-id="apt.txt"><span class="hea
</section>
<section id="postbuild" class="level2" data-number="2.6">
<h2 data-number="2.6" class="anchored" data-anchor-id="postbuild"><span class="header-section-number">2.6</span> postBuild</h2>
<p>The <code>run-postbuild.sh</code> script can be run as root or jovyan (<code>${NB_USER}</code>). This script does not accept a file name. You need to name your postBuild script <code>postBuild</code> and put at the base level with your Docker file.</p>
<pre><code>RUN /pyrocket_scripts/run-postbuild.sh</code></pre>
<p>The <code>run-postbuild.sh</code> script can be run as root or jovyan (<code>${NB_USER}</code>). The script has some extra code to remove leftover files after installing Python extensions.</p>
<pre><code>COPY postBuild postBuild
RUN /pyrocket_scripts/run-postbuild.sh postBuild
RUN rm postBuild</code></pre>
<p>postBuild</p>
<pre><code>#!/bin/bash -l
set -e
Expand All @@ -287,9 +299,11 @@ <h2 data-number="2.6" class="anchored" data-anchor-id="postbuild"><span class="h
</section>
<section id="start" class="level2" data-number="2.7">
<h2 data-number="2.7" class="anchored" data-anchor-id="start"><span class="header-section-number">2.7</span> start</h2>
<p>The <code>start</code> bash code is run when the image starts. py-rocker-base has a start script at <code>${REPO_DIR}/start</code> which loads the Desktop applications. If you change that start file (by copying your start file onto that location), then the Desktop apps will not be loaded properly. Instead, the <code>setup-start.sh</code> will add your start file to the end of <code>${REPO_DIR}/start</code> so that the Desktop is still setup properly.</p>
<p>The <code>setup-start.sh</code> script does not accept a file name. You need to name your start script <code>start</code> and put at the base level with your Docker file.</p>
<pre><code>RUN /pyrocket_scripts/setup-start.sh</code></pre>
<p>The <code>start</code> bash code is run when the image starts. py-rocker-base has a start script at <code>${REPO_DIR}/start</code> which loads the Desktop applications. If you change that start file (by copying your start file onto that location), then the Desktop apps will not be loaded properly. Instead, the <code>setup-start.sh</code> will add your start file to a directory <code>${REPO_DIR}/childstarts</code> and will run all those scripts after <code>${REPO_DIR}/start</code>.</p>
<p>The <code>setup-start.sh</code> script will move the file you provide into <code>${REPO_DIR}/childstarts</code>. As usual you can name your script something other than <code>start</code>.</p>
<pre><code>COPY start start
RUN /pyrocket_scripts/setup-start.sh start
RUN rm start</code></pre>
</section>
<section id="desktop-applications" class="level2" data-number="2.8">
<h2 data-number="2.8" class="anchored" data-anchor-id="desktop-applications"><span class="header-section-number">2.8</span> Desktop applications</h2>
Expand Down
Loading

0 comments on commit 732c273

Please sign in to comment.