feat: Allow downloading the data without ssl certificate verification (…

…#43) (#44)

README.md

@@ -71,15 +71,26 @@ cp docker-compose.override.yml-dev docker-compose.override.yml
 
 ### Download Dev Data
 
-Now you need to obtain the data to serve by the mehari, viguno, and annonars container.
-For this, we have prepared strongly reduced data sets (overall less than 2GB rather than hundreds of GB of data).
+To serve data via the mehari, viguno, and annonars containers, you need to obtain the required datasets. We have prepared significantly reduced datasets (totaling less than 2GB as opposed to hundreds of GB) for development purposes.
 
-We provide a script that will setup the necessary directories, download the data, and create symlinks.
+We provide a script that sets up the necessary directories, downloads the data, and creates symlinks.
+
+By default, the script verifies SSL certificates when downloading data. If you encounter SSL verification issues or operate in an environment where SSL verification is not required, you can disable SSL verification by setting the `NO_VERIFY_SSL` variable to `1` when running the script.
+
+To download the data with SSL verification (default behavior):
 
 ```bash session
 bash download-data.sh
 ```
 
+To download the data without SSL verification:
+
+```bash session
+NO_VERIFY_SSL=1 bash download-data.sh
+```
+
+Note: Disabling SSL verification can make the connection less secure. Use this option only if you understand the risks and it is necessary for your environment.
+
 ### Setup Configuration
 
 The next step step is to create the configuration files in `.dev/config`.

download-data.sh

@@ -14,6 +14,8 @@ export QUIET=${QUIET-0}
 export VERBOSE=${VERBOSE-0}
 # Dry-run mode.
 export DRY_RUN=${DRY_RUN-0}
+# No-verify-ssl mode: do not verify SSL certificates.
+export NO_VERIFY_SSL=${NO_VERIFY_SSL-0}
 # Download options: reduced-dev (default), reduced-exomes, full.
 export DOWNLOAD=${DOWNLOAD-reduced-dev}
 # Directory for static data.
@@ -25,6 +27,13 @@ export DATA_DIR=${DATA_DIR-$DIR_PREFIX/volumes/$STATIC_DIR/data}
 # S3 endpoing URL.
 export S3_ENDPOINT_URL=https://ceph-s3-public.cubi.bihealth.org
 
+# Set S5CMD_NO_VERIFY_SSL_ARG based on NO_VERIFY_SSL
+if [ "$NO_VERIFY_SSL" -eq 1 ]; then
+    S5CMD_NO_VERIFY_SSL_ARG="--no-verify-ssl"
+else
+    S5CMD_NO_VERIFY_SSL_ARG=""
+fi
+
 # -- Versions -----------------------------------------------------------------
 
 # annonars
@@ -251,6 +260,7 @@ while read -r line; do
     run s5cmd \
         --endpoint-url=$S3_ENDPOINT_URL \
         --no-sign-request \
+        $S5CMD_NO_VERIFY_SSL_ARG \
         sync \
         "s3://varfish-public/$(prefix_for $line)/$line/*" \
         $DATA_DIR/download/$line \