Gfarm-S3-MinIO-Web

Overview

• Gfarm-S3-MinIO-Web is a web server to manage a Gfarm S3 MinIO server, which is an S3 compatible object storage server for Gfarm (MinIO with the Gfarm gateway), and manages access control list to flexibly share files with others.
• All software are executed by Docker containers. It is easy to set up.

Hardware requirements

• 1GB memory or more per user (per MinIO process)

Software requirements

• Docker Engine and Docker Compose
  • (Docker Desktop is not required.)
• GNU make

Quick start (Install using Docker Compose)

% git clone https://github.com/oss-tsukuba/gfarm-s3-minio-web.git
% cd gfarm-s3-minio-web/docker

Create config.env (see details below)

SERVER_NAME=localhost
PROTOCOL=https
MYPROXY_SERVER=myproxy-server.domain.name:7512
GFARM_S3_SHARED_DIR=/share
GFARM_CONF_DIR=/etc

This example assumes user's directory for Gfarm S3 MinIO server is /share/username in Gfarm file system. To share files among other users, all users' directories should be under the /share directory. The Gfarm configuration file gfarm2.conf is placed in /etc.

Create gfarm-s3-usermap.conf in GFARM_CONF_DIR directory for all users as the following format;

Gfarm username:local username:S3 access key ID

Copy docker-compose.override.yml.https to docker-compose.override.yml. You can modify docker-compose.override.yml if required.

% cp -p docker-compose.override.yml.https docker-compose.override.yml

Check configuration

% make check-config

Create a self-signed certificate for Gfarm S3 MinIO web server

% make selfsigned-cert-generate

Create and start containers

% make reborn

If old containers already exist, they will be updated as necessary.

If there is a server certificate for Gfarm S3 MinIO web server, copy to gfarm-s3-revproxy-1:/etc/nginx/certs volume, and restart the reverse proxy container.

% sudo docker cp ${SERVER_NAME}.key gfarm-s3-revproxy-1:/etc/nginx/certs/${SERVER_NAME}.key
% sudo docker cp ${SERVER_NAME}.csr gfarm-s3-revproxy-1:/etc/nginx/certs/${SERVER_NAME}.csr
% sudo docker cp ${SERVER_NAME}.crt gfarm-s3-revproxy-1:/etc/nginx/certs/${SERVER_NAME}.crt
% make restart@revproxy

Or, you can create docker-compose.override.yml to use, for example, the Let's Encrypt certificate using acme-companion for nginx-proxy. See https://github.com/nginx-proxy/acme-companion and https://github.com/nginx-proxy/acme-companion/blob/main/docs/Docker-Compose.md

How to use Gfarm-S3-MinIO-Web

1. Access https://${SERVER_NAME}:${HTTPS_PORT}

2. Login with a Gfarm username. Password is a hash of the Gfarm shared key generated by openssl sha256 -r ~/.gfarm_shared_key | cut -d " " -f 1, a passphrase of a client certificate, or a password for a myproxy server.

3. Start a Gfarm S3 MinIO server for the login user by Start button.

4. Manage access control list for each bucket by Sharing page.

How to access Gfarm S3 MinIO server

You can use any S3 compatible client;

• AWS CLI (S3 client)
  • https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html
  • aws configure --profile
    • Access Key ID
    • Secret Access Key
  • export AWS_EC2_METADATA_DISABLED=true
  • set --profile for aws cli to specify the profile name
  • set --endpoint-url for aws cli to specify the Gfarm S3 server URL
  • Example: aws --endpoint-url ... --profile ... s3 mb s3://abcdefg
• Other S3 clients
  • path-style option is required

Install on real machine (not recommended, no details provided)

Please refer to:

• gfarm-s3-minio-web
  • docker/gfminio/Dockerfile
  • docker/gfminio/entrypoint.sh
• gfarm
  • docker/dev/common/s3/setup.sh

Customization

docker-compose.override.yml.https is an example to setup using a reverse proxy and self-signed certificates.

You can use different reverse proxy by modifying docker-compose.override.yml for your environment.

Configuration file (docker/config.env)

Configuration format

KEY=VALUE

Mandatory parameters

• SERVER_NAME: server name of Gfarm S3 MinIO web server (without port number, not URL)
• PROTOCOL: https or http
• GFARM_S3_SHARED_DIR: Gfarm top directory for Gfarm S3 MinIO server
• GFARM_CONF_DIR : a directory for configuration files, gfarm2.conf and gfarm-s3-usermap.conf, on the host OS

Required parameters when using myproxy server

• MYPROXY_SERVER: myproxy server (hostname:port)

Optional parameters

Default is specified by docker-compose.yml.

• HTTP_PORT: http port. When PROTOCOL is https, redirected to the https port.
• HTTPS_PORT: https port
• GSI_PROXY_HOUR: expiration hours of the certificate for grid-proxy-init or myproxy-logon
• GSI_CERTIFICATES_DIR: a directory for public keys for trusted certificate authorities on the host OS
• DEBUG: debug mode (1: enable)
• DJANGO_DEBUG: debug mode of Django (True or False)
• TZ: TZ environment variable
• http_proxy: http_proxy environment variable
• https_proxy: http_proxy environment variable
• GFARM_S3_MINIO_SRC_GIT_URL: gfarm-s3-minio URL (git repository)
• GFARM_S3_MINIO_SRC_GIT_BRANCH: gfarm-s3-minio branch
• GFARM_S3_MINIO_SRC_DIR: local gfarm-s3-minio instead of downloading
• GFARM_S3_WEBUI_THREADS: the number of threads of Web UI
• GFARM_S3_WEBUI_WORKERS: the number of workers of Web UI
• GFARM_S3_ROUTER_THREADS: the number of threads of S3 router
• GFARM_S3_ROUTER_WORKERS: the number of workers of S3 router
• GFARM_S3_LOCALTMP_SIZE_MB: size of local temp file directory per user
• ALLOWED_HOSTS: ALLOWED_HOSTS for Django
• CSRF_TRUSTED_ORIGINS: CSRF_TRUSTED_ORIGINS for Django
• GO_URL: Golang binary package URL
• HOMEDIR_BASE: parent directory for local (host OS) home directories
• SHARE_HOSTDIR: shared directory between host OS and containers

Directory structure for Gfarm-S3-MinIO server

GFARM_S3_SHARED_DIR is a top directory of Gfarm file system for Gfarm-S3-MinIO server. Physically, all user's directory is GFARM_S3_SHARED_DIR/username, which is the top directory for a S3 client. There is sss virtual bucket to share files among other users. Under the sss directory, S3 client can access other users' directories if permitted.

- sss (virtual bucket to other users)
    - username1
        - bucket
           - object
           - ...
        - ....
    - username2
    - ...
- your_bucket1
    - object
    - ...
- your_bucket2
- ...

Management of containers

All the following actions should be executed in ./docker directory.

Stop all containers

% make stop

Start all containers

% make restart

Update and restart containers when editing configurations or updating the Gfarm-S3-MinIO-Web source files

% make reborn

or

% make reborn-withlog
### `ctrl-c` to stop log messages

Execute a shell in the container of Gfarm S3 MinIO server (the main gfminio container)

% make shell

Copy Gfarm configuration files and credentials for users from the host OS to containers manually, although these files will be copied automatically.

% make copy-home

Backup

Save configuration files, config.env and docker-compose.override.yml if required.

Logging

Output the main gfminio container logs

% make logs

Output logs of a specific container

% make logs@<container name>

Output appended logs as the log file grows

% make logs-follow

or

% make logs-follow@<container name>

NOTE: These logs are lost when running make reborn or make down. If you need to keep logs, keep them manually.

You can change a logging driver by modifying docker-compose.override.yml.

• https://docs.docker.com/compose/compose-file/compose-file-v3/#logging
• https://docs.docker.com/config/containers/logging/configure/

For developers

It is easy to develop Gfarm-S3-MinIO-Web using Gfarm docker/dev environment.

• Add the contents of config.env-docker_dev to config.env, and modify appropriately.
• ln -s <path to gfarm/docker/dev/mnt/COPY_DIR> /work/gfarm-dev
• Log in as user1, and start the Gfarm S3 MinIO server.
• make shell, sudo -i -u user1 and /test-s3.sh; echo $?