Folder binding issue with Docker

[ArthurCalvi]

Hello all,

I am a PhD student working on detecting and classifying forest disturbances using satellite imagery. I am using different Satellites and was doing the processing pipeline myself, but I discovered this library and it looks like a game changer. Unfortunately, I have struggles using it with Docker. I am new to Docker and don't quite understand how the binding between the Docker image and my computer's folders works.

Here is my problem:
I am using a Mac and installed Force using Docker without any issues. I followed the usage guide for Docker and added the following alias to my .zshrc file:

alias dforce="docker run -v /Users/username/Force:/opt/data --user \\\\"$(id -u):$(id -g)\\\\" --env FORCE_CREDENTIALS=/app/credentials -v $HOME:/app/credentials davidfrantz/force"

If I understand correctly, this step creates a binding between the directory /opt/data in the Docker image and the directory on my computer, /Users/username/Force. I sourced the .zshrc file and created subfolders in the Force directory (level1, level2, param, etc.). I assume that the Docker image now has access to these subfolders.

Then, I started the first tutorial on Level 1 Cloud Storage Downloader. I managed to authorize access to the GCP API with "gsutil config -e" without any issues. Next, I tried to download the metadata by typing:

dforce force-level1-csd -u /opt/data`

The download took some time but it ran correctly. At the end of the download, I moved the metadata to the directory /Users/username/Force/metadata.
Continuing with the tutorial, I typed the following command to see the available images:

dforce force-level1-csd -n -k -c 0,75 -d 20210101,20220101 opt/data/metadata opt/data/level1 opt/data/level1/pool.txt opt/data/shapefile/disturbance1.shp

This is where I ran into an issue. I received an error message saying that "opt/data/level1" is not a directory :
Error: The specified path for the Level 1 data pool does not seem to be a directory.
I am a bit confused about the folder binding inside the container. Shouldn't opt/data point to /Users/username/Force and have access to its subfolders? It seems that this is not the case.

I really want to give a try to the library. Could someone help me identify the issue?
Cheers,

Arthur

[ernstste]

Hi Arthur,

it seems like you are missing a forward slash in front of your paths opt/data/metadata should be /opt/data/metadata. You might also want to make sure that the folder for your level 1 data pool was created already before you start force-level1-csd.

Hope that helps and you get to play with FORCE. Depending on what you are planning to do you might want to think about deploying FORCE on another machine though. You might run into limitations with your mac (assuming we are talking about a laptop here).

Stefan

[ArthurCalvi]

Hello Stefan,

Thank you for the response, it was a beginner's mistake! I managed to launch force-level1-csd with the -n argument. However, when I want to download the tiles with this command :

dforce force-level1-csd -k -c 0,75 -d 20180601,20181231 /opt/data/metadata /opt/data/ /opt/data/level1/pool.txt /opt/data/shapefile/disturbance1.shp

I received this error :

tput: No value for $TERM and no -T specified
|         |   0 %                                               Your "OAuth 2.0 Service Account" credentials are invalid. For more help, see "gsutil help creds", or re-run the gsutil config command (see "gsutil help config").
[Errno 2] No such file or directory: '/Users/[user]/force-389709-45c116b71df9.json'
CommandException: 1 file/object could not be transferred.

There is a similar issue here.

I checked, the boto file created when using gsutil config -e is at ~/.boto and it contains those lines :

[Credentials]

# Google OAuth2 service account credentials (for "gs://" URIs):
gs_service_key_file = /Users/[user]/force-389709-45c116b71df9.json

So I guess that the container does not have access to this particular file. Do you know how to fix this error ?

Best,
Arthur

[ernstste]

Hi Arthur,

it looks like the google service key file is stored in your home directory on the host machine, but the home directory of the host is not mounted as home directory in the docker container.
Two ways to solve this:

1. Mount the home directory (add -v $HOME:/$HOME to your dforce alias)
2. Change path to the key file in the .boto file to /app/credentials/force-389709-45c116b71df9.json

Stefan

[ArthurCalvi]

Hello Stefan,

It worked! Thank you for your help.

However, today I ran into another issue. It's related to the alias, which seems to not be working anymore:

docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "-v": executable file not found in $PATH: unknown.
ERRO[0001] error waiting for container: context canceled

But for instance the following command is working :

docker run davidfrantz/force force

Nothing has changed on my computer, and all files are in the same place. So, this error is a bit weird. Have you encountered this error before?

[ernstste]

Seems like the -v flag ended up i the wrong place in your alias. Try to make sure you pass -v to docker run, rather than as a command that is passed to the container itself.