Google Cloud Storage

Google Cloud Storage

Google offers a mostly S3-compatible API to its Cloud Storage offering. However you must explicitly configure interoperable storage access to enable this for a given project. Further you should run s3fs as follows:

s3fs ${YOUR_BUCKET_NAME} ${YOUR_MOUNT_POINT} \
    -o nomultipart \
    -o passwd_file=${YOUR_PASSWORD_FILE} \
    -o sigv2 \
    -o url=https://storage.googleapis.com

More detailed howto for dummies: Mount Google Cloud Storage (GCS) Bucket in Linux:

First, install s3fs-fuse (includes dummy Debian package creation)

You have to have fpm (https://github.com/jordansissel/fpm) installed: apt-get install ruby ruby-dev gem install fpm

Install s3fs:

git clone https://github.com/s3fs-fuse/s3fs-fuse.git
cd s3fs-fuse
./autogen.sh
./configure
 make
 mkdir /tmp/destdir/
 make install DESTDIR=/tmp/destdir
 fpm -s dir -t deb -n s3fs-fuse -v `date +%Y%m%d`-git -C /tmp/installdir -p s3fs-fuse_VERSION_ARCH.deb
 dpkg -i *.deb
 rm -fr /tmp/destdir

If you don't want to create the Debian package, just omit the above command from DESTDIR directive until the end (meaning just run make install), and of course do not install ruby + fpm.

Get credentials

To generate a developer key:

1. Go to the Google Developers Console.
2. Click the name of the project for which you would like to generate a developer key.
3. In the left sidebar, go to Storage > Cloud Storage > Storage settings.
4. Select Interoperability.
5. If you have not set up interoperability before, click Enable interoperability access.
6. Click Create a new key.
7. echo ACCESSKEY:SECRET > /etc/gcs-auth.txt
8. chmod 600 /etc/gcs-auth.txt

Source: https://cloud.google.com/storage/docs/migrating#migration-simple

Next, mount GCS bucket (inspired by first paragraph of this page):

s3fs gs-bucket-name /mnt/gs-bucket-name -o passwd_file=/etc/gcs-auth.txt -o url=https://storage.googleapis.com -o sigv2 -o nomultipart

Bugs + Debugging

• If you fail mounting the bucket for any reason, you still probably have to do umount /mnt/gs-bucket-name before you try again, otherwise you would get ls: cannot access /mnt/gs-bucket-name: Transport endpoint is not connected
• If you want to see debug messages use: s3fs -d -d ...
• If you need even more debug messages, use s3fs -d -d -f ... (runs the program in foreground)