Duplicate uploader records when client SSH version changes

[wettenhj]

The default format for SSH public key fingerprints has changed in recent versions of OpenSSH.

In OpenSSH v6.7 and earlier, it used to look like this:

$ ssh-keygen -yl -f ~/.ssh/id_rsa
2048 6d:7b:94:49:1d:bf:e9:a9:ef:8f:e9:88:b5:25:6c:c0 comment (RSA)

In OpenSSH v6.8 and later, the default format looks like this:

$ ssh-keygen -yl -f ~/.ssh/id_rsa
2048 SHA256:Vza/1h5wx9quxfZSzsjaS7M7w6zGDZQOn4C2XKwWvdM comment (RSA)

But you can get something which looks similar to the old format as follows:

ssh-keygen -E md5 -yl -f ~/.ssh/id_rsa
2048 MD5:6d:7b:94:49:1d:bf:e9:a9:ef:8f:e9:88:b5:25:6c:c0 comment (RSA)

MyData already has some code to deal with this on Windows where we have more control over what SSH version is being used with MyData:

https://github.com/mytardis/mydata/blob/develop/mydata/utils/openssh.py#L250

However on Mac OS X, as people start to use new SSH versions with the new fingerprint format, we need to ensure that MyData can reconcile this new format with what is already on the MyTardis server.

Here's a description of the problem from MyData's perspective:

1. MyData: I want to be able to upload data with SCP using my private key in ~/.ssh/MyData
2. MyData: I need to check if uploads to MyTardis using that private key have already been approved, so let's generate the corresponding public key fingerprint (a summarized version of the public key) from the private key in ~/.ssh/MyData (using ssh-keygen -yl -f private_key) and query the MyTardis server to check whether our private key will be able to authenticate us for SCP uploads.
3. If the client's SSH version has changed, changing the fingerprint format, then the MyTardis server (actually the "mydata" app within the MyTardis server) can respond with "No, uploads haven't been approved for that key, in fact they haven't even been requested yet".
4. MyData: I need to create a new uploader registration request, because MyTardis told me that it doesn't have a valid uploader request for my SSH key.
5. Then, when MyData creates a datafile record via the MyTardis API (actually via the mytardis-app-mydata's extensions to the API), the API tries to determine an appropriate storage box for the datafile object, based on the uploader registration request. (When the uploader request is approved, the MyTardis administrator assigns a storage box.) Currently, this code assumes that there is only one uploader registration request for each uploader, which can raise an exception if multiple requests have been created for the same uploader, due to multiple SSH fingerprint formats: https://github.com/wettenhj/mytardis-app-mydata/blob/master/api.py#L414

Here's an example of this exception in tardis.log:

[27/Oct/2015 09:44:38] WARNING api obj_create Traceback (most recent call last):
  File "/home/mytardis/mytardis/tardis/apps/mydata/api.py", line 414, in obj_create
    UploaderRegistrationRequest.objects.get(uploader=uploader)
  File "/home/mytardis/virtualenvs/mytardis/local/lib/python2.7/site-packages/django/db/models/manager.py", line 127, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/mytardis/virtualenvs/mytardis/local/lib/python2.7/site-packages/django/db/models/query.py", line 338, in get
    (self.model._meta.object_name, num)
MultipleObjectsReturned: get() returned more than one UploaderRegistrationRequest -- it returned 2!

As a result of the exception being raise above, the URI field of the DataFileObject is not set appropriately - it remains at its default value of None, and so when MyTardis attempts to upload via SSH/SCP, it doesn't have a valid remote path to upload to.

MyData's log could show a failed attempt to create a directory on the staging server, due to the missing URI in the DataFileObject:

2015-10-27 09:31:17,953 - openssh.pyc - 608 - UploadFileFromPosixSystem - UploadWorkerThread-2 - DEBUG - "/usr/bin/ssh" -i "/Users/james/.ssh/MyData" -c arcfour128 -oControlPath="/var/folders/fx/4r1f3hv56gsd5280xhfbb0lc0000gp/T/tmp80PQ2a" -oIdentitiesOnly=yes -oPasswordAuthentication=no -oStrictHostKeyChecking=no -l mydata mytardisdemo.erc.monash.edu.au "mkdir -p \"/mnt/MYTARDIS_STAGING\""
2015-10-27 09:31:18,908 - folders.pyc - 1441 - run - UploadWorkerThread-2 - DEBUG - Upload failed for datafile photo1.jpg in folder Photos
2015-10-27 09:31:18,957 - folders.pyc - 1467 - run - UploadWorkerThread-2 - DEBUG - Traceback (most recent call last):
  File "mydata/controllers/folders.pyc", line 1216, in run
  File "mydata/utils/openssh.pyc", line 551, in UploadFile
  File "mydata/utils/openssh.pyc", line 774, in UploadFileFromPosixSystem
SshException: bash: /mnt/MYTARDIS_STAGING/: Is a directory