Merge branch 'develop' of https://github.com/CenterForOpenScience/osf.io

 into remove-quickfiles-code

* 'develop' of https://github.com/CenterForOpenScience/osf.io: (44 commits)
  Re-add non-anonymized fields removed in #10009 (#10022)
  Update shield logo for colorado (UC Boulder)
  Update description for maglab
  [ENG-3249] Improve registration anonymization (#10009)
  "backport" artifact changes and swap schema_response.justification (#10003)
  Add new instn purdue
  Ensure BitBucket token is string, not bytes
  Only redirect to cas if not logged in
  OSFI: Update Shared SSO and Add MagLab/FSU [ENG-3654]
  [ENG-3898][ENG-3899]Model support for OutcomeArtifact update and delete (#9989)
  Make OutcomeArtifact.identifier nullable (#9986)
  [ENG-3894] Outcome models (#9975)
  revert color picker to working version (#9968)
  Instrument the ORCiD SSO affiliation flow * Existing user with verified ORCiD ID * Existing user confirmation of linking ORCiD ID * New user confirmation of account creation with ORCiD ID
  Add a django command script to handle instn sso email domain changes
  Bump version and CHANGELOG
  Add better logic for routing Dataverse files and their stupid duplicate (#9963)
  [ENG-3872] Fix dataverse 502s and pass version info to FE (#9959)
  Fix File `show_as_unviewed` behavior (#9960)
  Fix OSFUser.has_resources
  ...

# Conflicts:
#	api_tests/files/views/test_file_detail.py
#	osf/models/nodelog.py
#	osf/models/user.py
#	tests/test_addons.py
#	website/static/js/anonymousLogActionsList.json
#	website/static/js/logActionsList.json

.pre-commit-config.yaml

@@ -1,3 +1,5 @@
+default_language_version:
+  python: python3.6
 repos:
 - repo: https://github.com/asottile/add-trailing-comma
   rev: v0.7.0

CHANGELOG

@@ -2,6 +2,20 @@
 
 We follow the CalVer (https://calver.org/) versioning scheme: YY.MINOR.MICRO.
 
+22.06.0 (2022-06-23)
+====================
+- Fix support for Dataverse files
+- Match Legacy behavior with new `show_as_unviewed` File field
+- Other assorted fixes for new Files page
+
+22.05.0 (2022-06-09)
+====================
+- Add institutional affiliations via ROR to minted DOIs
+- Improve file sorting by `date_modified`
+- Update help links
+- Add ability to create RegistrationSchemas via the Admin App
+- Update files page routing for upcoming FE release
+
 22.04.0 (2022-03-31)
 ====================
 - Update JS and Python dependencies

addons/base/exceptions.py

@@ -13,3 +13,9 @@ class InvalidAuthError(AddonError):
 
 class HookError(AddonError):
     pass
+
+class QueryError(AddonError):
+    pass
+
+class DoesNotExist(AddonError):
+    pass

addons/base/views.py

@@ -19,6 +19,7 @@
 
 from api.caching.tasks import update_storage_usage_with_size
 
+from addons.base import exceptions as addon_errors
 from addons.base.models import BaseStorageAddon
 from addons.osfstorage.models import OsfStorageFileNode
 from addons.osfstorage.utils import update_analytics
@@ -55,7 +56,6 @@
 )
 from osf.metrics import PreprintView, PreprintDownload
 from osf.utils import permissions
-from website.ember_osf_web.views import use_ember_app
 from website.profile.utils import get_profile_image_url
 from website.project import decorators
 from website.project.decorators import must_be_contributor_or_public, must_be_valid_project, check_contributor_auth
@@ -726,21 +726,32 @@ def addon_view_or_download_file(auth, path, provider, **kwargs):
             })
 
     savepoint_id = transaction.savepoint()
-    file_node = BaseFileNode.resolve_class(provider, BaseFileNode.FILE).get_or_create(target, path)
-    if isinstance(target, Node) and waffle.flag_is_active(request, features.EMBER_FILE_PROJECT_DETAIL):
-        return use_ember_app()
 
-    if action != 'download' and isinstance(target, Registration) and waffle.flag_is_active(request, features.EMBER_FILE_REGISTRATION_DETAIL):
-        if file_node.get_guid():
-            guid = file_node.get_guid()
-        else:
-            guid = file_node.get_guid(create=True)
-            guid.save()
-            file_node.save()
-        return redirect(f'{settings.DOMAIN}{guid._id}/')
+    try:
+        file_node = BaseFileNode.resolve_class(
+            provider, BaseFileNode.FILE
+        ).get_or_create(
+            target, path, **extras
+        )
+    except addon_errors.QueryError as e:
+        raise HTTPError(
+            http_status.HTTP_400_BAD_REQUEST,
+            data={
+                'message_short': 'Bad Request',
+                'message_long': str(e)
+            }
+        )
+    except addon_errors.DoesNotExist as e:
+        raise HTTPError(
+            http_status.HTTP_404_NOT_FOUND,
+            data={
+                'message_short': 'Not Found',
+                'message_long': str(e)
+            }
+        )
 
     # Note: Cookie is provided for authentication to waterbutler
-    # it is overriden to force authentication as the current user
+    # it is overridden to force authentication as the current user
     # the auth header is also pass to support basic auth
     version = file_node.touch(
         request.headers.get('Authorization'),
@@ -749,6 +760,16 @@ def addon_view_or_download_file(auth, path, provider, **kwargs):
             cookie=request.cookies.get(settings.COOKIE_NAME)
         )
     )
+
+    # There's no download action redirect to the Ember front-end file view and create guid.
+    if action != 'download':
+        if isinstance(target, Node) and waffle.flag_is_active(request, features.EMBER_FILE_PROJECT_DETAIL):
+            guid = file_node.get_guid(create=True)
+            return redirect(f'{settings.DOMAIN}{guid._id}/')
+        if isinstance(target, Registration) and waffle.flag_is_active(request, features.EMBER_FILE_REGISTRATION_DETAIL):
+            guid = file_node.get_guid(create=True)
+            return redirect(f'{settings.DOMAIN}{guid._id}/')
+
     if version is None:
         # File is either deleted or unable to be found in the provider location
         # Rollback the insertion of the file_node

addons/bitbucket/api.py

@@ -3,14 +3,14 @@
 from addons.bitbucket import settings
 
 from framework.exceptions import HTTPError
-
+from osf.utils.fields import ensure_str
 from website.util.client import BaseClient
 
 
 class BitbucketClient(BaseClient):
 
     def __init__(self, access_token=None):
-        self.access_token = access_token
+        self.access_token = ensure_str(access_token)
 
     @property
     def _default_headers(self):

addons/dataverse/models.py

@@ -1,14 +1,14 @@
 # -*- coding: utf-8 -*-
 from rest_framework import status as http_status
 
+from addons.base import exceptions as addon_errors
 from addons.base.models import (BaseOAuthNodeSettings, BaseOAuthUserSettings,
                                 BaseStorageAddon)
+from django.contrib.contenttypes.models import ContentType
 from django.db import models
 from framework.auth.decorators import Auth
 from framework.exceptions import HTTPError
 from osf.models.files import File, Folder, BaseFileNode
-from osf.utils.permissions import WRITE
-from framework.auth.core import _get_current_user
 from addons.base import exceptions
 from addons.dataverse.client import connect_from_settings_or_401
 from addons.dataverse.serializer import DataverseSerializer
@@ -17,6 +17,36 @@
 class DataverseFileNode(BaseFileNode):
     _provider = 'dataverse'
 
+    @classmethod
+    def get_or_create(cls, target, path, **query_params):
+        '''Override get_or_create for Dataverse.
+
+        Dataverse is weird and reuses paths, so we need to extract a "version"
+        query param to determine which file to get. We also don't want to "create"
+        here, as that might lead to integrity errors.
+        '''
+        version = query_params.get('version', None)
+        if version not in {'latest', 'latest-published'}:
+            raise addon_errors.QueryError(
+                'Dataverse requires a "version" query paramater. '
+                'Acceptable options are "latest" or "latest-published"'
+            )
+
+        content_type = ContentType.objects.get_for_model(target)
+        try:
+            obj = cls.objects.get(
+                target_object_id=target.id,
+                target_content_type=content_type,
+                _path='/' + path.lstrip('/'),
+                _history__0__extra__datasetVersion=version,
+            )
+        except cls.DoesNotExist:
+            raise addon_errors.DoesNotExist(
+                f'Requested Dataverse file does not exist with version "{version}"'
+            )
+
+        return obj
+
 
 class DataverseFolder(DataverseFileNode, Folder):
     pass
@@ -33,25 +63,11 @@ def _hashes(self):
             return None
 
     def update(self, revision, data, save=True, user=None):
-        """Note: Dataverse only has psuedo versions, pass None to not save them
+        """Note: Dataverse only has psuedo versions (_history), pass None to not save them
         Call super to update _history and last_touched anyway.
-        Dataverse requires a user for the weird check below
         """
-        version = super(DataverseFile, self).update(None, data, user=user, save=save)
+        version = super().update(None, data, user=user, save=save)
         version.identifier = revision
-
-        user = user or _get_current_user()
-        if not user or not self.target.has_permission(user, WRITE):
-            try:
-                # Users without edit permission can only see published files
-                if not data['extra']['hasPublishedVersion']:
-                    # Blank out name and path for the render
-                    # Dont save because there's no reason to persist the change
-                    self.name = ''
-                    self.materialized_path = ''
-                    return (version, '<div class="alert alert-info" role="alert">This file does not exist.</div>')
-            except (KeyError, IndexError):
-                pass
         return version
 
 

addons/dropbox/models.py

@@ -2,25 +2,22 @@
 import logging
 import os
 
-from oauthlib.common import generate_token
-
 from addons.base.models import (BaseOAuthNodeSettings, BaseOAuthUserSettings,
                                 BaseStorageAddon)
 from django.db import models
 from dropbox.dropbox import Dropbox
 from dropbox.exceptions import ApiError, DropboxException
 from dropbox.files import FolderMetadata
-from dropbox import DropboxOAuth2Flow, oauth
-from flask import request
+from furl import furl
 from framework.auth import Auth
 from framework.exceptions import HTTPError
-from framework.sessions import session
 from osf.models.external import ExternalProvider
 from osf.models.files import File, Folder, BaseFileNode
+from osf.utils.fields import ensure_str
 from addons.base import exceptions
 from addons.dropbox import settings
 from addons.dropbox.serializer import DropboxSerializer
-from website.util import api_v2_url, web_url_for
+from website.util import api_v2_url
 
 logger = logging.getLogger(__name__)
 
@@ -49,62 +46,31 @@ class Provider(ExternalProvider):
     client_id = settings.DROPBOX_KEY
     client_secret = settings.DROPBOX_SECRET
 
-    # Explicitly override auth_url_base as None -- DropboxOAuth2Flow handles this for us
-    auth_url_base = None
-    callback_url = None
-    handle_callback = None
-
-    @property
-    def oauth_flow(self):
-        if 'oauth_states' not in session.data:
-            session.data['oauth_states'] = {}
-        if self.short_name not in session.data['oauth_states']:
-            session.data['oauth_states'][self.short_name] = {
-                'state': generate_token()
-            }
-        return DropboxOAuth2Flow(
-            self.client_id,
-            self.client_secret,
-            redirect_uri=web_url_for(
-                'oauth_callback',
-                service_name=self.short_name,
-                _absolute=True
-            ),
-            session=session.data['oauth_states'][self.short_name], csrf_token_session_key='state'
-        )
+    auth_url_base = settings.DROPBOX_OAUTH_AUTH_ENDPOINT
+    callback_url = settings.DROPBOX_OAUTH_TOKEN_ENDPOINT
+    auto_refresh_url = settings.DROPBOX_OAUTH_TOKEN_ENDPOINT
+    refresh_time = settings.REFRESH_TIME
 
     @property
     def auth_url(self):
-        ret = self.oauth_flow.start('force_reapprove=true')
-        session.save()
-        return ret
-
-    # Overrides ExternalProvider
-    def auth_callback(self, user):
-        # TODO: consider not using client library during auth flow
-        try:
-            access_token = self.oauth_flow.finish(request.values).access_token
-        except (oauth.NotApprovedException, oauth.BadStateException):
-            # 1) user cancelled and client library raised exc., or
-            # 2) the state was manipulated, possibly due to time.
-            # Either way, return and display info about how to properly connect.
-            return
-        except (oauth.ProviderException, oauth.CsrfException):
-            raise HTTPError(http_status.HTTP_403_FORBIDDEN)
-        except oauth.BadRequestException:
-            raise HTTPError(http_status.HTTP_400_BAD_REQUEST)
+        # Dropbox requires explicitly requesting refresh_tokens via `token_access_type`
+        # https://developers.dropbox.com/oauth-guide#implementing-oauth
+        url = super(Provider, self).auth_url
+        return furl(url).add({'token_access_type': 'offline'}).url
 
+    def handle_callback(self, response):
+        access_token = response['access_token']
         self.client = Dropbox(access_token)
-
         info = self.client.users_get_current_account()
-        return self._set_external_account(
-            user,
-            {
-                'key': access_token,
-                'provider_id': info.account_id,
-                'display_name': info.name.display_name,
-            }
-        )
+        return {
+            'key': access_token,
+            'provider_id': info.account_id,
+            'display_name': info.name.display_name,
+        }
+
+    def fetch_access_token(self, force_refresh=False):
+        self.refresh_oauth_key(force=force_refresh)
+        return ensure_str(self.account.oauth_key)
 
 
 class UserSettings(BaseOAuthUserSettings):
@@ -119,7 +85,7 @@ def revoke_remote_oauth_access(self, external_account):
 
         Tells Dropbox to remove the grant for the OSF associated with this account.
         """
-        client = Dropbox(external_account.oauth_key)
+        client = Dropbox(Provider(external_account).fetch_access_token())
         try:
             client.auth_token_revoke()
         except DropboxException:
@@ -158,6 +124,9 @@ def folder_path(self):
     def display_name(self):
         return '{0}: {1}'.format(self.config.full_name, self.folder)
 
+    def fetch_access_token(self):
+        return self.api.fetch_access_token()
+
     def clear_settings(self):
         self.folder = None
 
@@ -177,7 +146,7 @@ def get_folders(self, **kwargs):
                 }
             }]
 
-        client = Dropbox(self.external_account.oauth_key)
+        client = Dropbox(self.fetch_access_token())
 
         try:
             folder_id = '' if folder_id == '/' else folder_id
@@ -230,7 +199,7 @@ def deauthorize(self, auth=None, add_log=True):
     def serialize_waterbutler_credentials(self):
         if not self.has_auth:
             raise exceptions.AddonError('Addon is not authorized')
-        return {'token': self.external_account.oauth_key}
+        return {'token': self.fetch_access_token()}
 
     def serialize_waterbutler_settings(self):
         if not self.folder:

addons/dropbox/settings/defaults.py

@@ -3,6 +3,9 @@
 DROPBOX_SECRET = None
 
 DROPBOX_AUTH_CSRF_TOKEN = 'dropbox-auth-csrf-token'
+DROPBOX_OAUTH_AUTH_ENDPOINT = 'https://www.dropbox.com/oauth2/authorize'
+DROPBOX_OAUTH_TOKEN_ENDPOINT = 'https://www.dropbox.com/oauth2/token'
+REFRESH_TIME = 14399  # 4 hours
 
 # Max file size permitted by frontend in megabytes
 MAX_UPLOAD_SIZE = 150

addons/googledrive/README.md

@@ -11,7 +11,8 @@
 3. Click on the "Google Drive API" link, and enable it
 4. Click on "Credentials", and "create credentials". Select "Oath Client ID", with "web application" and set the redirect uri to `http://localhost:5000/oauth/callback/googledrive/`
 5. Submit your new client ID and make a note of your new ID and secret
-6. (Optional) You may find that the default 10 "QPS per User" rate limit is too restrictive. This can result in unexpected 403 "User Rate Limit Exceeded" messages. You may find it useful to request this limit be raised to 100. To do so, in the Google API console, from the dashboard of your project, click on "Google Drive API" in the list of APIs. Then click the "quotas" tab. Then click any of the pencils in the quotas table. Click the "apply for higher quota" link. Request that your "QPS per User" be raised to 100.  
+6. Add yourself as a test user and ensure the oauth app is configured securely
+7. (Optional) You may find that the default 10 "QPS per User" rate limit is too restrictive. This can result in unexpected 403 "User Rate Limit Exceeded" messages. You may find it useful to request this limit be raised to 100. To do so, in the Google API console, from the dashboard of your project, click on "Google Drive API" in the list of APIs. Then click the "quotas" tab. Then click any of the pencils in the quotas table. Click the "apply for higher quota" link. Request that your "QPS per User" be raised to 100.  
 
 ### Enable for OSF
 1. Create a local googledrive settings file with `cp addons/googledrive/settings/local-dist.py addons/googledrive/settings/local.py`

addons/osfstorage/models.py

@@ -85,7 +85,7 @@ def get(cls, _id, target):
         return cls.objects.get(_id=_id, target_object_id=target.id, target_content_type=ContentType.objects.get_for_model(target))
 
     @classmethod
-    def get_or_create(cls, target, path):
+    def get_or_create(cls, target, path, **unused_query_params):
         """Override get or create for osfstorage
         Path is always the _id of the osfstorage filenode.
         Use load here as its way faster than find.