485 public datasets visible when not logged in (#832)

* adding new file * modal allows selecting public * changing front end to allow changing to PUBLIC * changing to public now works * adding new routes for public datasets * routers for public files and datasets added download is available, though button will not work on page when added * running codegen for new files * running codegen for new files * does not work right now * small fixes, a few typos * working now * new direction if public * adding PublicDataset will probably need to refactor content from Dataset so that they can both import the same things, with 'publicView' as boolean that will determine what is shown * fixing indents * adding public folders * error on folders and details * folders do not show details not working * moving layout to new component this should make having a public dataset easier * public dataset view uses DatasetLayout now * console logs, should remove later * cannot read properties of undefined? * cannot read properties of undefined? * fix package lock * error gone but with hardcoded limit,not sure why * works now with limit * this seemed to get rid of the error * public view works, shows only public datasets changes coming to PublicDataset need some refactoring of Dataset component * fixing import, working on publicDataset component now * files table has option for public now, will use different route * publicactions menu added removing tabs we don't use for public datasets * new tab panel components * removing unused components * adding public file * adding actions and states for public folders gradually switching public views to use different states than private ones * fixing package lock * fixing wrong method called for public dataset details we see file, some features removed that a public file should not have * public file - no editing or modifying user metadata * index now has authenticated and public using booleans instead of statuses here for now, might change later * adding public and authenticated to index these will be used for searching later * adding new routes for public search from public pages, the search will only return stuff that is public * adding tests * sort of works * using a status field rather than 2 booleans since both cannot be true * tests pass for public datasets, need tests for authenticated added * adding public and authenticated to the user clause * public no longer private router, but problems with layout * errors on files, using new publicLayout for later * public file actions not tested yet, * dataset works, need to fix file * public file added tabs removed for public dataset * public file page shows up, has error * something breaks on page, might be related to visualizations * errors seem to be on visualization dab * routes had dependency that was not needed * public file menu boolean, not all options should appear adding public visualizations, we need that and metadata eventually * adding public visualizations * public visualization view * adding to reducers index, fixing names * will need to use another file actions menu * removing some unused imports adding public metadata * deleting the metadata public routers just adding method to the public files and datasets * public metadata * adding metadata * fixing public dataset details * no more error on public file * user metadata tab should work now checks if public view, uses other route if public * adding method to public_datasets.js for metadata listener metadata tab * public layout removes links we do not use in this view * change explore to public * some fake data will be public change to DatasetIn * fix public metadata extraction tab for file missing import * using public layout in public dataset * download public file version * route to download public dataset * HACK ADDED - probably not a good idea if the user is public, an email is returned for [email protected] * register or login for public page * adding authenticated status * return public datasets * have some authenticated datasets as well * fixing auth for public datasets * checking for public dataset files * changing auth for public or authenticated things * adding filestatus for easier checks * changing status of files? * adding has public access method * new method * adding public as well as authenticated * can now view files if they are public but it looks like the viewer can do things they should not be able to * check public access method added, fixed to work for files * viewer cannot run extraction on file * formatting * running codegen * formatting * fixing package lock, ws wrong * formatting * remove extraction history tab on public view fixing update of status on dataset patch * fix file summary on public file * reverting change * fixing file status change, looks like that broke removing unnecessary file status field just using downloadResource * return public access * fixing public visualization * fixed the broken visualization on public files * formatting * does not work * think this fixed it * putting downloadPublicResource back in, errors not fixed * putting downloadPublicResource back in, errors not fixed * adding publicView option to the various visualizations. this means the right URL will be used if we are seeing this in public view. otherwise the raw bytes do not download * fix package lock * removing console logs * removing console logs * removing console logs * catch case if authorization is None and no public access for file * role was not working for dataset, fixed there but still not showing on tab * does not break on front end, still needs clean up * back end fixes after merge, some authorizations were not checked correctly. files did not switch to public or authenticated properly * fix file tab extractor * file metadata from listeners now displays for public files * button back for add metadata dataset * ran codegen * public metadata definitions * metadata should work now for public when not signed in * fixed text visualization, endpoint for public_files had a typo * fixing more typoes * formatting * codegen * removing search for now * increment should be false for getting public files * adding public check to dataset files * check public or authenticated in dataset routes * don't use public route * removing console logs * fake data should populate both PUBLIC and AUTHENTICATED datasets * fixing download * fixing type on public/files dataset download fixed download file and file version fixed public elasticsearch added to router * removing unused methods * remove divider * codegen * put divider back in public dataset card allows dataset download when public * removed divider * can download public file from file menu now * remove unused method black formatting * black formatting * black formatting --------- Co-authored-by: Chen Wang <[email protected]>
clowder-framework · Jan 29, 2024 · d76b2c4 · d76b2c4
1 parent cf3877e
commit d76b2c4
Show file tree

Hide file tree

Showing 76 changed files with 5,744 additions and 263 deletions.
diff --git a/backend/app/deps/authorization_deps.py b/backend/app/deps/authorization_deps.py
@@ -5,14 +5,41 @@
 from app.keycloak_auth import get_current_username
 from app.models.authorization import RoleType, AuthorizationDB
 from app.models.datasets import DatasetDB, DatasetStatus
-from app.models.files import FileDB
-from app.models.groups import GroupDB
+from app.models.files import FileOut, FileDB, FileStatus
+from app.models.groups import GroupOut, GroupDB
 from app.models.metadata import MetadataDB
 from app.models.pyobjectid import PyObjectId
 from app.routers.authentication import get_admin
 from app.routers.authentication import get_admin_mode
 
 
+async def check_public_access(
+    resource_id: str,
+    resource_type: str,
+    role: RoleType,
+    current_user=Depends(get_current_username),
+) -> bool:
+    has_public_access = False
+    if role == RoleType.VIEWER:
+        if resource_type == "dataset":
+            if (
+                dataset := await DatasetDB.get(PydanticObjectId(resource_id))
+            ) is not None:
+                if (
+                    dataset.status == DatasetStatus.PUBLIC.name
+                    or dataset.status == DatasetStatus.AUTHENTICATED.name
+                ):
+                    has_public_access = True
+        elif resource_type == "file":
+            if (file := await FileDB.get(PydanticObjectId(resource_id))) is not None:
+                if (
+                    file.status == FileStatus.PUBLIC.name
+                    or file.status == FileStatus.AUTHENTICATED.name
+                ):
+                    has_public_access = True
+    return has_public_access
+
+
 async def get_role(
     dataset_id: str,
     current_user=Depends(get_current_username),
@@ -31,6 +58,11 @@ async def get_role(
             AuthorizationDB.user_ids == current_user,
         ),
     )
+    public_access = await check_public_access(
+        dataset_id, "dataset", RoleType.VIEWER, current_user
+    )
+    if authorization is None and public_access:
+        return RoleType.VIEWER
     return authorization.role
 
 
@@ -55,15 +87,11 @@ async def get_role_by_file(
             if (
                 dataset := await DatasetDB.get(PydanticObjectId(file.dataset_id))
             ) is not None:
-                if dataset.status == DatasetStatus.AUTHENTICATED.name:
-                    auth_dict = {
-                        "creator": dataset.author.email,
-                        "dataset_id": file.dataset_id,
-                        "user_ids": [current_user],
-                        "role": RoleType.VIEWER,
-                    }
-                    authenticated_auth = AuthorizationDB(**auth_dict)
-                    return authenticated_auth
+                if (
+                    dataset.status == DatasetStatus.AUTHENTICATED.name
+                    or dataset.status == DatasetStatus.PUBLIC.name
+                ):
+                    return RoleType.VIEWER
                 else:
                     raise HTTPException(
                         status_code=403,
@@ -199,6 +227,7 @@ async def __call__(
             ) is not None:
                 if (
                     current_dataset.status == DatasetStatus.AUTHENTICATED.name
+                    or current_dataset.status == DatasetStatus.PUBLIC.name
                     and self.role == "viewer"
                 ):
                     return True
@@ -249,7 +278,15 @@ async def __call__(
                     detail=f"User `{current_user} does not have `{self.role}` permission on file {file_id}",
                 )
             else:
-                raise HTTPException(status_code=404, detail=f"File {file_id} not found")
+                if (
+                    file.status == FileStatus.PUBLIC.name
+                    or file.status == FileStatus.AUTHENTICATED.name
+                ) and self.role == RoleType.VIEWER:
+                    return True
+                else:
+                    raise HTTPException(
+                        status_code=404, detail=f"File {file_id} not found"
+                    )
 
 
 class MetadataAuthorization:

diff --git a/backend/app/main.py b/backend/app/main.py
@@ -30,22 +30,27 @@
 from app.models.users import UserDB, UserAPIKeyDB, ListenerAPIKeyDB
 from app.models.visualization_config import VisualizationConfigDB
 from app.models.visualization_data import VisualizationDataDB
-from app.routers import folders, groups, status
+from app.routers import folders, groups, public_folders, status
 from app.routers import (
     users,
     authorization,
     metadata,
+    public_metadata,
     files,
+    public_files,
     metadata_files,
     datasets,
+    public_datasets,
     metadata_datasets,
     authentication,
     keycloak,
     elasticsearch,
+    public_elasticsearch,
     listeners,
     feeds,
     jobs,
     visualization,
+    public_visualization,
     thumbnails,
 )
 
@@ -113,12 +118,22 @@
     tags=["metadata"],
     dependencies=[Depends(get_current_username)],
 )
+api_router.include_router(
+    public_metadata.router,
+    prefix="/public_metadata",
+    tags=["public_metadata"],
+)
 api_router.include_router(
     files.router,
     prefix="/files",
     tags=["files"],
     dependencies=[Depends(get_current_username)],
 )
+api_router.include_router(
+    public_files.router,
+    prefix="/public_files",
+    tags=["public_files"],
+)
 api_router.include_router(
     metadata_files.router,
     prefix="/files",
@@ -132,14 +147,27 @@
     dependencies=[Depends(get_current_username)],
 )
 api_router.include_router(
-    metadata_datasets.router, prefix="/datasets", tags=["metadata"]
+    public_datasets.router,
+    prefix="/public_datasets",
+    tags=["public_datasets"],
+)
+api_router.include_router(
+    metadata_datasets.router,
+    prefix="/datasets",
+    tags=["metadata"],
+    dependencies=[Depends(get_current_username)],
 )
 api_router.include_router(
     folders.router,
     prefix="/folders",
     tags=["folders"],
     dependencies=[Depends(get_current_username)],
 )
+api_router.include_router(
+    public_folders.router,
+    prefix="/public_folders",
+    tags=["public_folders"],
+)
 api_router.include_router(
     listeners.router,
     prefix="/listeners",
@@ -164,6 +192,11 @@
     tags=["elasticsearch"],
     dependencies=[Depends(get_current_username)],
 )
+api_router.include_router(
+    public_elasticsearch.router,
+    prefix="/public_elasticsearch",
+    tags=["public_elasticsearch"],
+)
 api_router.include_router(
     feeds.router,
     prefix="/feeds",
@@ -182,6 +215,11 @@
     tags=["visualizations"],
     dependencies=[Depends(get_current_username)],
 )
+api_router.include_router(
+    public_visualization.router,
+    prefix="/public_visualizations",
+    tags=["public_visualizations"],
+)
 api_router.include_router(
     thumbnails.router,
     prefix="/thumbnails",

diff --git a/backend/app/models/datasets.py b/backend/app/models/datasets.py
@@ -27,6 +27,7 @@ class DatasetStatus(AutoName):
 class DatasetBase(BaseModel):
     name: str = "N/A"
     description: Optional[str] = None
+    status: str = DatasetStatus.PRIVATE.name
 
 
 class DatasetIn(DatasetBase):
@@ -65,7 +66,7 @@ class DatasetDBViewList(View, DatasetBase):
     modified: datetime = Field(default_factory=datetime.utcnow)
     auth: List[AuthorizationDB]
     thumbnail_id: Optional[PydanticObjectId] = None
-    status: Optional[str]
+    status: str = DatasetStatus.PRIVATE.name
 
     class Settings:
         source = DatasetDB

diff --git a/backend/app/models/files.py b/backend/app/models/files.py
@@ -1,7 +1,7 @@
 from datetime import datetime
 from enum import Enum
 from typing import Optional, List
-
+from enum import Enum, auto
 from beanie import Document, View, PydanticObjectId
 from pydantic import Field, BaseModel
 
@@ -10,6 +10,19 @@
 from app.models.users import UserOut
 
 
+class AutoName(Enum):
+    def _generate_next_value_(name, start, count, last_values):
+        return name
+
+
+class FileStatus(AutoName):
+    PRIVATE = auto()
+    PUBLIC = auto()
+    AUTHENTICATED = auto()
+    DEFAULT = auto()
+    TRIAL = auto()
+
+
 class StorageType(str, Enum):
     """Depending on the StorageType,the file may need different properties such as local path or URL.
     Also, some StorageTypes do not support versioning or anonymous sharing."""
@@ -47,6 +60,7 @@ class Settings:
 
 class FileBase(BaseModel):
     name: str = "N/A"
+    status: str = FileStatus.PRIVATE.name
 
 
 class FileIn(FileBase):

diff --git a/backend/app/models/search.py b/backend/app/models/search.py
@@ -46,3 +46,4 @@ class ElasticsearchEntry(BaseModel):
     bytes: Optional[int]
     # metadata fields
     metadata: Optional[List[dict]] = []
+    status: Optional[str]
diff --git a/backend/app/routers/authorization.py b/backend/app/routers/authorization.py
@@ -1,3 +1,5 @@
+import datetime
+
 from beanie import PydanticObjectId
 from beanie.operators import Or, In
 from bson import ObjectId
@@ -92,7 +94,10 @@ async def get_dataset_role(
         if (
             current_dataset := await DatasetDB.get(PydanticObjectId(dataset_id))
         ) is not None:
-            if current_dataset.status == DatasetStatus.AUTHENTICATED.name:
+            if (
+                current_dataset.status == DatasetStatus.AUTHENTICATED.name
+                or current_dataset.status == DatasetStatus.PUBLIC.name
+            ):
                 public_authorization_in = {
                     "dataset_id": PydanticObjectId(dataset_id),
                     "role": RoleType.VIEWER,