CRDC Submission Portal 3.0.0

CRDC Submission Portal 3.0.0 Release Notes (09/30/2024)

[Summary]:

This release includes the following key features:

• Support ICDC Data Model and Submissions
• Support CTDC Data Model and Submissions
• Add Data View within a data submission
• Support cross validation among multiple data submission for the same study
• Auto transfer curated data submission to Data Commons repo
• Auto sync latest data model with Data Commons
• Support Data Submission with DELETE type to delete already released data
• Uploader CLI default configuration file
• Submitter can do data submission using APIs**

[Build Details]

• Frontend: 3.0.0.402
• Backend: 3.0.0.597
• AuthN: 2.1.1.116
• File Validation: 2.0.0.211
• Metadata Validation: 2.0.0.211
• Essential Validation: 2.0.0.211
• Export Validation: 2.0.0.211

[New Features]

• Model Navigator: Public Users can view ICDC Data Model schema and definitions in Model
• Model Navigator: Public Users can view CTDC Data Model schema and definitions in Model
• Model Navigator: Standardize All Submitted Template Names
• Model Navigator: Model navigator correctly displays list types
• Data Model: Automate sync with Latest Data Commons Data Model
• Data Model: List delimiter for properties with list type is configurable
• Data Model: Exclude system generated properties from submitted templates
• Submission Request: Study name should be unique
• Submission Request: Requester can input an Institution if not on the list
• Data Submissions: Support ICDC Data Submissions
• Data Submissions: Support CTDC Data Submissions
• Data Submission: Support Metadata Only Data Submission
• Data Submission: Support Delete Data Submission Type to delete released data
• Data Submission: dbGap ID is required for study with controlled data
• Data Submission: System generates unique file IDs
• Data Submission: Prevent data submissions from inactive organizations
• Data Submission: User can see validation stats chart for data files
• Data Submission: Submitter can perform data submissions via APIs
• Data Submission: Auto delete the Data Submission after 120 days inactivity
• Data Submission: Retain user settings when returning to Data Submissions List Page
• Data Submission: UI improvements, including hover tooltips and layout adjustments
• Email Notification: Trigger Email Notification when a data submission is about to Expire
• Data Submission Workflow: Auto Transfer Metadata to Data Commons when released
• Data Submission Workflow: Auto Transfer data Files to CRDC centralized Data Files Bucket when Completed
• Data Submission Workflow: Submitter can resubmit rejected Data Submission
• Data Submission Workflow: System generates DCF manifests
• Data View: Authorized users can view uploaded Data Files
• Data View: Submitter can view the data uploaded within a Data Submission
• Data View: Submitter can delete data from Data View for a data submission
• Data View: Authorized Users can download the uploaded data from a data submission
• Data View: Authorized User can see individual node details and its relationships
• Validation: Authorized users can export the validation results
• Validation: Display Validation errors if released data to be deleted doesn’t exist
• Validation: Submitter can filter on Data File errors on Validation Results
• Validation: Submitter can see missing or orphaned data files errors
• Validation: Improve the Data Upload and Validation error messages
• Validation: Display Data Submission validation status
• Validation: Display a warning if a data submission contains data that has already been released.
• Cross Validation: Data Curator performs Validation cross multiple data submissions for the same
• Cross Validation: Data Curator can see Cross Validation Results
• Cross Validation: Data Curator can export Cross Validation issues
• Uploader CLI Tool: Submitter can download a default Uploader CLI configuration file
• Documentation: Users can view the Submission Request Instructions
• Documentation: Users can view the Data Submission Instructions
• Documentation: CRDC Submission Portal API Documentation page
• Admin Tools: Organization pull-down list should be sorted alphabetically
• Admin Tools: Retain search criteria when returning to Manage Organizations
• Admin Tools: Retain search criteria when returning to Manage Users

CRDC Data Hub 2.1.0 (MVP2-M3)

CRDC Data Hub 2.1.0 (MVP2-M3) Release Notes (06/25/2024)

Summary:

Introduce the Data Submission. This release includes the following four key features:
•       Support individual Submission Templates
•       Data Uploads
•       Perform Full validations
•       Data Submission Visualization
•       Generate curated Data Submission package and release to Data Commons
•       Enhance downstream Data Commons Data Loader to support Data Hub released package

Build Details
·       Frontend Core: mvp-2.1.0.339
·       Backend Core: crdc-hub-2.1.0.425
·       AuthN: 2.1.0.105
·       File Validation: 1.0.0.129
·       Metadata Validation: 1.0.0.135
·       Essential Validation: 1.0.0.137
·       Export Validation: 1.0.0.129

New Features

Epic	Summary
Admin Tool	Admin should refrain from removing a study from an organization when the study has active data submissions
CDS	CDS can load Data Submission packages released from Data Hub
CDS	[DEV] Migrate existing CDS data to Datahub
Code Refactoring	Login/User Management code refactoring
Code Refactoring	Merge AuthZ service into the BE Core service
Compliance	Adobe Analytics
Compliance	Data Hub Google Analytics Report
Compliance	Ensure 508 Compliance for accessibility
Data Model	Update CDS Data Model version to 3.1.2
Data Model	Show data model version in Model Navigator
Data Model	Update CDS Data Model to v2.1.0
Data Model	Multiple Data Model version support
Nodes Visualization	Data Submission Summary Chart: Sort Stacked Bar chart by counts
Nodes Visualization	Submitter should not see the skipped files included in the batch file counts
Data Upload	Data Submission Upsert mode
Data Upload	Submitter upload Data File batch must require File Manifest
Data Upload	Batch Error messages should be user friendly
Data Upload	Update Batch Type for Data Files upload
Data Upload	Submitter can delete previously uploaded data
Data Upload	The submitter can view list of the uploaded files within a batch
Data Upload	Submitter can update data previously uploaded within the same Data Submission
Data Upload	Generate a unique CRDC ID for metadata node
Data Upload	Submitter can see batch upload errors on Data Upload
Email Notification	Make the CRDC Review Committee configurable
Submission Request	[UAT] Submission Request - Delimiter for "Other Cancer Types" input
Submission Request	[UAT] Submission Request - Program and Study
Submission Request	Submission Request V2: Study Abbreviation should be optional
Submission Request	Submission Request V2: Data Types
Submission Request	Submission Request V2: Data Access and Disease
Submission Request	Submission Request V2: Program and Study
Submission Request	Auto Save Submission Request page
UI Improvements	Individual Nodes scrolling area Improvement
UI Improvements	Data Submission Dashboard Improvements
UI Improvements	[UAT] Change the refresh frequency for long running operations to 1 Sec
UI Improvements	[UAT] Data Curator can perform data validation after data submission is submitted
UI Improvements	Change Upload Type labels within a data submission
UI Improvements	Remove CRDC ID from the Validation Results Table
UI Improvements	Validation Results Table Changes
UI Improvements	Submitter should NOT Update/Delete new data submission
UI Improvements	Data Submissions UI Improvements
UI Improvements	The submitter is informed of the status of the Data Upload completion
UI Improvements	Data Submission Dashboard Improvements
UI Improvements	Update the button colors according to the established standards
UI Improvements	The submitter can initiate validation only for the applicable types
UI Improvements	Improve Data Submission Dashboard Validation Results tab
UI Improvements	Submitter can see user friendly batch IDs
UI Improvements	The submitter is informed of the status of the validation completion
Uploadaer CLI	Uploader CLI Console Log should be user friendly
Uploadaer CLI	Submitter can download Uploader CLI from the website
Validation	Tolerant/ignore empty columns, rows or extra cells
Validation	BE: list delimiter needs to be configurable
Validation	Submitter can submit only the child nodes and pass relationship checks
Validation	Validation Issue messages should be user friendly
Validation	Submitter can filter on validation results
Validation	Submitter can see detailed validation errors
Validation	Submitter can validate both metadata and files
Validation	Submitter can see validation results in Validation Results tab
Validation	Submitter can validate metadata
Validation	Submitter can validate data files
Validation	Submitter can visualize submitted Nodes and counts
Workflow	Reviewer can provide comments when rejecting a data submission
Workflow	Admin provides justification for administrative submit
Workflow	Submitter can see rejection comments for their data submission
Workflow	Data Curator can release Metadata files to CDS
Workflow	Admin can do an Admin Overwrite to "Submit" the Data Submission
Workflow	Submitter can "Submit" the Data Submission

What's Changed

• Remove unused folder and add eslint rules by @amattu2 in #2
• CRDCDH-33 Questionnaire Section A Implementation by @amattu2 in #1
• CRDCDH-80 Implement Unsaved Changes Prompt by @amattu2 in #3
• CRDCDH-35 Implement Form Section B by @amattu2 in #4
• CRDCDH-92 Restructure FormView to support redesign by @amattu2 in #6
• CRDCDH-38 Implement Form Progress Sidebar by @amattu2 in #7
• CRDCDH-95 Add apollo client and integrated with mock api server by @jonkiky in #8
• Fix incorrect Application type by @amattu2 in #10
• CRDCDH-35 Redesign Form Section B by @Alejandro-Vega in #9
• Tracy build by @tracy18 in #5
• Dev by @jonkiky in #14
• merge devops changes to dev by @jonkiky in #15
• CRDCDH-37 Add Form Status Bar and test cases by @amattu2 in #12
• Update graphql by @jonkiky in #16
• Update application types on FE based on mock api by @Alejandro-Vega in #18
• Crdcdh 81 82 by @mtangmt in #13
• CRDCDH-39 by @JoonLeeNIH in #17
• CRDCDH-93 Questionnaire header/banner by @Alejandro-Vega in #19
• Few updates after merge PRs by @jonkiky in #21
• Minor updates by @Alejandro-Vega in #22
• CRDCDH-33 Implement redesigned Section A by @amattu2 in #20
• CRDCDH-80 Unsaved Changes Dialog by @Alejandro-Vega in #23
• Adding underline styles, and fixing dropdown style by @mtangmt in #24
• Small updates on ESlint issues found on dev branch by @jonkiky in #25
• CRDCDH-36 Implement Section C by @Alejandro-Vega in #27
• Crdcdh 45 by @mtangmt in #26
• CRDCDH-33 Update Verbiage & General QoL Updates by @amattu2 in #28
• Adding ids to interactable items in header and form by @mtangmt in #29
• CRDCDH-37 Update Status Bar verbiage by @amattu2 in #30
• CRDCDH-39 Part 2 by @JoonLeeNIH in #32
• Form detecting false positive changes by @Alejandro-Vega in #31
• CRDCDH-39 Part 3 by @JoonLeeNIH in #33
• CRDCDH-131 Remove eRAA field and update placeholders by @amattu2 in #34
• Crdcdh 135 by @mtangmt in #36
• Crdcdh 128 by @mtangmt in #35
• CRDCDH-80 Hide validation when saving from dialog by @Alejandro-Vega in #38
• CRDCDH-36 Adding dataDeIdentified field by @Alejandro-Vega in #37
• Implementing ticket 134 by @mtangmt in #39
• adding ids to section b and formview by @mtangmt in #41
• Crdcdh 135 refinement by @mtangmt in #40
• CRDCDH-133 Updating Program/Study by @Alejandro-Vega in #42
• CRDCDH-39 by @JoonLeeNIH in #43
• deleting file other types by @mtangmt in #44
• CRDCDH-136 Add save success alert popup by @amattu2 in #47
• CRDCDH-39 by @JoonLeeNIH in #48
• change favicon by @mtangmt in #51
• CRDCDH-39 Extend implementation to add AuthZ by @amattu2 in https://github.com/CBIIT/crdc-datahu...

v2.0.0 (MVP2-M2)

CRDC Data Hub 2.0.0 (MVP2-M2) Release Notes (02/26/2024)

Summary:

Introduce the Data Submission. This release includes the following four key features:

• A Data Submission platform to submit study associated metadata and data files.
• A streamlined Data Submission end-to-end Workflow.
• A Data Uploader CLI tool for data files upload
• An Administrative Portal, for system admin to manage organizations

Build Tags:

• Frontend Core: mvp-2.213
• Backend Core: 2.0.0.301
• AuthN: 2.0.0.99
• AuthZ: 2.0.0.69

Epic	Feature
Submission Request	Enforce Program and study abbreviation to all caps
Data Submission	Submitter can view CDS Data Model in Data Model Navigator
Data Submission	Submitter can download Uploader CLI
Data Submission	Make dbGaP ID optional for Data Submission
Data Submission	Submitter can see and copy Data Submission ID
Data Submission	Data Submission Workflow States/Actions
Data Submission	Submitter can see and copy Data Submission ID
Data Submission	Remove Error Count from the Batch Transaction table
Data Submission	Reviewer can reject Submission Request
Data Submission	Reviewer can inquire additional changes on Submission Request
Data Submission	Submitter can select a Data Commons when creating a new Data Submission
Data Upload	Data Loader: Metadata Validation Results Excel format
Data Upload	Metadata Uploader CLI tool
Data Upload	Authenticated user can download API token
Validation	Submitter can download well-formed metadata validation results
Validation	Data Curator use Prefect to validate submitted metadata and data files
Validation	Submitter can download the data submission validation results
Email Notification	Email Notification - when user is marked Inactive by Admin
Email Notification	Email Notification - Inquire Additional Changes on Submission Requests
Email Notification	Email Notification - Complete Data Submission
Email Notification	Add TIER info in the Accounts disabled email
Email Notification	Email Notification - Cancel Data Submission
Email Notification	Email Notification - Reject Data Submission
Email Notification	Email Notification - Submitter Withdraw Data Submission
Email Notification	Email Notification - Release Data Submission
Email Notification	Email Notification - Submit Data Submission
Admin Tool	Admin can add new organizations
Admin Tool	Admin can edit organization
Admin Tool	Admin can manage Organizations
Admin Tool	Admin can assign Data Commons POC

v1.0.1

CRDC Data Hub 1.0.1 Release Notes

Summary:

• implement gov shutdown banner by @mtangmt in #199

Build Tags
·      Frontend Core:  1.0.1.180
·      Backend Core: 1.0.0.124
·      AuthN Service: 1.0.0.90
·      AuthZ Service: 1.0.0.42

Full Changelog: mvp-1.0.0.139...v1.0.1.180

v1.0.0

Introduce the initial version of CRDC Data Submission Portal (Version 1.0.0). This release includes the following three key features:
 
·      An intuitive CRDC Submission Request online form.
·      A streamlined Submission Request Review Workflow.
·      An Administrative Portal, for system admin to manage users and their access.

Build Details
·      Frontend Core:  mvp-1.0.0.139
·      Backend Core: 1.0.0.124
·      AuthN Service: 1.0.0.90
·      AuthZ Service: 1.0.0.42

New Features

Epic	Feature
Site Framework	Data Hub Login Page
Site Framework	Data Hub Welcome Page
Site Framework	Use the CRDC footer for Data Hub Footer
Site Framework	Use the CRDC Header for Data Hub Header
Site Framework	CRDC Data Hub Navigation Bar
Authentication	Authenticated user can view/Edit User Profile
Authentication	CRDC Data Hub Authentication
Authentication	Users after authenticated will see their first name on the top
Authentication	Authenticated user gets a warning before the session timeout
Authentication	Authenticated End User can manually logout
Authentication	Prompt Login when being redirect from CRDC
Authentication	Inactivated user accounts should not log in to Data Hub
Authentication	Default displayed name to login email if login user’s name is not available
Authorization	CRDC DH Role Based Control
Submission Request	CRDC Submission Request List
Submission Request	Submission Request page Layout
Submission Request	Organization owner can only see their own submission requests
Submission Request	Authenticated User can view Submission Request Progress Bar
Submission Request	Authenticated User can view Submission Request Status Header
Submission Request	Authenticated User can edit Program and Study Information
Submission Request	Populate predefined Program Name in Submission Request
Submission Request	Study Abbreviation must be unique
Submission Request	System auto-populate program description
Submission Request	Authenticated User can edit PI and Contacts
Submission Request	System auto populate PI info from previous submission
Submission Request	Authenticated User can edit Data Access and Disease Information
Submission Request	Authenticated User can edit Data Types
Submission Request	Submission Request require at least one Data Type
Submission Request	Add TSV extension to Clinical Data Type
Submission Request	Authenticated User can review and submit Submission Request Form
Submission Request	Authenticated User can only view Submission Request Form
Submission Request	Fed leads can review Submission Request to Approve or Reject
Submission Request	When user leave an unsaved Submission Request Form
Submission Request	Authenticated User can save Submission Request Page
Submission Request	System auto delete Submission Request after 45 days inactivity
Submission Request	Do not auto deleted locked submission requests after 45 days inactivity
Email Notification	System triggered email when Submission Requests are deleted
Email Notification	System triggered email when Submission Requests are about to expire
Email Notification	System triggered email when a Submission Request is approved
Email Notification	System triggered email when a Submission Requests is rejected
Email Notification	System triggered email when a Submit Submission Request
Email Notification	Do not notify Org Owner when submission request is rejected/deleted/expired
Admin Portal	Admin can edit user account access
Admin Portal	Admin can manage Users
Admin Portal	Admin can assign user an organization based on User's role
Security Control	Account Management - Account Audit Trails
Security Control	Account Management - inactivity after 60 Days
Compliance	System Use Notification - Warning Banner