DPO3DPKRT-701/generate downloads support for scenes (scene packages) #568
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* switched to streaming approach for file uploads - removed body size check for Nginx to remove filesize cap when uploading - added 'Name' to WorkflowReport to assist with identification and file generation * cleanup of commented prior implementations for UploadWorker * change 'Name' property of WorkflowReport to standard varchar(512)
EDAN has new endpoints that need to be reflected deprecating previous configurations. new: https://console.si.edu/apis/3d-api-dev/api/v1.0/admin/upsertResource * Updated GLTF dependencies and config to use new EDAN endpoints * Fixes #760: updating to new EDAN endpoints - updated defaults when environment variables aren't set to the EDAN dev environment for safety. Note: all developers need to update their environment variables and related scripts (e.g. env.DEV.bat) to point to the new EDAN server. PACKRAT_EDAN_3D_API = https://console.si.edu/apis/3d-api-dev/ Additionally, if doing development against EDAN update the dev environment variables for the EDAN server. PACKRAT_EDAN_SERVER = https://edandev.si.edu/
updated nginx-dev.conf to avoid CORS conflicts in modern browsers when running locally in proxy mode.
* Temporarily bypass Generate Downloads while development wraps up Stakeholders require publishing capability but an issue with the generate downloads process pollutes the system with duplicates. this temporary fix is only for while that development is wrapped up (active) * remove additional reference to generating downloads * cleanup of unused variables due to temporary fix * fixed conditional logic when ingesting and added log output
LDAP/LDAPS endpoints have been updated to better enforce compliance and correct configurations. All environment variables (.env.dev, .env.prod, etc.) need to be updated to use the new endpoint/port for authentication against ActiveDirectory to work. No code has been committed as the environment files sit outside of source control. PACKRAT_LDAP_SERVER=ldap://si-hdc-usdc01.US.SINET.SI.EDU:389 Note: that this will need to be updated further as the port (389) is not their preferred port (636), but that is currently not working as expected and will require deeper configuration and testing.
* (new) table for CookResources * (new) default data in db for 4 Cook instances * (new) 'resources/cook' endpoint for getting best fit for job type. looks at stored support for a specific job type, how many jobs are waiting, and how many jobs are running. include 'job' query param. accepted values: inspect, scene_generation, generate_downloads, photogrammetry. * (new) centralized functionality into class 'CookResource' * (new) getCookEndpoint routine for returning ideal endpoint for CookJobs * (new) refactored getCookResources to support returning one/many results * (new) added async 'initialize' method to IJob interface to allow point for pre-execution initialization of a job. (e.g. find the best fit Cook resource for the job) * (new) log statements in OCFL path * (new) detailed statement on resource selection added to workflow report. * (new) exporting CookResourceInfo * (new) additional logging forCook resources and utility routine * (fix) updated 'resources/cook' endpoint to use central class * (fix) improved logging output * (fix) long timeouts for checking state of a resource * (fix) several errors in CookJob were not appended to any reports.
#557) * (new) factoring in jobs created field for better distribution across resources.
Environment variables need to point to ldaps://ldaps.si.edu:636
(new) improved name sanitizing for ingest titles to avoid conflicts with other systems (e.g. Voyager, Cook, etc.). replaces all special characters with '-' except for -, _, and . (new) updated name sanitizing to include all special characters (fix) switched to encodeURIComponent to handle & correctly in JobCook (fix) localized encodeURIComponent to right before sending to Voyager (fix) proper use of encodeURI vs. encodeURIComponent
(new) remove all apostrophe and single-quotes for filename sanitization
* (new) streams always log any errors now * (new) failed stream operations now handle errors and/or return the error
(new) verify streams are valid before writing files to disk (fix) more flexibility in handling returned status codes from Cook (fix) more details failure and during execution (fix) increased max transmit size for webdav to 100Gb (fix) additional comments and logging for storing assets
avoid conflation with other logs (new) flag for generating downloads (or not) in WorkflowEngine ingestion events (fix) restoring generate-downloads functionality (fix) additional contextual data for Inspection logs (e.g. idModel)
assigned to 'Download:<downloadType>' to allow for capturing it's use and it's type.
(fix) additional comments through the publishing pipeline (fix) extended session timeout to 3hrs
EMaslowskiQ
added a commit
that referenced
this pull request
Jan 25, 2024
…rt (#570) * DPO3DPKRT-701/generate downloads support for scenes (scene packages) (#568) (fix) EDAN resource titles showing undefined model type. modified tags assigned to 'Download:<downloadType>' to allow for capturing it's use and it's type. (fix) inclusion of usdz/draco files in scene package (fix) additional comments through the publishing pipeline (fix) extended session timeout to 3hrs * DPO3DPKRT-778/svx-validation-fails-with-image-credit-field (#569) (fix) updating Voyager parsing/validation of scenes to support modern annotations
EMaslowskiQ
deleted the
DPO3DPKRT-701/generate-downloads-support-for-scenes
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
inclusion of usdz/draco assets in the scene package AND resources package.
Note: branch/commit history is corrupted showing old commits already in the code base. needs rebasing/cleanup