fix/non-ascii character support for derivatives & stream logging #564
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Collaborator
EMaslowskiQ
commented
Jan 3, 2024
EMaslowskiQ
commented
- sanitizing EDAN titles for derivative filenames to remove non-ascii characters
- additional logging details on read/write stream failures
* switched to streaming approach for file uploads - removed body size check for Nginx to remove filesize cap when uploading - added 'Name' to WorkflowReport to assist with identification and file generation * cleanup of commented prior implementations for UploadWorker * change 'Name' property of WorkflowReport to standard varchar(512)
EDAN has new endpoints that need to be reflected deprecating previous configurations. new: https://console.si.edu/apis/3d-api-dev/api/v1.0/admin/upsertResource * Updated GLTF dependencies and config to use new EDAN endpoints * Fixes #760: updating to new EDAN endpoints - updated defaults when environment variables aren't set to the EDAN dev environment for safety. Note: all developers need to update their environment variables and related scripts (e.g. env.DEV.bat) to point to the new EDAN server. PACKRAT_EDAN_3D_API = https://console.si.edu/apis/3d-api-dev/ Additionally, if doing development against EDAN update the dev environment variables for the EDAN server. PACKRAT_EDAN_SERVER = https://edandev.si.edu/
updated nginx-dev.conf to avoid CORS conflicts in modern browsers when running locally in proxy mode.
* Temporarily bypass Generate Downloads while development wraps up Stakeholders require publishing capability but an issue with the generate downloads process pollutes the system with duplicates. this temporary fix is only for while that development is wrapped up (active) * remove additional reference to generating downloads * cleanup of unused variables due to temporary fix * fixed conditional logic when ingesting and added log output
LDAP/LDAPS endpoints have been updated to better enforce compliance and correct configurations. All environment variables (.env.dev, .env.prod, etc.) need to be updated to use the new endpoint/port for authentication against ActiveDirectory to work. No code has been committed as the environment files sit outside of source control. PACKRAT_LDAP_SERVER=ldap://si-hdc-usdc01.US.SINET.SI.EDU:389 Note: that this will need to be updated further as the port (389) is not their preferred port (636), but that is currently not working as expected and will require deeper configuration and testing.
* (new) table for CookResources * (new) default data in db for 4 Cook instances * (new) 'resources/cook' endpoint for getting best fit for job type. looks at stored support for a specific job type, how many jobs are waiting, and how many jobs are running. include 'job' query param. accepted values: inspect, scene_generation, generate_downloads, photogrammetry. * (new) centralized functionality into class 'CookResource' * (new) getCookEndpoint routine for returning ideal endpoint for CookJobs * (new) refactored getCookResources to support returning one/many results * (new) added async 'initialize' method to IJob interface to allow point for pre-execution initialization of a job. (e.g. find the best fit Cook resource for the job) * (new) log statements in OCFL path * (new) detailed statement on resource selection added to workflow report. * (new) exporting CookResourceInfo * (new) additional logging forCook resources and utility routine * (fix) updated 'resources/cook' endpoint to use central class * (fix) improved logging output * (fix) long timeouts for checking state of a resource * (fix) several errors in CookJob were not appended to any reports.
#557) * (new) factoring in jobs created field for better distribution across resources.
Environment variables need to point to ldaps://ldaps.si.edu:636
(new) improved name sanitizing for ingest titles to avoid conflicts with other systems (e.g. Voyager, Cook, etc.). replaces all special characters with '-' except for -, _, and . (new) updated name sanitizing to include all special characters (fix) switched to encodeURIComponent to handle & correctly in JobCook (fix) localized encodeURIComponent to right before sending to Voyager (fix) proper use of encodeURI vs. encodeURIComponent
(new) remove all apostrophe and single-quotes for filename sanitization
* (new) streams always log any errors now * (new) failed stream operations now handle errors and/or return the error
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.