Derivative Rodeo Handling Pre-existing Files that are not normally generated via Derivative Rodeo generators

[jeremyf]

Overview:

Existing files are Set A. The generated files are Set B. We want to account A ∪ B

Given a one page PDF (named basename.pdf) with the parent work's identifer of work_id. The basename.pdf that has the following existing derivatives:

• Plain text file (basename.txt)
• Reader file (basename.reader.pdf)
• Thumbnail (basename.jpeg)

When we pre-process the files
Then we will have the following files in the given locations:

• PDF Original: s3://work_id/basename.pdf
• PDF Thumbnail: s3://work_id/basename.thumbnail.jpeg (we copied basename.jpeg to this location)
• PDF Plain Text: s3://work_id/basename.txt
• PDF Reader: s3://work_id/basename.reader.pdf
• Page Image: s3://work_id/basename--page-1.tiff
• Page's thumbnail: s3://work_id/basename--page-1.thumbnail.jpeg
• Page's plain text: s3://work_id/basename--page-1.txt
• Page's alto xml: s3://work_id/basename--page-1.alto.xml
• Page's word coordinates: s3://work_id/basename--page-1.coordinates.json

Given the above pre-processed files
When we ingest the basename.pdf (via the OAI feed)
Then we will have the following resulting data:

• Work: A "work" with identifier of work_id
• Work FileSet: with parent of Work and the PDF Original as the original file, and PDF Thumbnail, PDF Plain Text, and PDF Reader as derived files.
• Page: A "work" with parent of Work
• Page FileSet: with parent of Page and the Page Image as the original file, with Page's Thumbnail, Pages' plain text, Page's alto xml, and Page's word coordinates as derived files.

As of <2023-06-06 Tue>, the IIIF Print Derivative Rodeo service does not account for additional derivatives that might have been "pre-processed". That is to say, the derivative service does not know how to generate the PDF Plain Text, nor does it know to go looking for it.

In the above scenarios, when we're processing the derivatives for the Work FileSet we need to look for additional files that were not generated via generators but instead brought over from pre-existing storage. We would use the copy generator and some kind of glob selector on the files.