[Improvement] Have predictable document ids for lineage and idempotency

[baitsguy]

We are currently using random uuids to generate unique ids for documents and elements during the ingestion process. This works fine in generally ensuring we're able to uniquely ingest documents into the search store, but makes it hard to track the lineage of a user facing raw document. Additionally, each run of the workflow treats the entire input dataset as new entries and doesn't allow for updates/skips.

We need to add the ability to attach a user-provided identifier (and probably more metadata optionally) to a document that is visible at each stage of the workflow. This will allow users to identify documents, and also allow us to build idempotency into the system. This can be in the form of an explicit manifest file, or just an attribute of the document (e.g. s3 path) to use.

Additionally, for cases where we need to generate ids (i.e. user doesn't provide one), we can consider using a document content hash for idempotency