Create E2E "Golden Copy" tests

[GilmoreA6]

Story

As an ETOR partner, in order to ensure complete mappings and transformations for ETOR messages continue to function as expected, we need to create the ability to have "golden copy" E2E tests where an input test message received from the lab/hospital is run through the entire message processing workflow and the output is compared to a known "good" copy.

Pre-conditions

• none

Acceptance Criteria

• Every validation message from the UCSD implementation is tested as defined in https://docs.google.com/spreadsheets/d/1AnPN2h8Gxf1234wbykmHTP4leqBOq2P4/edit?gid=340931274#gid=340931274
• Golden copy tests failing do not impact whether or not existing RS e2e test will run, and vice versa.

Tasks

• Modify automated staging test workflows (submit and run)
• Create test
• Add input and expected files
• Possible modify cron schedule if needed for golden copy
• Add new receivers in RS
  • Once RS PR is merge, update org settings to nameFormat: "golden-copy"
• Update/add README
• Modify blob in RS
• Test
• Update ADR

Additional Context

• Goes through entire flow and any relevant mappings/transforms
• Is part of RS E2E tests
  • Intended to be implemented within TI
• This will this be in addition to the existing end-to-end tests
• This should run in the pipeline, as with existing RS e2e tests
  • Golden copy tests failing should not impact the existing RS e2e tests running
  • The golden copy tests and RS E2E tests should be independent of each other - if one fails, the other should still run

Definition of Done

• Documentation tasks completed
  • Documentation and diagrams created or updated
    • ADRs (/adr folder)
    • Main README.md
    • Other READMEs in the repo
    • If applicable, update the ReportStream Setup section in README.md
  • Threat model updated
  • API documentation updated
• Code quality tasks completed
  • Code refactored for clarity and no design/technical debt
  • Adhere to separation of concerns; code is not tightly coupled, especially to 3rd party dependencies
• Testing tasks completed
  • Load tests passed
  • Additional e2e tests created
  • Additional RS e2e assertions created in the rs-e2e project for any new transformations. Includes improvements to the assertion code required to make the new assertions
• Build & Deploy tasks completed
  • Build process updated
  • API(s) are versioned
  • Feature toggles created and/or deleted. Document the feature toggle
  • Source code is merged to the main branch

Note: Please remove any DoD items that are not applicable

Research Questions

• Optional: Any initial questions for research

Decisions

• Optional: Any decisions we've made while working on this story

Notes

• Optional: Any reference material or thoughts we may need for later reference

[GilmoreA6]

Created from #1539

[GilmoreA6]

@brick-green @scleary1cs @JohnNKing Ticket that will need refinement for introducing a different way of validating end2end tests for the CLIA validation scenarios

[pluckyswan]

@dbgolson Can you verify if the spreadsheet that is linked within the card is the current copy? It is in the Archived folder. If so, where can I get the files mentioned within Original CDPH File Name row? Also, what is the Archived File Name step?

[dbgolson]

@dbgolson Can you verify if the spreadsheet that is linked within the card is the current copy? It is in the Archived folder. If so, where can I get the files mentioned within Original CDPH File Name row? Also, what is the Archived File Name step?

That spreadsheet is a "living document" as it is updated, I have archived the previous version with a date stamp prefix in the file name and placed it in a subdirectory called "Archived". The most current working document titled "CA PHL NBS Test Results - Test Configurations Concept Codes and Test Case Scenarios.xlsm" is found here - https://drive.google.com/drive/folders/17_jQZj8KVrvtslnn_bAXmb6lEh-o0eNO?usp=drive_link

[pluckyswan]

We currently have the modified workflow for automated staging test back up and running. We need to fix the encoding issue with our input example files, send them through the workflow, and store the expected files in our repo. We may also need to modify RS to have the blob use the external file name.

[GilmoreA6]

PR for modifying RS blob transport to utilize external filename https://github.com/CDCgov/prime-reportstream/pull/16920/files

[pluckyswan]

Once the PR mentioned above is merged in, we will be able to test the full functionality of both workflows.