PSC-STM-A6: Complete PSC Stream Ingester App

[tiredpixel]

The stream ingester hasn’t been run or updated since the first prototype.

Some work will be required to bring it up to date with how the other apps behave.

Estimate: 4 hours

[StephenAbbott]

Delete translations are sent down the streaming API by Companies House but these only go back 7 days at the moment. They said they wanted to develop streaming API snapshots but haven't seen any announcements from them on this front.

I will email contacts there to make sure this hasn't been prioritised in case that would be useful to us.
See "deleted" event.type here https://developer-specs.company-information.service.gov.uk/streaming-api/resources/pscstatementsstream?v=latest

When it comes to working on this, we should bear in mind that deletions should trigger recordStatus changing to closed for BODS v0.4 https://docs.google.com/document/d/1SX1nm3MSC81oGJ16voBj41xrKYHuxvFi-QVUZ_OjGRc/edit

[tiredpixel]

The streaming ingestion code has been updated to work with the later changes, and it seems to be working for streaming updates in development.

I think deleted events or similar are out-of-scope for this, since BODS 0.4 hasn't yet been released, and it's not at all clear to me how deleted events should or could be handled within the existing Register 2 code and data storage.

[StephenAbbott]

@tiredpixel We're close to the code freeze for version 0.4 of BODS. This will hopefully happen by the end of April - and I want to give you plenty of time to acquaint yourself with the changes so that we're ready to update the Register application later this year.

For now, can you look over these publicly accessible bits of technical guidance about how to handle information updates to records in BODS version 0.4 and also record identifiers:
https://standard.openownership.org/en/main/schema/guidance/information-updates.html
https://standard.openownership.org/en/main/schema/guidance/record-ids.html