Add collector payload to the bad stream data

[szareiangm]

When we have a new integration, usually it has some events going to the bad stream. Finding them is not straight-forward because the bad stream data is not searchable in Elasticsearch due to its encoding (Base64).

Our proposal is to parse the Collector payload as a Json with its Snowplow library and add it beside the the bad row data.

[chuwy]

Hi @szareiangm, that's exactly what we are planning to do in a new bad rows https://discourse.snowplowanalytics.com/t/a-new-bad-row-format/2558

[szareiangm]

Hi @chuwy !

That's great. That stream of work will significantly improve our workflow.

However, about this small piece, the collector payload exists here but base64 encoded. Like no deep overhaul is needed. We have added the decoded value of the payload to test its usefulness and we believe we could share this with you guys, too. Not a lot of code change and so much benefit.

[chuwy]

Ah, @szareiangm, you mean that you'd like to implement it as a short-term solution before new bad rows arrived? I definitely don't oppose that and think this is a great idea (which we had to implement long time ago), but I doubt that we'll manage to do another ES Loader release before bad rows arrive.

Just in case, in new bad rows, this will be not a "collector payload", but "partially enriched event payload": snowplow/iglu-central@378f485#diff-e552d9f744e44db7e869d0771da0b645R277-R414, which preserves all data from the collector payload, but more structured.

[szareiangm]

Sure.
Then it means that I should not go forward with this. 👍