Send event to Sentry when event can't be inserted to ES good index

[benjben]

On top of a bad row being emitted, we want to be notified when this happens to detect ASAP ES mapping failures with union types.

[jbeemster]

Worth windowing this a bit - on a large pipeline this could be thousands of events failing! You might need some form of internal counter which flushes to Sentry every N minutes to protect the Sentry server from accidental DDoS.

[benjben]

You're right. We're sending events to ES per batch, so we can send only one notification to Sentry per batch.

[jbeemster]

Even per batch it could be a lot - the batch is determined by the sink parameters (time, bytes, record count) if I understand correctly. If your time-limit is say 100ms then you could get 10 failures per second coming to Sentry (extrapolating out to 864,000 per day at that rate if consistent). This is obviously a very pessimistic view but could eventuate...

[benjben]

Do we pay for a maximum number of events sent to Sentry ?

[jbeemster]

No - but we will need to scale up Sentry enormously if we send huge volume to it :)

[benjben]

Oh we are self hosting it ! I had missed this bit !