Incomplete tasks due service crash

[nuclearcat]

Often might happen that service crash due various reasons, for example pipeline-tarball might try to use more memory than available at node, then it will restart and we might have checkout node finished and stay in incomplete state.
We need a way to retry last job that was being executed, if it is possible.

Proposal: to do this we need mechanism that will take event, set this event as "current" of service, and on event completion - delete this state. In case service restarted, it will check if there was event that was being processed and if yes, it will retry to process it again.

Question is:
Where to store event? Persistent storage on kubernetes? some additional API endpoint?

Any other suggestions?