Proposal: Hooking into yunikorn-core event system would allow snapshots of the object state with small performance degregation of the scheduler

[nikola-jokic]

Problem statement

Yunikorn documentation describes the purpose of the event system, its challenges and limitations. Since the purpose of this project is to preserve historical data and keep object states, the event system is missing crucial information about the application lifecycle, such as allocation tags, etc.

Partial information propagated through the event stream is not rich enough to preserve the actual historical data that was stored in Yunikorn.

On the other hand, using API to periodically snapshot the queue state would also be incomplete since objects are deleted as soon as the scheduler is done with them. Another drawback of this approach is that frequent requests would cause degradation in the scheduler's performance.

Proposal

This problem would be completely solved if the event system, along with the current data, sends the snapshot of the object inside the event field. This approach would effectively be the same as using in-memory object cache for scheduler operations, while asynchronously persisting the data through the event stream.

Here is the list of proposed changes that could be added to the Yunikorn to make it work:

1. Add a configuration flag that signals whether the event system should be extended or not. By default, it shouldn't be extended (preserve the current behavior of Yunikorn)
2. Each object is instantiated with that flag, so whenever that object raises the event, it can optionally snapshot its state or not.
3. Extend each object with the private method that either snapshots the object state (in case extended events are configured), or returns nil if an object is not configured with the extended event system
4. Extend the EventRecord to contain the state field of type any. Nil by default. If Yunikorn is configured without the extension, state would be nil and no additional data would be sent through the wire. On the other hand, if the extension is configured, the state would be the snapshot of the object.
5. Extend the event interface for each object to accept the state in its method invocation, so it can build up the event with optional state data.

Pros:

• On each state change, we would get the actual object with its current state.
• Minimal performance hit calling the new event system if the extension is not configured
• Users have to opt-in for this to work. If users do opt-in, they are okay with the performance hit from this approach
• Relatively easy to maintain. It requires a few additional lines before passing the data to the event system.
• Backwards compatible.

Cons

• This approach would require the maintainers of Yunikorn to accept this proposal
• Would slightly impact the performance of the Yunikorn scheduler when the event system is not extended
• Requires changes to remove events. Methods should get the application instead of just testing for its existence.

Conclusion

Having changes to the event stream, to push the actual data would allow us to persist states and transitions. This proposal outlines the potential approach that can be taken. We are open to suggestions and modifications of this approach. ☺️

[dave-gantenbein]

Is it feasible to tweak this to make the changes even less intrusive upstream by allowing configuration at build time to pull in our alternate event stream implementation instead of requiring this code to live upstream?

[nikola-jokic]

As far as I know, there is no way. The scheduler object raises the event on operation. The event doesn't have any snapshots of the object we could use to swap the event stream to our implementation provided in the build time (example).