Load CDK: Stream Manager Support for Id-Based Checkpointing

[johnny-schmidt]

What

This is a nonfunctional change, as nothing is wired into the CDK pipeline yet.

This adds support for index-based checkpointing.

Currently we use range-based, which works like:

• each record has an associated index
• the indexes are aggregated into ranges for batches, then into rangesets in the stream manager
• when all contiguous rangesets up to record X have been persisted, state for up to that index can be evicted
• when all have been completed (and all the records have been read) then the stream can be closed

This causes some issues:

• managing rangesets is inefficient
• managing rangesets is error-prone

Changes for the new interface make bookkeeping a little more generic, which increases the costs of the above enough that they actually start having performance implications. (25% perf hit hacking around it in my POC PR).

This adds partial support for checkpointing by checkpoint id:

• each time we checkpoint a stream, we increment a monotonic id
• all records up to the checkpoint are associated with that id
• when we persist, we just count the persisted records against that id
• when we complete, we count completed records against that id

Sufficiency checks then are just

• persisted for id N: sum of all persisted counts up to id N == sum of all read counts up to id N
• completed: sum of all completed counts == total records read, and we've seen EOS

This is way easier than trying to manage rangesets, though we lose idempotence.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Feb 14, 2025 0:46am