Change feed (pull model) behavior on physical partition split?

[JensSchadron]

Hii,

I've been experimenting with change feed (pull model) with continuation tokens, because it's scoped to a select few logical partitions. But now that I've a working implementation, I'm wondering what the behavior is for the change feed in case of a physical partition split?

• Does the existing continuation token keep on working or will it become invalid?
• In case of the later, how can I renew the token on-the-fly, as I seemingly can't start a full fidelity (AllVersionsAndDeletions) change feed at a point in time in the past (regardless of whether it was in the continuous backup retention period)?

Note:
The only similar discussion that I could find was this one (#2216), but there was not really a clear-cut answer in there.

[Pilchie]

@jcocchi ?

[jcocchi]

@JensSchadron continuation tokens for the change feed pull model will continue to work post partition split, and the iterator processing the continuation token will receive changes from both child partitions after the split. The continuation token is created for the FeedRange defined when creating the FeedIteraror and will continue working for that range regardless of the underlying partition layout.

Let's imagine there's initially two physical partitions in your container and you create two FeedIterators to process changes from them. The FeedRange for your first iterator covers [A, M] which represents the first physical partition. If that physical partition splits, you now will have two child partitions- let's say their ranges are [A, F] and [G, M]. The original continuation token will now be covering two physical partitions but still is valid for the full initial range [A, M].

Where the change feed processor automatically distributes work for consuming changes from new partitions via leases, the change feed pull model requires you to handle scaling yourself. There's nothing wrong with having one iterator consume changes from multiple physical partitions, but it will be less efficient.

For one time or intermittent processing where few partition splits are expected, change feed pull model is a good choice. For ongoing processing where dynamic scaling is required, change feed processor is the better method.

Regarding your second question- AllVersionsAndDeletes change feed doesn't support specifying a specific start time yet, however this is on the roadmap and expected later this year. The start time will need to be in the continuous backup retention window.

[JensSchadron]

Thank you kindly for addressing my concerns and a happy new year to you all 🥳