Async programming model in kube-rs and potential reordering between requests

[marshtompsxd]

kube-rs uses async functions widely and every API request method (e.g., create, get) is async. This means that there is a chance the ordering of API requests taking effect at the k8s API differs from the ordering of the controller issuing them. I did some experiments to check when the reordering could happen. Major findings are as follows:

(1) kube-rs itself does not try to reorder any such request sent to the k8s API. kube-rs indeed maintains some queues of notifications from the API server but those are for triggering reconciliation with watch notifications. The request method simply serializes the request content into bytes, sends an HTTP request, and deserializes the response.

(2) immediately calling .await for each async request method makes the requests purely sequential. This is not surprising at all since .await should block until the result (response) is available of the request. When the next request method is invoked, the previous one (invoked with .await) already takes effect.

(3) the ordering of API request taking effect purely relies on the ordering of calling .await. This is a bit surprising to me. Consider the following code snippet:

let f1 = async_request_1();
let f2 = async_request_2();
f1.await(); 
f2.await();

f1 will not start to run until you call f1.await (similar to f2). This is a bit different from some other programming languages where the async function starts to run as soon as you invoke it (even without .await). This is because Rust makes async functions lazy. See the documentation https://rust-lang.github.io/async-book/06_multiple_futures/02_join.html ("However, Rust futures won't do any work until they're actively .awaited."). In other words, in this example there is no reordering between f1 and f2.

(4) reordering could happen with join!. join! takes multiple async functions and concurrently poll them. I did an experiment by invoking a create pod and a get pod request and join! the two requests as follows:

let create_f = async_create_request();
let get_f = async_get_request();
join!(create_f, get_f);

The get request always returns "pod not found" error because when it hits the API server the create request is not done yet.