Mute CQEs of send/write to reduce wakeups

[pyhd]

wait_timeout(nr) is generally a good way to reduce wakeups from kernel, while CQEs of send/write can bring unnecessary "noise", especially from plenty of zero-copy. In essence, it is difficult to estimate when send/write will return, yet their CQEs are generally not latency sensitive. So I think a possible solution is to flag MUTE_SUCCESS in the SQE, then its CQE will not be counted as wakeable.

if (sq_ready) {
    submit_and_wait_timeout(nr, 1ms);
} else {
    if (inflight_sends)
        wait_timeout(1, 100ms);  // even if no wakeup CQEs, muted CQEs will still be reaped in a poll way.
    else
        wait(1); // if no pending send/write CQEs
}

[axboe]

Yep this is not a bad idea, we've bounced around ideas for this very thing in the past as well. Send is a good example - generally they complete inline (eg immediatley), but it's not guaranteed. And while you don't need an immediate notification for them, generally you do want to see one so that you know the data it sent can get reused. Hence IOSQE_CQE_SKIP_SUCCESS isn't really useful for this case.

I think what we'd need is something like a low priority completion, in the sense that it doesn't need to wakeup the task waiting, but it should be included in the "I'm waiting for this number of events" accounting.

A quick work-around with the existing code may be to just discount the write/send in the wait_nr.