CPU Offload for experts in Runtime

[mryab]

Given that we're only requesting one expert in the server at a time, it might be possible to keep many experts in CPU memory and to process larger batches in a single step as a result. Since we know the upcoming request queue, we can also prefetch the experts that are going to be required to minimize the latency.

[justheuristic]

As a side-note: we could also probably load experts into multiple GPUs and process them concurrently.

[justheuristic]

TIL: torch has a great mechanism for asynchronous offloading called torch.cuda.stream

Here's an example of how it works:
https://github.com/facebookresearch/fairscale/blob/8d82db43eca3c6d88f02c60bce5ba80177d2cf12/fairscale/experimental/nn/offload.py#L128-L129