Decoupled Async Execute

[kthui]

Related PR: triton-inference-server/python_backend#350

Tests for Python backend decoupled async execute function. Covered cases:

• A async execute function may start multiple coroutines and those are ran concurrently.
• Multiple async execute coroutines are ran concurrently.
• An async execute coroutine started late but finished early will respond before another async execute coroutine that started early but finished late.
• A coroutine may raise an exception and it will be logged.

[Tabrizian]

Nice testing. Could you also profile this model with perf analyzer with high concurrencies just to make sure everything works fine?

[kthui]

Nice testing. Could you also profile this model with perf analyzer with high concurrencies just to make sure everything works fine?

Yes, the perf_analyzer run on "async_execute_decouple" model works great

# perf_analyzer -i grpc -m async_execute_decouple --async --streaming true -b 8 --concurrency-range 128:128 --input-data zero
*** Measurement Settings ***
  Batch size: 8
  Service Kind: Triton
  Using "time_windows" mode for stabilization
  Measurement window: 5000 msec
  Latency limit: 0 msec
  Concurrency limit: 128 concurrent requests
  Using asynchronous calls for inference
  Detected decoupled model, using the first response for measuring latency
  Stabilizing using average latency

Request concurrency: 128
  Client: 
    Request count: 44624
    Throughput: 19743.8 infer/sec
    Response Throughput: 2467.97 infer/sec
    Avg latency: 51762 usec (standard deviation 4416 usec)
    p50 latency: 53131 usec
    p90 latency: 54035 usec
    p95 latency: 54909 usec
    p99 latency: 55042 usec
    
  Server: 
    Inference count: 357016
    Execution count: 44627
    Successful request count: 44627
    Avg request latency: 49176 usec (overhead 1 usec + queue 48797 usec + compute input 25 usec + compute infer 349 usec + compute output 3 usec)

Inferences/Second vs. Client Average Batch Latency
Concurrency: 128, throughput: 19743.8 infer/sec, latency 51762 usec