Create benchmark harness using a stubbed HTTP client

[aemous]

Description of changes:

• Adds a script that runs and benchmarks AWS CLI commands in a configurable environment with a stubbed HTTP client.
  • The following benchmarks are collected:
    • Average memory/CPU usage (over samples of each iteration)
    • p50, p95, p100 memory/CPU (over samples of each iteration)
    • Total execution time
    • Time to first operation invocation
• Adds a default JSON file containing benchmark definitions of high-level S3 commands.

Description of tests:

• Ran the script with the following arguments and produced the output below.
  `./scripts/performance/run-benchmarks --result-dir ~/Desktop/results --num-iterations 1

{
  "results": [
    {
      "name": "s3.cp.upload",
      "dimensions": [
        {
          "FileSize": "32MB"
        },
        {
          "S3TransferClient": "Classic"
        }
      ],
      "measurements": [
        {
          "average_memory": 53263676.77697842,
          "average_cpu": 2.137410071942445,
          "max_memory": 76808192,
          "max_cpu": 6.1,
          "memory_p50": 51855360,
          "memory_p95": 75628544,
          "cpu_p50": 2.1,
          "cpu_p95": 2.6,
          "total_time": 0.1773359775543213,
          "first_client_invocation_time": 0.1729750633239746
        }
      ]
    },
    {
      "name": "s3.cp.upload",
      "dimensions": [
        {
          "FileSize": "32MB"
        },
        {
          "S3TransferClient": "CRT"
        }
      ],
      "measurements": [
        {
          "average_memory": 86294059.88571429,
          "average_cpu": 1.206476190476189,
          "max_memory": 80363520,
          "max_cpu": 5.8,
          "memory_p50": 78004224,
          "memory_p95": 80150528,
          "cpu_p50": 0.0,
          "cpu_p95": 2.4,
          "total_time": 0.7082681655883789,
          "first_client_invocation_time": 0.17952513694763184
        }
      ]
    },
    {
      "name": "s3.mv.upload",
      "dimensions": [
        {
          "FileSize": "32MB"
        }
      ],
      "measurements": [
        {
          "average_memory": 344560626.7012987,
          "average_cpu": 6.244805194805187,
          "max_memory": 74711040,
          "max_cpu": 7.8,
          "memory_p50": 51216384,
          "memory_p95": 73482240,
          "cpu_p50": 2.1,
          "cpu_p95": 2.6,
          "total_time": 0.19771814346313477,
          "first_client_invocation_time": 0.19340896606445312
        }
      ]
    },
    {
      "name": "s3.mv.download",
      "dimensions": [
        {
          "FileSize": "32MB"
        },
        {
          "S3TransferClient": "Classic"
        }
      ],
      "measurements": [
        {
          "average_memory": 446090808.8888889,
          "average_cpu": 9.250370370370373,
          "max_memory": 76251136,
          "max_cpu": 3.7,
          "memory_p50": 51871744,
          "memory_p95": 75202560,
          "cpu_p50": 2.1,
          "cpu_p95": 2.5,
          "total_time": 0.17141294479370117,
          "first_client_invocation_time": 0.16584086418151855
        }
      ]
    },
    {
      "name": "s3.sync.upload",
      "dimensions": [
        {
          "FileCount": "5,000"
        },
        {
          "FileSize": "4KB"
        },
        {
          "S3TransferClient": "Classic"
        }
      ],
      "measurements": [
        {
          "average_memory": 112169603.30997425,
          "average_cpu": 2.481540355983502,
          "max_memory": 134987776,
          "max_cpu": 18.9,
          "memory_p50": 106889216,
          "memory_p95": 132530176,
          "cpu_p50": 2.4,
          "cpu_p95": 2.7,
          "total_time": 11.36503291130066,
          "first_client_invocation_time": 0.6536939144134521
        }
      ]
    },
    {
      "name": "s3.sync.upload",
      "dimensions": [
        {
          "FileCount": "5,000"
        },
        {
          "FileSize": "4KB"
        },
        {
          "S3TransferClient": "CRT"
        }
      ],
      "measurements": [
        {
          "average_memory": 156220381.4580351,
          "average_cpu": 1.0487899990370957,
          "max_memory": 189431808,
          "max_cpu": 22.8,
          "memory_p50": 140099584,
          "memory_p95": 185647104,
          "cpu_p50": 0.0,
          "cpu_p95": 3.5,
          "total_time": 80.57338333129883,
          "first_client_invocation_time": 0.6558301448822021
        }
      ]
    }
  ]
}

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[ashovlin]

1. Can you add a README to the scripts/performance directory covering:
   1. How to run the current benchmarks
   2. How to add a new benchmark, including a definition for each of the JSON keys (a JSON schema would be nice too, but okay to skip for now).
2. What is the plan for the existing S3 benchmark scripts? If this has feature parity, can we delete them?

[aemous]

1. Can you add a README to the scripts/performance directory covering:

   1. How to run the current benchmarks
   2. How to add a new benchmark, including a definition for each of the JSON keys (a JSON schema would be nice too, but okay to skip for now).

2. What is the plan for the existing S3 benchmark scripts? If this has feature parity, can we delete them?

@ashovlin

1. Sure I will have this added in my next revision.
2. Good point. All benchmarks in this PR cover everything in the existing scripts (barring that the existing scripts integrate with the S3 service). Since we want to focus on stubbed-client performance for now, I will delete these scripts in the next revision.

[kdaily]

Can you describe the choice of the interface for this? It looks like its similar to a unittest.TestCase. Why use this instead of a standard __init__?

[aemous]

I wanted the client object to be reusable across different iterations / command benchmarks. Using setup and teardown allows us to setup and reset the state of the client in between each invocation.

In particular, it prevents the case where the user supplies more stubbed responses than there are actual requests made, leaving the responses list non-empty, and thus the next execution would reuse stale stubs from the previous execution.

Alternatively I'd be reinstantiating a new client for each iteration/execution, which seems redundant.

[kdaily]

nit: Could you use csv library here? That might help to make sure the data file has descriptive headers. The README should have an accompanying doc/data dictionary to describe the values.

[aemous]

The original scripts used csv files for writing / reading, presumably since s3transfer commands can run for quite a while and not having to load all the samples collected into memory.

In the next revision, I ended up removing the file I/O for benchmark and summarizing entirely, since we should just load all samples into memory anyways to compute percentiles. Should also save some disk operation overhead.

I have not observed the number of samples exceed 9,000 in my testing.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Please upload report for BASE (benchmarks@14e84b6). Learn more about missing BASE report.

Additional details and impacted files

@@             Coverage Diff              @@
##             benchmarks   #9239   +/-   ##
============================================
  Coverage              ?       0           
============================================
  Files                 ?       0           
  Lines                 ?       0           
  Branches              ?       0           
============================================
  Hits                  ?       0           
  Misses                ?       0           
  Partials              ?       0           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.