ONNX Resnet50 int8 performance

[zjgarvey]

The performance for resnet50 is quite poor on Mi300 (our int8 inference time with batch size 1 is around 4.7ms). Here is the mlir and tracy profile for sharing/discussion.

MLIR (torch-onnx)

Can use wget, azure cli, or direct download here (less than 50MB).

wget https://onnxstorage.blob.core.windows.net/onnxstorage/e2eshark/mlir_share/resnet50_vaiq.mlir

MI300 Tracy profile

I compiled with:

iree-compile --iree-hal-target-backends=rocm --iree-hip-target=gfx942 resnet50_vaiq.mlir -o resnet50_vaiq.vmfb

Then ran:

IREE_PY_RUNTIME=tracy TRACY_NO_EXIT=1 iree-run-module \
  --device=hip://1 \
  --module=resnet50_vaiq.vmfb \
  --function='main_graph' \
  --input='1x3x224x224xf32=@/home/zagarvey/SHARK-TestSuite/alt_e2eshark/test-run/resnet50_vaiq/input.0.bin'

And captured with

iree-tracy-capture -o resnet50_vaiq_gfx942.tracy

you can get this profile from https://onnxstorage.blob.core.windows.net/onnxstorage/e2eshark/mlir_share/resnet50_vaiq_gfx942.tracy or wget or azure cli:

wget https://onnxstorage.blob.core.windows.net/onnxstorage/e2eshark/mlir_share/resnet50_vaiq_gfx942.tracy

[ScottTodd]

Can you add this flag when compiling so we can see executable sources in the trace?

--iree-hal-executable-debug-level=3

(https://iree.dev/developers/performance/profiling-with-tracy/#quickstart)

Might also want to run through iree-benchmark-module to get a few iterations in.

Time scales are pretty small here, so overhead (and lack of some fusions) may be a larger factor than in other programs: