IOByte computation in benchmarks

[Priya2698]

We currently use the input-ouputs consumed by nvfuser definitions as reference for the IOBytes computation for all executors.
This has certain limitations:

1. Requires manual effort to identify the reference IOBytes from nvfuser definitions when adding Thunder-nvfuser benchmarks (PR rope_benchmark #3550)
2. There is also the possibility of the IOBytes being different between executors (torch.compile, eager and Thunder).

[naoyam]

There is also the possibility of the IOBytes being different between executors (torch.compile, eager and Thunder).

Why is this?

[Priya2698]

There is also the possibility of the IOBytes being different between executors (torch.compile, eager and Thunder).

Why is this?

If the executors save different variables for backward pass or choose to rematerialize any intermediate variables, that strategy may differ across executors, particularly for larger fusions. I don't think we see this in our current benchmarks though.

[naoyam]

There is also the possibility of the IOBytes being different between executors (torch.compile, eager and Thunder).

Why is this?

If the executors save different variables for backward pass or choose to rematerialize any intermediate variables, that strategy may differ across executors, particularly for larger fusions. I don't think we see this in our current benchmarks though.

If that's the case, I wonder if it still make sense to compare the performances between the backends because they don't seem to compare apples to apples.

Can this happen only between the torch.compile and thunder backends but not between the thunder-torch.compile and thunder backends?