Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

benchmark: use plain text proxy. #22

Merged
merged 1 commit into from
Jan 29, 2025

Conversation

pvbouwel
Copy link
Contributor

In order to not have misleading benchmark outcomes as detailed in #21 (comment) we will benchmark with a plaintext proxy from now on. This will make the process slower (quite a bit for PutObject as it wouldn't need the extra pass on the body) but the results will be more intuitive because the way the S3 SDK client sends the data will be similar.

In order to not have misleading benchmark outcomes as detailed in VITObelgium#21 (comment) we will benchmark with a plaintext proxy from now on. This will make the process slower but the results will be more intuitive.
@pvbouwel
Copy link
Contributor Author

As shown in the benchmark report: https://github.com/VITObelgium/fakes3pp/actions/runs/13036429752/artifacts/2505676500

The HTTP proxy endures a penalty since the client calculates a payload hash:

FakeS3Proxy/S3ProxyBeforeFakeS3Backend-createRandomObject256MB-4                   939.9m ± 1%   1300.1m ± 1%  +38.32% (p=0.000 n=10)

As a result it is close (15 ms slower) to the benchmark of just the fake testing backend:

FakeS3Proxy/FakeS3Backend-createRandomObject256MB-4                                 1.279 ± 1%     1.285 ± 1%        ~ (p=0.063 n=10)

list bucket & getObject actions are in millisecond range and very close together. The proxy case for getObject reports faster than without but with the +- 1 % they overlap and the case without the proxy (which is exact the same run also shows a 5.08% increase so this observation can be attributed to benchmark variability)

list bucket performance  2.477m ± 1% (noproxy) <>2.849m ± 1% (proxy) show 
getObject performance 62.05m ± 1%  (noproxy) <> 61.06m ± 1% (proxy)

Overall I prefer this benchmark approach as the results are easier to interprete and don't require knowledge on difference in processing by the client side SDK depending on the endpoint uses TLS or not.

@pvbouwel pvbouwel merged commit 1e19ed7 into VITObelgium:main Jan 29, 2025
5 checks passed
@pvbouwel pvbouwel deleted the benchmark/use_plain_text_proxy branch January 29, 2025 18:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant