Joy!!! But also questions...

[moeller0]

Hi team,
I just gave the current main branch a spin on my ubuntu host and discovered a number of new fancy features, like --relative-rpm and fine-grained controls over some parameters (also sattimeout^W rpmtimeout, was renamed again to rpm.timeout, I have a hunch that third time might be a charm :) )

Now here is what I get:
´´´
goresponsiveness: --quality-attenuation --relative-rpm
07-18-2023 14:29:54 UTC Go Responsiveness to mensura.cdn-apple.com:443...
Baseline RPM: 3222 (P90)
Baseline RPM: 3377 (Single-Sided 5% Trimmed Mean)
Download: 97.555 Mbps ( 12.194 MBps), using 9 parallel connections.
Extended Statistics:
Maximum Path MTU: 1492
Maximum Send MSS: 1208
Maximum Recv MSS: 1208
Total Retransmissions: 0
Total Reorderings: 27
Average RTT: 53881.22222222222

Quality Attenuation Statistics:
Number of losses: 0
Number of samples: 525
Loss: 0.000000 %
Min: 0.139002 s
Max: 2.062940 s
Mean: 0.437496 s
Variance: 0.128942 s
Standard Deviation: 0.359085 s
PDV(90): 0.908568 s
PDV(99): 1.569600 s
P(90): 1.047569 s
P(99): 1.708601 s
RPM: 137
Gaming QoO: 0
Download RPM: 97 (P90)
Download RPM: 526 (Single-Sided 5% Trimmed Mean)
Upload: 31.105 Mbps ( 3.888 MBps), using 16 parallel connections.
Extended Statistics:
Maximum Path MTU: 0
Maximum Send MSS: 0
Maximum Recv MSS: 0
Total Retransmissions: 0
Total Reorderings: 0
Average RTT: 0

Quality Attenuation Statistics:
Number of losses: 0
Number of samples: 637
Loss: 0.000000 %
Min: 0.016881 s
Max: 2.062940 s
Mean: 0.364577 s
Variance: 0.131210 s
Standard Deviation: 0.362229 s
PDV(90): 0.931082 s
PDV(99): 1.637709 s
P(90): 0.947963 s
P(99): 1.654590 s
RPM: 165
Gaming QoO: 0
Test did not run to stability, these results are estimates:
Upload RPM: 2392 (P90)
Upload RPM: 3331 (Single-Sided 5% Trimmed Mean)
Final RPM: 105 (P90)
Final RPM: 3234 (Single-Sided 5% Trimmed Mean)
Working Conditions RPM Effect: 187% (P90)
Working Conditions RPM Effect: 4% (Single-Sided 5% Trimmed Mean)

real 2m21.266s
user 0m9.308s
sys 0m5.263s
´´´
Let's focus on the P90 first:
´´´
Baseline RPM: 3222 (P90)
Final RPM: 105 (P90)
Working Conditions RPM Effect: 187% (P90)
´´´

When I calculate the relative effect from 3222 to 105 I get:
100*(105/3222) = 3.3% (with 100% denoting equality)
or
100*(105/3222) -100 = −96.74% points
or a ratio of
1/(3222/105) = 1/30.7 or 0.032588454
how am I supposed to interpret the reported 187%?

For the trimmed mean I get
´´´
Baseline RPM: 3377 (Single-Sided 5% Trimmed Mean)
Final RPM: 3234 (Single-Sided 5% Trimmed Mean)
Working Conditions RPM Effect: 4% (Single-Sided 5% Trimmed Mean)
´´´
When I calculate the relative effect i GET:
100*(3234/3377) = 95.8% (with 100% denoting equality)
or
100*(3234/3377) -100 = −4.2% points
or a ratio of
1/(3377/3234) = 1/1.04 or 0.957654723
how am I supposed to interpret the reported 4%?

Now here, this looks like it might report the change in percentage points (which does not correspond to a fixed ratio IIUC).

Don't get me wrong I absolutely love this new feature and think the baseline RPM numbers should become part of the IETF draft and part of the default output, I just want to understand how I am supposed to read the reported Effect size.

[hawkinsw]

Hi team, I just gave the current main branch a spin on my ubuntu host and discovered a number of new fancy features, like --relative-rpm and fine-grained controls over some parameters (also sattimeout^W rpmtimeout, was renamed again to rpm.timeout, I have a hunch that third time might be a charm :) )

I think that you are right -- I am much happier with the "rationalized" form of the test-related options being prefixed with rpm.. Do you agree? Any suggestions?

Now here is what I get: ´´´ goresponsiveness: --quality-attenuation --relative-rpm 07-18-2023 14:29:54 UTC Go Responsiveness to mensura.cdn-apple.com:443... Baseline RPM: 3222 (P90) Baseline RPM: 3377 (Single-Sided 5% Trimmed Mean) Download: 97.555 Mbps ( 12.194 MBps), using 9 parallel connections. Extended Statistics: Maximum Path MTU: 1492 Maximum Send MSS: 1208 Maximum Recv MSS: 1208 Total Retransmissions: 0 Total Reorderings: 27 Average RTT: 53881.22222222222

Quality Attenuation Statistics: Number of losses: 0 Number of samples: 525 Loss: 0.000000 % Min: 0.139002 s Max: 2.062940 s Mean: 0.437496 s Variance: 0.128942 s Standard Deviation: 0.359085 s PDV(90): 0.908568 s PDV(99): 1.569600 s P(90): 1.047569 s P(99): 1.708601 s RPM: 137 Gaming QoO: 0 Download RPM: 97 (P90) Download RPM: 526 (Single-Sided 5% Trimmed Mean) Upload: 31.105 Mbps ( 3.888 MBps), using 16 parallel connections. Extended Statistics: Maximum Path MTU: 0 Maximum Send MSS: 0 Maximum Recv MSS: 0 Total Retransmissions: 0 Total Reorderings: 0 Average RTT: 0

Quality Attenuation Statistics: Number of losses: 0 Number of samples: 637 Loss: 0.000000 % Min: 0.016881 s Max: 2.062940 s Mean: 0.364577 s Variance: 0.131210 s Standard Deviation: 0.362229 s PDV(90): 0.931082 s PDV(99): 1.637709 s P(90): 0.947963 s P(99): 1.654590 s RPM: 165 Gaming QoO: 0 Test did not run to stability, these results are estimates: Upload RPM: 2392 (P90) Upload RPM: 3331 (Single-Sided 5% Trimmed Mean) Final RPM: 105 (P90) Final RPM: 3234 (Single-Sided 5% Trimmed Mean) Working Conditions RPM Effect: 187% (P90) Working Conditions RPM Effect: 4% (Single-Sided 5% Trimmed Mean)

real 2m21.266s user 0m9.308s sys 0m5.263s ´´´ Let's focus on the P90 first: ´´´ Baseline RPM: 3222 (P90) Final RPM: 105 (P90) Working Conditions RPM Effect: 187% (P90) ´´´

When I calculate the relative effect from 3222 to 105 I get: 100*(105/3222) = 3.3% (with 100% denoting equality) or 100*(105/3222) -100 = −96.74% points or a ratio of 1/(3222/105) = 1/30.7 or 0.032588454 how am I supposed to interpret the reported 187%?

A great question! It is calculated as percent change -- but I am very flexible on what the choice should be:

$(\frac{old - new)}{new})*100$

As I said, I can easily change the formula! I was just brainstorming!

For the trimmed mean I get ´´´ Baseline RPM: 3377 (Single-Sided 5% Trimmed Mean) Final RPM: 3234 (Single-Sided 5% Trimmed Mean) Working Conditions RPM Effect: 4% (Single-Sided 5% Trimmed Mean) ´´´ When I calculate the relative effect i GET: 100*(3234/3377) = 95.8% (with 100% denoting equality) or 100*(3234/3377) -100 = −4.2% points or a ratio of 1/(3377/3234) = 1/1.04 or 0.957654723 how am I supposed to interpret the reported 4%?

Now here, this looks like it might report the change in percentage points (which does not correspond to a fixed ratio IIUC).

Don't get me wrong I absolutely love this new feature and think the baseline RPM numbers should become part of the IETF draft and part of the default output, I just want to understand how I am supposed to read the reported Effect size.

Thanks. It was not my idea (I honestly cannot even remember who suggested it) but I thought it was a good one.

[moeller0]

I think that you are right -- I am much happier with the "rationalized" form of the test-related options being prefixed with rpm.. Do you agree? Any suggestions?

Yes, this form lends itself naturally for additional parameters while retaining a clear indicator that these are fully optional. But then I generally like the dot as separator between items, so I am certainly biased.

A great question! It is calculated as percent change -- but I am very flexible on what the choice should be:
((old-new)/new) * 100
Mmmh, when I set new to be the 105 and old the 3222 I get:
((3222−105)÷105)×100 = 2968.571428571
or if I reverse the 'ages':
((105−3222)÷3222)×100 = −96.741154562
both of which seem not all that close to the reported number
So I guess I am still confused...

Regarding what to report, in vision we would call your number some form of Weber contrast (which uses feature for your 'old' and background for your 'new'). We could think about reporting a Michelson contrast instead:
(old - new) / (old + new)
if both old and new are > 0 this has the nice property of mapping the somewhat unwieldy range of -infinity->+infinity of ratios to the nice range of -1-> +1... However as end-user I probably would be happier with the pure ratio of working/baseline as that would be similar to the ratio that the underlying delay changes between the two conditions :)

I know RPM is after few simple numbers, but personally I always want to translate that back into how much does the absolute delay in the time domain change.

[hawkinsw]

I think that you are right -- I am much happier with the "rationalized" form of the test-related options being prefixed with rpm.. Do you agree? Any suggestions?

Yes, this form lends itself naturally for additional parameters while retaining a clear indicator that these are fully optional. But then I generally like the dot as separator between items, so I am certainly biased.

I love the . notation, too! I think it makes it look like OOP programming!

A great question! It is calculated as percent change -- but I am very flexible on what the choice should be:
((old-new)/new) * 100
Mmmh, when I set new to be the 105 and old the 3222 I get:
((3222−105)÷105)×100 = 2968.571428571
or if I reverse the 'ages':
((105−3222)÷3222)×100 = −96.741154562
both of which seem not all that close to the reported number
So I guess I am still confused...

Regarding what to report, in vision we would call your number some form of Weber contrast (which uses feature for your 'old' and background for your 'new'). We could think about reporting a Michelson contrast instead: (old - new) / (old + new) if both old and new are > 0 this has the nice property of mapping the somewhat unwieldy range of -infinity->+infinity of ratios to the nice range of -1-> +1... However as end-user I probably would be happier with the pure ratio of working/baseline as that would be similar to the ratio that the underlying delay changes between the two conditions :)

I thought about this all last night -- and I think that this is "above my pay grade". Let's call in @cpaasch and see what he has to say!

As always, thank you so much for the feedback!

I know RPM is after few simple numbers, but personally I always want to translate that back into how much does the absolute delay in the time domain change.