Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RUM-5553: Add DatadogMeter to read vital data for benchmark #2144

Merged
merged 1 commit into from
Jul 30, 2024

Conversation

ambushwork
Copy link
Member

What does this PR do?

Add DatadogMeter to read vital data for benchmark.

With this commit, the sample application can activate the DatadogMeter by adding this:

val datadogExporterConfiguration = com.datadog.benchmark.DatadogExporterConfiguration
            .Builder(apiKey = "xxxxxx")
            .setApplicationName("Sample Application")
            .setApplicationId(BuildConfig.APPLICATION_ID)
            .setApplicationVersion(BuildConfig.VERSION_NAME)
            .setIntervalInSeconds(20L)
            .setRun("baseline")
            .build()
 val meter = DatadogMeter.create(datadogExporterConfiguration)

meter.startGauges()

meter.stopGauges()

Motivation

RUM-5553

Additional Notes

Some vital readers classes and unit tests are copied from feature modules for following reasons:

  • This module should depend or use as less as possible the functionalities from SDK to get better result.
  • We might have different requirements of metrics apart from the vitals for client ( for example for the cpu vital we wish to have a cpu ticks timeseries instead of a cumulative value in production) , so isolate them from SDK is easier to maintain, and will have a risk of breaking production.
  • In the future we might make PR of this module to public opentelemetry-java repository, all the classes need to be independent from our SDK.

Anything else we should know when reviewing?

Review checklist (to be filled by reviewers)

  • Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
  • Make sure you discussed the feature or bugfix with the maintaining team in an Issue
  • Make sure each commit and the PR mention the Issue number (cf the CONTRIBUTING doc)

@ambushwork ambushwork requested review from a team as code owners July 25, 2024 16:05
@ambushwork ambushwork force-pushed the yl/add-datadog-meter branch from 19999f5 to c3b1923 Compare July 25, 2024 16:13
@codecov-commenter
Copy link

codecov-commenter commented Jul 25, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 69.97%. Comparing base (63e35ed) to head (acf70d2).
Report is 2 commits behind head on feature/metric-benchmark.

Additional details and impacted files
@@                     Coverage Diff                      @@
##           feature/metric-benchmark    #2144      +/-   ##
============================================================
+ Coverage                     69.94%   69.97%   +0.02%     
============================================================
  Files                           726      726              
  Lines                         26962    26962              
  Branches                       4519     4519              
============================================================
+ Hits                          18858    18864       +6     
+ Misses                         6832     6830       -2     
+ Partials                       1272     1268       -4     

see 22 files with indirect coverage changes

@ambushwork ambushwork force-pushed the yl/add-datadog-meter branch 3 times, most recently from 66b7c54 to 47dbd9c Compare July 26, 2024 07:18
@ambushwork ambushwork force-pushed the yl/add-datadog-meter branch from 47dbd9c to 1bd32a9 Compare July 29, 2024 07:40
@ambushwork ambushwork requested a review from xgouchet July 29, 2024 07:53
xgouchet
xgouchet previously approved these changes Jul 29, 2024
mariusc83
mariusc83 previously approved these changes Jul 29, 2024
@ambushwork ambushwork dismissed stale reviews from mariusc83 and xgouchet via 0589a5d July 29, 2024 11:54
@ambushwork ambushwork force-pushed the yl/add-datadog-meter branch from 1bd32a9 to 0589a5d Compare July 29, 2024 11:54
@ambushwork
Copy link
Member Author

@jonathanmos I just made the fix on the gauge as we discussed, please review it

jonathanmos
jonathanmos previously approved these changes Jul 30, 2024
@ambushwork ambushwork merged commit 3be8491 into feature/metric-benchmark Jul 30, 2024
19 checks passed
@ambushwork ambushwork deleted the yl/add-datadog-meter branch July 30, 2024 13:53
*/

@Suppress("TooGenericExceptionCaught", "SwallowedException")
private fun <T> File.safeCall(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need these methods? they are originally were developed for the SDK usage, so that exception is not propagated to the client code. But if I get it right, this benchmark tooling is just internal tooling? So we probably don't care?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed, the whole file will be removed

lastFrameTime = currentFrameTime
frameCount = 0
}
Choreographer.getInstance().postFrameCallback(this)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why Choreographer is chosen over what androidx.metrics gives?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't know androidx.metrics can do the same thing, is there any advantage of android.metrics comparing to Choreographer?

Copy link
Member

@0xnm 0xnm Aug 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, a lot of advantages: it is a wrapper around different new APIs allowing collecting more granular metrics. Also by registering Choreographer callback and not un-registering it when app goes into background/losing focus, you make CPU running more (this point probably doesn't matter a lot for internal tooling).

Anyway, FPS is not a good metric to judge about the performance and we were discussion to ditch it and replace it with something else (especially considering dynamic refresh rate). Much more meaningful metric will be something like the % of skipped/junky frames, for example.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good to know, I will first make a pull request to clean up the code on all the places where you mentioned, and another dedicated commit to replace the Choreographer

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall maybe for this work we shouldn't use FPS as a metric at all. But this is just my opinion.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe you are right, with current synthetics test and metrics result, we can hardly see any difference between baseline and instrumented test, I will try to see if androidx.metrics can provide some more valuable result.

private var lastFrameTime: Long = 0
private val intervalMs = FPS_SAMPLE_INTERVAL_IN_MS

private val frameCallback = object : Choreographer.FrameCallback {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same question: why Choreographer over androidx.metrics?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants