Custom StackDriver Monitoring Metrics on Dataproc

[shsunil11]

Please answer these questions before submitting a bug report.

What version of OpenCensus are you using?

0.19.2

What JVM are you using (java -version)?

1.8.0._171

What did you do?

If possible, provide a recipe for reproducing the error.

Tested the example code on a Dataproc cluster master node.
https://github.com/GoogleCloudPlatform/java-docs-samples/blob/master/opencensus/src/main/java/com/example/opencensus/Quickstart.java

The Dataproc Cluster had a service account tied to it that had the Project Editor Role.
The scopes used for the Dataproc Cluster was 'https://www.googleapis.com/auth/cloud-platform'

Also tried to use the project_id in the builder as described in
https://github.com/census-instrumentation/opencensus-java/tree/master/exporters/stats/stackdriver

StackdriverStatsExporter.createAndRegister(
StackdriverStatsConfiguration.builder().setProjectId("MyStackdriverProjectId").build());

What did you expect to see?

Expected to see the custom metrics in the Stackdriver Monitoring Workspace in the Metrics Explorer under resource type Global

What did you see instead?

The custom metrics did not show up in the Metrics Explorer

[songy23]

The issue here is main thread exited before data got exported. The QuickStart used the default export interval which is 60s while main thread also waited 60s before exiting. This is a bit risky since it took a bit time to upload metrics to Stackdriver.

Two quick way to make the sample work:

1. Make the main thread wait a bit longer, e.g change EXPORT_INTERVAL to 65s or greater.
2. Configure the Stackdriver exporter to export more frequently by using StackdriverStatsConfiguration. For example StackdriverStatsConfiguration.builder().setExportInterval(TEN_SECONDS).build(). Note in real production Stackdriver recommended the export interval to be at least 1 minute.

[yzhou2001]

It works now with the 60->70 seconds change

[songy23]

Thanks @yzhou2001 for the confirmation. I'm going to mark this as resolved for now, please feel free to follow up if you've seen other issues.

[ocervell]

@shsunil11 nice to see it's finally working !
@songy23 can we document this export interval ? sounds like a gotcha

[yzhou2001]

While the sleep solves the issue, I'm wondering whether there is a "flushing" mechanism to force out the collected data to the StackDriver backend. As for applications, it's often undesirable to be forced to sleep a period, in particular for low-latency and/or chained apps.

[yzhou2001]

One more clarification: the resource type is not "Global" but "GCE VM Instance" somehow.

[songy23]

can we document this export interval ? sounds like a gotcha

Yes, we'll add this to opencensus.io. Tracked in census-instrumentation/opencensus-website#652.

I'm wondering whether there is a "flushing" mechanism to force out the collected data to the StackDriver backend. As for applications, it's often undesirable to be forced to sleep a period, in particular for low-latency and/or chained apps.

Agree. There's a spec issue on supporting this: census-instrumentation/opencensus-specs#152.

One more clarification: the resource type is not "Global" but "GCE VM Instance" somehow.

If the application and exporter is running on a GCE instance, this will be captured by the resource detection util and the metrics uploaded to Stackdriver will have a GCE instance resource associated. See more details at https://github.com/census-instrumentation/opencensus-java/tree/master/exporters/stats/stackdriver#set-monitored-resource-for-exporter.