Unable to Export Metrics Events from node.js from Collector to Dataprepper

[shyam1975]

Describe the bug
We have a simple .js file called through node as follows :-

'use strict';

const { DiagConsoleLogger, DiagLogLevel, diag } = require('@opentelemetry/api');
const { CollectorMetricExporter } = require('@opentelemetry/exporter-collector');
// const { CollectorMetricExporter } = require('@opentelemetry/exporter-collector-grpc');
// const { CollectorMetricExporter } = require('@opentelemetry/exporter-collector-proto');
const { MeterProvider } = require('@opentelemetry/metrics');

// Optional and only needed to see the internal diagnostic logging (during development)
diag.setLogger(new DiagConsoleLogger(), DiagLogLevel.DEBUG);

const metricExporter = new CollectorMetricExporter({
serviceName: 'basic-metric-service',
url: 'http://localhost:55681/v1/metrics',
//url: 'http://localhost:55681',
});

const meter = new MeterProvider({
exporter: metricExporter,
interval: 1000,
}).getMeter('example-exporter-collector');

const requestCounter = meter.createCounter('requests', {
description: 'Example of a Counter',
});

const upDownCounter = meter.createUpDownCounter('test_up_down_counter', {
description: 'Example of a UpDownCounter',
});

const recorder = meter.createValueRecorder('test_value_recorder', {
description: 'Example of a ValueRecorder',
});

const labels = { pid: process.pid, environment: 'staging' };

setInterval(() => {
requestCounter.bind(labels).add(1);
upDownCounter.bind(labels).add(Math.random() > 0.5 ? 1 : -1);
recorder.bind(labels).record(Math.random());
}, 1000);

Metrics reach the collector but the collector is unable to communicate with the dataprepper. The following error is observed on the collector console :-

2021-04-23T22:02:38.812+0800 INFO loggingexporter/logging_exporter.go:375 MetricsExporter {"#metrics": 3}
2021-04-23T22:02:38.815+0800 ERROR exporterhelper/queued_retry.go:240 Exporting failed. The error is not retryable. Dropping data. {"component_kind": "exporter", "component_type": "otlp", "component_name": "otlp/2", "error": "failed to push metrics data via OTLP exporter: Permanent error: rpc error: code = Unimplemented desc = Not Found: HTTP status code 404; transport: received the unexpected content-type "text/plain; charset=utf-8"", "dropped_items": 3}

**Environment

• OS: Ubuntu 20.04.2 LTS
• Collector Version aws-otel-collector-0.8.0
• Dataprepper Version 0.8.0-beta

[shyam1975]

config files for the otel collector and the dataprepper are attached.

trace_analytics_no_ssl.yml.txt
otel-collector-config.yml.txt

[chenqi0805]

@shyam1975 Thank you for the interest in Data-Prepper and the report on your use case. By examining your shared otel-collector-config.yml.txt. I noticed

metrics:
      receivers: [otlp]
      exporters: [logging, otlp/2]

Unfortunately, we have not yet supported exporting metrics from otel-collector to data-prepper and thus the Unimplemented error code you have seen in the collector console. I suggest removing otlp/2 from the metrics exporters to see if it works fine for traces:

metrics:
      receivers: [otlp]
      exporters: [logging]

BTW, did you refer to any of our documentation for making the otel-collector-config.yml you shared? Just want to make sure our docs have not misled you to the error.

[contractormarketplace]

Thank you @chenqi0805 for your inputs. I will remove the metrics exporters to see if it works fine for traces.

While coming up with otel-collector-config.yml, I did NOT refer to any specific documentation. We use this collector config for collecting traces and metrics from our Java apps and I tried re-using the same configuration for our testing. We have a requirement whereby we need to have the same instance of the collector collecting for multiple languages including Java, C++, Node based js.

[contractormarketplace]

@chenqi0805 are there any plans for the data prepper to process metrics in the future?

[chenqi0805]

@contractormarketplace Sorry I missed your latest reply. Yes, we can definitely put this on the radar once we dive deeper into cx use case of metrics. I will open an issue to keep track of the feature request.