Is it possible to update the metrics timestamp from the receiver for splunk_hec exporter?

[sandeshrk]

Here is the setup:

OTEL Metrics parser script ==> OTLP:4317/4318 ==> OTEL collector ==> Splunk_HEC ==> Splunk Enterprise

The script updates a counter with a custom attribute for time (custom_time in epoch). I have not been able to update the _time attribute (metadata) of Splunk Enterprise. The _time attribute is always the ingestion time even if the custom_time is a few months in the past.

Am I missing something very basic or is this something that's not supported for metrics?

Script:

import time
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry.metrics import set_meter_provider, get_meter_provider
from opentelemetry.sdk.resources import Resource

# Configure the meter provider and exporter
otlp_exporter = OTLPMetricExporter(
    endpoint="localhost:4317",  # Default endpoint for OTLP GRPC
    headers={},  # Optional: You can add headers for authentication or other needs
    insecure=True
)
resource = Resource.create({"service.name": "test-service"})
reader = PeriodicExportingMetricReader(otlp_exporter)
provider = MeterProvider(resource=resource, metric_readers=[reader])
set_meter_provider(provider)

# Create a meter and instrument
meter = get_meter_provider().get_meter("test-app")
op_counter = meter.create_counter(name="total_test_requests", description="The total number of operations performed", unit="1")

def sim():
    # Record some measurements
    for i in range(5):
        op_counter.add(1, attributes={"custom_time": 1707156001})
        print("HELLO")

if __name__ == "__main__":
    print("Starting to simulate operations...")
    sim()
    print("Metrics exported to OTLP endpoint.")

OTEL collector config

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

exporters:
    debug:
      verbosity: detailed
    file:
      path: /log1/file.log
    splunk_hec/metrics:
        # Splunk HTTP Event Collector token.
        token: "00000000-0000-0000-0000-0000000000000"
        # URL to a Splunk instance to send data to.
        endpoint: "https://splunk:8088/services/collector"
        # Optional Splunk source: https://docs.splunk.com/Splexicon:Source
        source: "app"
        # Optional Splunk source type: https://docs.splunk.com/Splexicon:Sourcetype
        sourcetype: "test-otel"
        # Splunk index, optional name of the Splunk index targeted.
        index: "metrics"
        # Maximum HTTP connections to use simultaneously when sending data. Defaults to 100.
        #max_connections: 20
        # Whether to disable gzip compression over HTTP. Defaults to false.
        disable_compression: false
        # HTTP timeout when sending data. Defaults to 10s.
        timeout: 10s
        tls:
          # Whether to skip checking the certificate of the HEC endpoint when sending data over HTTPS. Defaults to false.
          # For this demo, we use a self-signed certificate on the Splunk docker instance, so this flag is set to true.
          insecure_skip_verify: true
processors:
  batch:
  attributes:
    actions:
      - key: _time
        action: upsert
        from_attribute: custom_time
      - key: time
        action: upsert
        from_attribute: custom_time
      - key: timestamp
        action: upsert
        from_attribute: custom_time

service:
    pipelines:
      metrics:
        receivers: [otlp]
        processors: [attributes]
        exporters: [debug,file,splunk_hec/metrics]

[atoulme]

_time is a Splunk specific field, and that's not how we typically manipulate time with OpenTelemetry. See the transform processor to manipulate the timestamp of metric data points.

[sandeshrk]

Thanks @atoulme. My mistake.
This transform processor for metrics datapoint updated the time.
processors:
batch:
transform:
metric_statements:
- context: datapoint
statements:
- set(start_time_unix_nano, attributes["custom_time"])
- set(time_unix_nano, attributes["custom_time"])
attributes:
actions:
- key: custom_time
action: delete

Let me know if this is what you recommended.