ADOT Collector as AWS managed Service

[niko-achilles]

Is your feature request related to a problem? Please describe.
Managed services as observable systems.
Given managed services are minimizing the effort to create production ready applications and the development focus is on delivering business value , there are some technical barriers and performance trade offs that application developers should consider in order to have observable systems.

Technically what the developers have as tools today in context of observability with opentelemetry is a Layer deployment artifact which consist of

• internal extension, for initializing the opentelemetry mechanics (SDK) and modifying the runtime
• external extension, which collects, processes and exports telemetry data - ADOT Collector

In a serverless platform, λ lambda developers have the execution runtime specification and APIs to adopt the lambda behavior to their special needs.
The runtime execution environment and APIs leverage the choreography approach with signals and the orchestration approach with order of extensions registration.

From the deployment perspective, the artifact which consists of internal extension and external extension in context of lambda and serverless , the deployment method is distributed . Each lambda needs its own deployment artifact.

Describe the solution you'd like
With this open issue , I would like to introduce by the deployment perspective to leverage a centralized component which has the responsibility to collect telemetry data of my lambda applications, processes the telemetry data by configuration and exports the telemetry data to desired destinations .
From the technical perspective , developers will have the ability to better use extensions and control the execution environment of lambda λ.

An idea can be the ADOT Collector as managed service by AWS.

Given the Collector of Opentelemetry defines:

• Receivers
• Processors
• Exporters

a similar AWS managed service by AWS can provide Receivers, processors and exporters,
e.g. :

• Receiver as Source of Data of wide structured events,
• processor for processing logic,
• exporters as destination to send data as wide structured events to backends ( inside AWS or various telemetry vendors )

Receivers can be the Telemetry API extension that AWS announced recently. A developer can leverage the telemetry API, take some level of control of the execution environment and move the telemetry data to a managed telemetry system , e.g. ADOT Collector as managed service.
Link telemetry API

The processor for processing logic can be also a specification that the developer can code in a managed service like Lambda,
e.g AWS released recently a specification to convert tracing data in OTEL Spans.

• https://docs.aws.amazon.com/lambda/latest/dg/telemetry-otel-spans.html

Describe alternatives you've considered
Reuse or extend capabilities of data pipeline components already available in the serverless domain, like:
E.g. Kinesis Streams for batching as an Event Log Data Structure and Firehose for collecting , processing and exporting data to destinations.

• Firehose

Additional context
In my opinion, the AWS Kinesis Data Steams and Firehose behavior and Configuration capabilities are at a level similar to ADOT Collector characteristics.

So AWS maybe inspired to create a managed service of ADOT Collector or extends the exist data pipeline managed services of Kinesis to support telemetry data processing special use cases.

[niko-achilles]

As addition to the Request some thoughts:

Other AWS community members wish as request Exporters to be supported by ADOT Collector, e.g:

• S3
• Kinesis , etc

AWS managed services in the Kinesis boundaries already support such capabilities for persisting, transforming, analyzing, sending.

By having the ADOT Collector as managed service can improve the Telemetry Data pipeline for debugging and analysis purposes

E.g. By emitting the Telemetry Data in Kinesis Streams and taking under consideration the context , someone can send the data directly for debugging purposes in a tracing capable System and in parallel persist the telemetry Data for analysis or deliver to other destinations.

[niko-achilles]

@mhausenblas i have edited my issue by restructuring my ideas and sharing my thoughts.
As I am experimenting with ADOT , having some experiences and being an open telemetry contributor (seasoned developer contribution) I think that otel-lamda can improve.

[eriweb]

We're interested in this as well. It would be a requirement for us that it allows/enables usage of exporters to 3rd parties (like AppDynamics etc.)

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[github-actions]

This issue was closed because it has been marked as stale for 30 days with no activity.