Expand GenAI project to cover instrumentation libraries

[lmolkova]

Based on the discussions in LLM semconv group, we'd like to suggest expanding the scope of the group to cover instrumentation libraries.

The long-term goal of the project is:

• provide space within otel for different GenAI observability vendors to collaborate on GenAI instrumentations
• provide official otel-hosted or native/first-party instrumentations for popular GenAI components covering current https://github.com/traceloop/openllmetry scope and transfer corresponding PyPI packages ownership to OTel
• evolve GenAI semantic conventions to meet generic and unique GenAI observability needs (already covered with existing LLM semconv)

We intend to add instrumentations one-by-one and limit features to those documented in semantic conventions. Instrumentations will evolve along with semconv, but may provide additional hooks or opt-in features allowing to customize them to specific vendor needs.

We're going to request opentelemetry-genai repo for the coordination/project management, but may request additional repos later to host WIP gen-ai instrumentation libraries when using existing contrib repos is problematic. with the long-term goal to contribute them upstream or merge to contrib once possible.

We're looking for engineers to work on instrumentation libraries, especially in JavaScript and existing JS-contrib approvers/maintainers to help with code review, best practices and general OTel.

/cc @alolita @tedsuo @trask @nirga @drewby @karthikscale3 @lzchen

[karthikscale3]

Thank you for bringing this topic up @lmolkova . From Langtrace, myself and my colleague, Ali are happy to contribute/maintain/approve the channges.

[lmolkova]

Tagging more people in the community who might be interested to work on instrumentation libraries @AmanAgarwal041 @cartermp @codefromthecrypt @langchain4j @trentm

[nirga]

@lmolkova happy to assist here! I think my co-founder @galkleinman would also want to participate

[alizenhom]

@lmolkova More than happy to help here!

[danielgblanco]

Very happy to see this taking shape!

[codefromthecrypt]

A couple org/scope related questions which are possibly just because I am new to how things are structured. Please forgive if they are misplaced.

[patcher9]

Great to see this in motion, Few nits as to keep vendor neutrality from this proposal. Me and @AmanAgarwal041 are happy to contribute instrumentations from https://github.com/openlit/openlit aswell

[codefromthecrypt]

I used to maintain js and python for zipkin and currently dusting off my skills, reviewing and maintaining these languages again.

If you need another approver who will pull changes local, test them, verify against current practice, etc before clicking ✔️ , feel free to use me. There's a lot of work to do, and I suspect this will help keep wheels in APAC moving.

[tedsuo]

I'd be happy to sponsor this SIG