How to migrate from home-grown tracing to OpenTelemetry (aka header munging)

[djMax]

I am trying to modernize a circa-2016 micro-services NodeJS stack. We have a poor mans version of distributed tracing in that:

• Every request gets assigned a correlation id (aka traceid?) either at the client end or by our API gateway (ambassador).
• Every inter-service calls gets a span id

But, the correlationid is a mongo object hash and the span id is a "tree like counter" - e.g. first span is 1, that makes a call it gets 1.1, then the parent makes another call, 1.2, and if it calls something else, it's 1.2.1, etc.

SO, obviously I don't want to have to replace all services at once and I'd like tools to mostly keep working while we enable new tools. (We're using Sumo now if that ends up mattering).

Is it possible to take our correlation id header c and somehow have that become the traceid in the OpenTelemetry stack? It seems like maybe this has to do with the "propagator"? Or maybe the context manager? I can't quite figure out what to look for in the docs.

Thanks!

[vmarchaud]

I think i would have a custom propagator that take the correlation id, hash it and use that as traceId for otel. Here is an example (pretty much in pseudo code) that you could start from:

// name of your header
const X_CUSTOM_CONTEXT = 'correlation-id'
// store the original correlation id in the context
export const CUSTOM_COMTEXT_SYMBOL = Symbol.for(X_CUSTOM_CONTEXT)

export class CustomContextPropagator implements TextMapPropagator {
  inject (context: Context, carrier: unknown, setter: SetterFunction) {
    const customContext = context.getValue(CUSTOM_COMTEXT_SYMBOL)
    if (typeof customContext !== 'undefined') {
      setter(carrier, X_CUSTOM_CONTEXT, JSON.stringify(customContext))
    }
  }

  extract (context: Context, carrier: unknown, getter: GetterFunction): Context {
    const customContext = getter(carrier, X_CUSTOM_CONTEXT)
    if (typeof customContext !== 'string') return context
    return trace.setSpan(context.setValue(CUSTOM_COMTEXT_SYMBOL, JSON.parse(customContext)), trace.wrapSpanContext({
          traceId: hash(customContext) // here we create a traceId from your header, should be consistent across your services,
          spanId: 0, // you may want to do the same thing with your header that contains span "counter" to link remote span 
          isRemote: true
        }))
  }
}

[djMax]

Are the otel traceids forced to be in a certain format? Because I'd rather the value was the same, not a hash of it. Otherwise searching in logs and such becomes a chore.

[vmarchaud]

Yeah there is a specific format: https://github.com/open-telemetry/opentelemetry-js/blob/main/api/src/trace/spancontext-utils.ts#L24