Using Jaeger for multiple worker nodes of Kubernetes

[nfsp3k]

Hi,

I'm trying to use Jaeger in my Kubernetes cluster that has multiple worker nodes.
An open-source microservice benchmark called DeathStarBench, which has 30 microservices, is running in my cluster.

I found a kind of 'delay' in each span, which I guess would be derived from the timestamp difference between worker nodes.

I synchronized the time between worker machines using the NTP command that uses the same source across machines.
Does the delay fundamentally exist? How can I remove the delay and retrieve the correct result?

[yurishkuro]

What "delay" are you referring to?

You can try running the query service with this flag:

      --query.max-clock-skew-adjustment duration   The maximum delta by which span timestamps may be adjusted in the UI due to clock skew; set to 0s to disable clock skew adjustments (default 0s)

[nfsp3k]

Some of the spans are not located appropriately in the UI. For example, the yellow line, whose name is 'media-service', is a child span of the span named 'nginx-web-server' and should be present after 'nginx-web-server'. However, It appears before the 'nginx-web-server', which seems odd.

How can I use that flag? Should I have to write that flag in the Jager .yaml file of the Kubernetes resource?
And how can I determine the value of 'duration'?

Please understand my poor background in it.

[yurishkuro]

The UI lays out the spans according to their timestamps. Due to clock skew on the hosts it's possible for child spans on another node to have timestamps earlier than the parent span.

If you were running the query service binary directly, you could pass the flag via command line: --query.max-clock-skew-adjustment 1s. I don't know exactly how you're running it on k8s, but I assume you can specify command line args or env vars (for env var just upper case all letters and replace other chars with _).

[nfsp3k]

Thank you for your quick reply.

I'm going to try to use that flag.
Thank you.