Is it possible to integrate clickhouse as backend storage with Jaeger 1.58+ ?

[chenlujjj]

Hi Team @yurishkuro , according to the documentation, Jaeger V2 will natively support clickhouse, that's an exciting news!

Now there is a jaeger-clickhouse plugin, which has not been updated for 2 years. And "as of v1.58, Jaeger will no longer support grpc-sidecar plugin model". So does it mean it's not possible to integrate Jaeger v1.58+ with clickhouse now ?

[dosubot[bot]]

Hey @chenlujjj, I'm here to help you with any bugs, questions, or contributions while you wait for a human maintainer. Let me know how I can assist you!

No, it is not possible to integrate Jaeger v1.58+ with ClickHouse as backend storage using the grpc-sidecar plugin model, as Jaeger no longer supports this model and the jaeger-clickhouse plugin has not been updated for 2 years. However, Jaeger does support a gRPC-based Remote Storage API that allows extending the Jaeger ecosystem with other storage backends, which can be deployed as a remote gRPC server. This might be a potential alternative to explore for integrating with ClickHouse ^[1][2][3].

To continue talking to Dosu, mention @dosu.

[chenlujjj]

Below is the output when I tried to run it as a server:

$ ./jaeger-clickhouse-darwin-amd64 -config=config.yaml
{"@level":"error","@message":"Failed to create a storage","@module":"jaeger-clickhouse","@timestamp":"2024-08-15T22:23:41.747791+08:00","EXTRA_VALUE_AT_END":"could not connect to database: \"dial tcp: lookup some-clickhouse-server: no such host\""}
❯ ./jaeger-clickhouse-darwin-amd64 -config=config.yaml
{"@level":"debug","@message":"Running SQL statement","@module":"jaeger-clickhouse","@timestamp":"2024-08-15T22:24:10.568296+08:00","statement":"CREATE TABLE IF NOT EXISTS jaeger_index_local\n\n(\n    timestamp  DateTime CODEC (Delta, ZSTD(1)),\n    traceID    String CODEC (ZSTD(1)),\n    service    LowCardinality(String) CODEC (ZSTD(1)),\n    operation  LowCardinality(String) CODEC (ZSTD(1)),\n    durationUs UInt64 CODEC (ZSTD(1)),\n    tags Nested\n    (\n        key LowCardinality(String),\n        value String\n    ) CODEC (ZSTD(1)),\n    INDEX idx_tag_keys tags.key TYPE bloom_filter(0.01) GRANULARITY 64,\n    INDEX idx_duration durationUs TYPE minmax GRANULARITY 1\n) ENGINE MergeTree()\n    \n    PARTITION BY (\n        toDate(timestamp)\n    )\n    ORDER BY (service, -toUnixTimestamp(timestamp))\n    SETTINGS index_granularity = 1024\n"}
{"@level":"debug","@message":"Running SQL statement","@module":"jaeger-clickhouse","@timestamp":"2024-08-15T22:24:10.580131+08:00","statement":"CREATE MATERIALIZED VIEW IF NOT EXISTS jaeger_operations_local\n\n    ENGINE SummingMergeTree\n    \n    PARTITION BY (\n        toYYYYMM(date)\n    )\n    ORDER BY (\n        date,\n        service,\n        operation\n    )\n    SETTINGS index_granularity = 32\n    POPULATE\nAS SELECT\n    toDate(timestamp) AS date,\n    service,\n    operation,\n    count() AS count,\n    if(\n        has(tags.key, 'span.kind'),\n        tags.value[indexOf(tags.key, 'span.kind')],\n        ''\n    ) AS spankind\nFROM default.jaeger_index_local\nGROUP BY\n    date,\n    service,\n    operation,\n    tags.key,\n    tags.value\n"}
{"@level":"debug","@message":"Running SQL statement","@module":"jaeger-clickhouse","@timestamp":"2024-08-15T22:24:10.592559+08:00","statement":"CREATE TABLE IF NOT EXISTS jaeger_spans_local\n\n(\n    timestamp DateTime CODEC (Delta, ZSTD(1)),\n    traceID   String CODEC (ZSTD(1)),\n    model     String CODEC (ZSTD(3))\n) ENGINE MergeTree()\n    \n    PARTITION BY (\n        toDate(timestamp)\n    )\n    ORDER BY traceID\n    SETTINGS index_granularity = 1024\n"}
{"@level":"debug","@message":"Running SQL statement","@module":"jaeger-clickhouse","@timestamp":"2024-08-15T22:24:10.595937+08:00","statement":"CREATE TABLE IF NOT EXISTS jaeger_spans_archive_local\n\n(\n    timestamp DateTime CODEC (Delta, ZSTD(1)),\n    traceID   String CODEC (ZSTD(1)),\n    model     String CODEC (ZSTD(3))\n) ENGINE MergeTree()\n    \n    PARTITION BY (\n        toYYYYMM(timestamp)\n    )\n    ORDER BY traceID\n    SETTINGS index_granularity = 1024\n"}
This binary is a plugin. These are not meant to be executed directly.
Please execute the program that consumes these plugins, which will
load any plugins automatically

[chenlujjj]

The error message comes from hashcorp go-plugin.
If I set environment variable STORAGE_PLUGIN=jaeger (the env key value are defined here), it can run as a grpc server:

$ STORAGE_PLUGIN=jaeger ./jaeger-clickhouse-darwin-amd64 -config=config.yaml
...
{"@level":"debug","@message":"plugin address","@timestamp":"2024-08-15T22:43:15.097696+08:00","address":"/var/folders/7n/l6kcxx7x4vz_4wlknrwrnd_w0000gn/T/plugin3356097990","network":"unix"}
1|1|unix|/var/folders/7n/l6kcxx7x4vz_4wlknrwrnd_w0000gn/T/plugin3356097990|grpc|

However, since it's a unix socket, and I run jaeger all-in-one in a docker container and jaeger-clickhouse grpc server in the host machine, so the client cannot connect to the server (explanation)

[chenlujjj]

@yurishkuro Please correct me if I miss anything :-) Thanks a lot!

BTW, may I know if the jaeger community will continue to update the clickhouse plugin ? Or it will be merged into jaeger v2 codebase?

[yurishkuro]

Your analysis seems correct. I still think it should be pretty straightforward though to tweak that binary to run as a server, using cmd/remote-storage/main.go (in the main repo) as example.

if the jaeger community will continue to update the clickhouse plugin

It is unlikely - this was never "officially" supported repository, so if its maintainers are non-responsive then it's probably dead.

Or it will be merged into jaeger v2 codebase?

Not really "merged", we plan to have native support for CH in v2.

[chenlujjj]

Thank you very much, I'll try the server solution

[selfing12]

@yurishkuro Hello, now in version 1.60 can we implement work with CH?