Jaeger - Distributed Tracing Platform
Jaeger ingester with OpenSearch #5139
Unanswered
gfes980615
asked this question in
Q&A
Replies: 1 comment
-
|
All there parameters are pass-through to Elasticsearch, I suggest reading documentation there (or in the olivere ES driver) |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hi team
Issue
I used the jaeger ingester with OpenSearch
Sometimes I will receive the 429 error, the error message like the following
{ "level": "error", "ts": 1705999951.4621732, "caller": "config/config.go:146", "msg": "Elasticsearch could not process bulk request", "request_count": 732, "failed_count": 2, "error": "elastic: uncommitted bulk respo nse items", "response": { "took": 9, "errors": true, "items": [ { "index": { "_index": "jaeger-span-2024-01-23-08", "_id": "xTOFNY0BGQIQjPT5ZkRb", "_version": 1, "result": "created", "_shards": { "total": 2, "successful": 2, "failed": 0 }, "_seq_no": 7488356, "_pri mary_term": 1, "status": 201 } }, { "index": { "_index": "jaeger-span-2024-01-23-08", "_id": "xjOFNY0BGQIQjPT5ZkRb", "status": 429, "error": { "type": "circuit_breaking_exception", "reason": "[parent] Data too large, data for [indices:data/write/bulk[s]] would be [16809625030/15.6gb], which is larger than the limit of [16320875724/15.1gb], real usage: [16809617624/15.6gb], new bytes reserved: [7406/7.2kb], usages [request=0/0b, fielddata=3437877477/3.2gb, in_flight_requests=7406/7.2k b]" } } }, { "index": { "_index": "jaeger-span-2024-01-23-08", "_id": "xzOFNY0BGQIQjPT5ZkRb", "_version": 1, "result": "created", "_shards": { "total": 2, "successful": 2, "failed": 0 }, "_seq_no": 7489162, "_primary_term": 1, "status": 201 } }, { "index": { "_index": " jaeger-span-2024-01-23-08", "_id": "yDOFNY0BGQIQjPT5ZkRb", "_version": 1, "result": "created", "_shards": { "total": 2, "successful": 2, "failed": 0 }, "_seq_no": 7442930, "_primary_term": 1, "status": 201 } }, { "index": { "_index": "jaeger-span-2024-01-23-08", "_id": "yTOFNY0BGQIQjPT5ZkRb", "_version": 1, "result": "created", "_shards": { "total": 2, "successful": 2, "failed": 0 }, "_seq_no": 7489163, "_primary_term": 1, "status": 201 } }, { "index": { "_index": "jaeger-span-2024-01-23-08", "_id": "yjOFNY0BGQIQjPT5ZkRb ", "_version": 1, "result": "created", "_shards": { "total": 2, "successful": 2, "failed": 0 }, "_seq_no": 7442931, "_primary_term": 1, "status": 201 } }, { "index": { "_index": "jaeger-span-2024-01-23-08", "_id": "yzOFNY0BGQIQjPT5ZkRb", "_version": 1, "result": "cr eated", "_shards": { "total": 2, "successful": 2, "failed": 0 }, "_seq_no": 7442932, "_primary_term": 1, "status": 201 } }, { "index": { "_index": "jaeger-span-2024-01-23-08", "_id": "zDOFNY0BGQIQjPT5ZkRb", "_version": 1, "result": "created", "_shards": { "total": 2, "successful": 2, "failed": 0 }, "_seq_no": 7442933, "_primary_term": 1, "status": 201 } }, { "index": { "_index": "jaeger-span-2024-01-23-08", "_id": "zTOFNY0BGQIQjPT5ZkRb", "_version": 1, "result": "created", "_shards": { "total": 2, "successful": 2, "failed": 0 }, "_seq_no": 7488357, "_primary_term": 1, "status": 201 } }, { "index": { "_index": "jaeger-span-2024-01-23-08", "_id": "zjOFNY0BGQIQjPT5ZkRb", "_version": 1, "result": "created", "_shards": { "total": 2, "successful": 2, "failed": 0 }, "_seq_no": 7487800, "_primar y_term": 1, "status": 201 } }, { "index": { "_index": "jaeger-span-2024-01-23-08", "_id": "zzOFNY0BGQIQjPT5ZkRb", "_version": 1, "result": "created", "_shards": { "total": 2, "successful": 2, "failed": 0 }, "_seq_no": 7442934, "_primary_term": 1, "status": 201 } }, { " index": { "_index": "jaeger-span-2024-01-23-08", "_id": "0DOFNY0BGQIQjPT5ZkRb", "_version": 1, "result": "created", "_shards": { "total": 2, "successful": 2, "failed": 0 }, "_seq_no": 7488358, "_primary_term": 1, "status": 201 } }, { "index": { "_index": "jaeger-sp an-2024-01-23-08", "_id": "0TOFNY0BGQIQjPT5ZkRb", "_version": 1, "result": "created", "_shards": { "total": 2, "successful": 2, "failed": 0 }, "_seq_no": 7465703, "_primary_term": 1, "status": 201 } }, { "index": { "_index": "jaeger-span-2024-01-23-08", "_id": "0j OFNY0BGQIQjPT5ZkRb", "_version": 1, "result": "created", "_shards": { "total": 2, "successful": 2, "failed": 0 }, "_seq_no": 7487801, "_primary_term": 1, "status": 201 } }, { "index": { "_index": "jaeger-span-2024-01-23-08", "_id": "0zOFNY0BGQIQjPT5ZkRb", "status ": 429, "error": { "type": "circuit_breaking_exception", "reason": "[parent] Data too large, data for [indices:data/write/bulk[s]] would be [16809625030/15.6gb], which is larger than the limit of [16320875724/15.1gb], real usage: [1680961762 4/15.6gb], new bytes reserved: [7406/7.2kb], usages [request=0/0b, fielddata=3437877477/3.2gb, in_flight_requests=7406/7.2kb]" } } }, { "index": { "_index": "jaeger-span-2024-01-23-08", "_id": "1DOFNY0BGQIQjPT5ZkRb", "_version": 1, "result": "creat ed", "_shards": { "total": 2, "successful": 2, "failed": 0 }, "_seq_no": 7488359, "_primary_term": 1, "status": 201 } }, { "index": { "_index": "jaeger-span-2024-01-23-08", "_id": "1TOFNY0BGQIQjPT5ZkRb", "_version": 1, "result": "created", "_shards": { "total": 2, "s uccessful": 2, "failed": 0 }, "_seq_no": 7465704, "_primary_term": 1, "status": 201 } }, { "index": { "_index": "jaeger-span-2024-01-23-08", "_id": "1jOFNY0BGQIQjPT5ZkRb", "_version": 1, "result": "created", "_shards": { "total": 2, "successful": 2, "failed": 0 }, "_ seq_no": 7442935, "_primary_term": 1, "status": 201 } }, { "index": { "_index": "jaeger-span-2024-01-23-08", "_id": "1zOFNY0BGQIQjPT5ZkRb", "_version": 1, "result": "created", "_shards": { "total": 2, "successful": 2, "failed": 0 }, "_seq_no": 7489164, "_primary_t erm": 1, "status": 201 } }, { "index": { "_index": "jaeger-span-2024-01-23-08", "_id": "2DOFNY0BGQIQjPT5ZkRb", "_version": 1, "result": "created", "_shards": { "total": 2, "successful": 2, "failed": 0 }, "_seq_no": 7488360, "_primary_term": 1, "status": 201 } }, { "ind ex": { "_index": "jaeger-span-2024-01-23-08", "_id": "2TOFNY0BGQIQjPT5ZkRb", "_version": 1, "result": "created", "_shards": { "total": 2, "successful": 2, "failed": 0 }, "_seq_no": 7489165, "_primary_term": 1, "status": 201 } }, { "index": { "_index": "jaeger-span- 2024-01-23-08", "_id": "2jOFNY0BGQIQjPT5ZkRb", "_version": 1, "result": "created", "_shards": { "total": 2, "successful": 2, "failed": 0 }, "_seq_no": 7442936, "_primary_term": 1, "status": 201 } } ] }, "stacktrace": "github.com/jaegertracing/jaeger/pkg/es/conf ig.NewClient.func2\n\tgithub.com/jaegertracing/jaeger/pkg/es/config/config.go:146\ngithub.com/olivere/elastic.(*bulkWorker).commit\n\tgithub.com/olivere/[email protected]+incompatible/bulk_processor.go:588\ngithub.com/olivere/elastic.(* bulkWorker).work\n\tgithub.com/olivere/[email protected]+incompatible/bulk_processor.go:487" }Question
I suppose the message means the bulk request bytes are too large, I need to tune the bulk config with Jaeger ingester
And the configs I need to tune are
--es.bulk.actions,--es.bulk.size, and--es.bulk.workersIs it right?
Does bulk request byte will be the
--es.bulk.actions*--es.bulk.size*--es.bulk.workers?Will any other config affect the bulk request bytes?
Or does it have nothing to do with these settings?
Beta Was this translation helpful? Give feedback.
All reactions