You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Setup:
Running traceshop locally with HT java agent and Hypertrace
I see following Deserialization exceptions occur intermittently in Span-normalizer service while consuming from jaeger-span kafka topic
I am not sure if it happens with existing agents as well. If it is a known issue please let me know.
hypertrace-ingester | 2020-12-04 07:52:44.839 [hypertrace-ingester-job-3badbaeb-656c-40c6-affc-58da86a0133d-StreamThread-1] WARN o.a.k.s.e.LogAndContinueExceptionHandler - Exception caught during Deserialization, taskId: 0_0, topic: jaeger-spans, partition: 0, offset: 141413 hypertrace-ingester | java.lang.RuntimeException: com.google.protobuf.InvalidProtocolBufferException: Protocol message had invalid UTF-8. hypertrace-ingester | at org.hypertrace.core.spannormalizer.jaeger.JaegerSpanSerde$De.deserialize(JaegerSpanSerde.java:45) ~[span-normalizer-0.3.4.jar:?] hypertrace-ingester | at org.hypertrace.core.spannormalizer.jaeger.JaegerSpanSerde$De.deserialize(JaegerSpanSerde.java:38) ~[span-normalizer-0.3.4.jar:?] hypertrace-ingester | at org.apache.kafka.common.serialization.Deserializer.deserialize(Deserializer.java:60) ~[kafka-clients-5.5.1-ccs.jar:?] hypertrace-ingester | at org.apache.kafka.streams.processor.internals.SourceNode.deserializeValue(SourceNode.java:63) ~[kafka-streams-5.5.1-ccs.jar:?] hypertrace-ingester | at org.apache.kafka.streams.processor.internals.RecordDeserializer.deserialize(RecordDeserializer.java:66) [kafka-streams-5.5.1-ccs.jar:?] hypertrace-ingester | at org.apache.kafka.streams.processor.internals.RecordQueue.updateHead(RecordQueue.java:175) [kafka-streams-5.5.1-ccs.jar:?] hypertrace-ingester | at org.apache.kafka.streams.processor.internals.RecordQueue.poll(RecordQueue.java:126) [kafka-streams-5.5.1-ccs.jar:?] hypertrace-ingester | at org.apache.kafka.streams.processor.internals.PartitionGroup.nextRecord(PartitionGroup.java:122) [kafka-streams-5.5.1-ccs.jar:?] hypertrace-ingester | at org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:368) [kafka-streams-5.5.1-ccs.jar:?] hypertrace-ingester | at org.apache.kafka.streams.processor.internals.AssignedStreamsTasks.process(AssignedStreamsTasks.java:475) [kafka-streams-5.5.1-ccs.jar:?] hypertrace-ingester | at org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:550) [kafka-streams-5.5.1-ccs.jar:?] hypertrace-ingester | at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:802) [kafka-streams-5.5.1-ccs.jar:?] hypertrace-ingester | at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:697) [kafka-streams-5.5.1-ccs.jar:?] hypertrace-ingester | at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:670) [kafka-streams-5.5.1-ccs.jar:?] hypertrace-ingester | Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message had invalid UTF-8. hypertrace-ingester | at com.google.protobuf.InvalidProtocolBufferException.invalidUtf8(InvalidProtocolBufferException.java:141) ~[protobuf-java-3.13.0.jar:?] hypertrace-ingester | at com.google.protobuf.Utf8$DecodeUtil.handleTwoBytes(Utf8.java:1909) ~[protobuf-java-3.13.0.jar:?] hypertrace-ingester | at com.google.protobuf.Utf8$DecodeUtil.access$700(Utf8.java:1883) ~[protobuf-java-3.13.0.jar:?] hypertrace-ingester | at com.google.protobuf.Utf8$UnsafeProcessor.decodeUtf8(Utf8.java:1411) ~[protobuf-java-3.13.0.jar:?] hypertrace-ingester | at com.google.protobuf.Utf8.decodeUtf8(Utf8.java:340) ~[protobuf-java-3.13.0.jar:?] hypertrace-ingester | at com.google.protobuf.CodedInputStream$ArrayDecoder.readStringRequireUtf8(CodedInputStream.java:809) ~[protobuf-java-3.13.0.jar:?] hypertrace-ingester | at io.jaegertracing.api_v2.JaegerSpanInternalModel$KeyValue.<init>(JaegerSpanInternalModel.java:352) ~[span-normalizer-api-0.3.4.jar:?] hypertrace-ingester | at io.jaegertracing.api_v2.JaegerSpanInternalModel$KeyValue$1.parsePartialFrom(JaegerSpanInternalModel.java:1261) ~[span-normalizer-api-0.3.4.jar:?] hypertrace-ingester | at io.jaegertracing.api_v2.JaegerSpanInternalModel$KeyValue$1.parsePartialFrom(JaegerSpanInternalModel.java:1255) ~[span-normalizer-api-0.3.4.jar:?] hypertrace-ingester | at com.google.protobuf.CodedInputStream$ArrayDecoder.readMessage(CodedInputStream.java:888) ~[protobuf-java-3.13.0.jar:?] hypertrace-ingester | at io.jaegertracing.api_v2.JaegerSpanInternalModel$Span.<init>(JaegerSpanInternalModel.java:4100) ~[span-normalizer-api-0.3.4.jar:?] hypertrace-ingester | at io.jaegertracing.api_v2.JaegerSpanInternalModel$Span$1.parsePartialFrom(JaegerSpanInternalModel.java:6560) ~[span-normalizer-api-0.3.4.jar:?] hypertrace-ingester | at io.jaegertracing.api_v2.JaegerSpanInternalModel$Span$1.parsePartialFrom(JaegerSpanInternalModel.java:6554) ~[span-normalizer-api-0.3.4.jar:?] hypertrace-ingester | at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:158) ~[protobuf-java-3.13.0.jar:?] hypertrace-ingester | at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:191) ~[protobuf-java-3.13.0.jar:?] hypertrace-ingester | at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:203) ~[protobuf-java-3.13.0.jar:?] hypertrace-ingester | at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:208) ~[protobuf-java-3.13.0.jar:?] hypertrace-ingester | at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:48) ~[protobuf-java-3.13.0.jar:?] hypertrace-ingester | at io.jaegertracing.api_v2.JaegerSpanInternalModel$Span.parseFrom(JaegerSpanInternalModel.java:4713) ~[span-normalizer-api-0.3.4.jar:?] hypertrace-ingester | at org.hypertrace.core.spannormalizer.jaeger.JaegerSpanSerde$De.deserialize(JaegerSpanSerde.java:43) ~[span-normalizer-0.3.4.jar:?] hypertrace-ingester | ... 13 more hypertrace-ingester | 2020-12-04 07:52:44.841 [hypertrace-ingester-job-3badbaeb-656c-40c6-affc-58da86a0133d-StreamThread-1] WARN o.a.k.s.p.i.RecordDeserializer - stream-thread [hypertrace-ingester-job-3badbaeb-656c-40c6-affc-58da86a0133d-StreamThread-1] task [0_0] Skipping record due to deserialization error. topic=[jaeger-spans] partition=[0] offset=[141413]
The text was updated successfully, but these errors were encountered:
Setup:
Running traceshop locally with HT java agent and Hypertrace
I see following Deserialization exceptions occur intermittently in Span-normalizer service while consuming from
jaeger-span
kafka topicI am not sure if it happens with existing agents as well. If it is a known issue please let me know.
hypertrace-ingester | 2020-12-04 07:52:44.839 [hypertrace-ingester-job-3badbaeb-656c-40c6-affc-58da86a0133d-StreamThread-1] WARN o.a.k.s.e.LogAndContinueExceptionHandler - Exception caught during Deserialization, taskId: 0_0, topic: jaeger-spans, partition: 0, offset: 141413 hypertrace-ingester | java.lang.RuntimeException: com.google.protobuf.InvalidProtocolBufferException: Protocol message had invalid UTF-8. hypertrace-ingester | at org.hypertrace.core.spannormalizer.jaeger.JaegerSpanSerde$De.deserialize(JaegerSpanSerde.java:45) ~[span-normalizer-0.3.4.jar:?] hypertrace-ingester | at org.hypertrace.core.spannormalizer.jaeger.JaegerSpanSerde$De.deserialize(JaegerSpanSerde.java:38) ~[span-normalizer-0.3.4.jar:?] hypertrace-ingester | at org.apache.kafka.common.serialization.Deserializer.deserialize(Deserializer.java:60) ~[kafka-clients-5.5.1-ccs.jar:?] hypertrace-ingester | at org.apache.kafka.streams.processor.internals.SourceNode.deserializeValue(SourceNode.java:63) ~[kafka-streams-5.5.1-ccs.jar:?] hypertrace-ingester | at org.apache.kafka.streams.processor.internals.RecordDeserializer.deserialize(RecordDeserializer.java:66) [kafka-streams-5.5.1-ccs.jar:?] hypertrace-ingester | at org.apache.kafka.streams.processor.internals.RecordQueue.updateHead(RecordQueue.java:175) [kafka-streams-5.5.1-ccs.jar:?] hypertrace-ingester | at org.apache.kafka.streams.processor.internals.RecordQueue.poll(RecordQueue.java:126) [kafka-streams-5.5.1-ccs.jar:?] hypertrace-ingester | at org.apache.kafka.streams.processor.internals.PartitionGroup.nextRecord(PartitionGroup.java:122) [kafka-streams-5.5.1-ccs.jar:?] hypertrace-ingester | at org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:368) [kafka-streams-5.5.1-ccs.jar:?] hypertrace-ingester | at org.apache.kafka.streams.processor.internals.AssignedStreamsTasks.process(AssignedStreamsTasks.java:475) [kafka-streams-5.5.1-ccs.jar:?] hypertrace-ingester | at org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:550) [kafka-streams-5.5.1-ccs.jar:?] hypertrace-ingester | at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:802) [kafka-streams-5.5.1-ccs.jar:?] hypertrace-ingester | at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:697) [kafka-streams-5.5.1-ccs.jar:?] hypertrace-ingester | at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:670) [kafka-streams-5.5.1-ccs.jar:?] hypertrace-ingester | Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message had invalid UTF-8. hypertrace-ingester | at com.google.protobuf.InvalidProtocolBufferException.invalidUtf8(InvalidProtocolBufferException.java:141) ~[protobuf-java-3.13.0.jar:?] hypertrace-ingester | at com.google.protobuf.Utf8$DecodeUtil.handleTwoBytes(Utf8.java:1909) ~[protobuf-java-3.13.0.jar:?] hypertrace-ingester | at com.google.protobuf.Utf8$DecodeUtil.access$700(Utf8.java:1883) ~[protobuf-java-3.13.0.jar:?] hypertrace-ingester | at com.google.protobuf.Utf8$UnsafeProcessor.decodeUtf8(Utf8.java:1411) ~[protobuf-java-3.13.0.jar:?] hypertrace-ingester | at com.google.protobuf.Utf8.decodeUtf8(Utf8.java:340) ~[protobuf-java-3.13.0.jar:?] hypertrace-ingester | at com.google.protobuf.CodedInputStream$ArrayDecoder.readStringRequireUtf8(CodedInputStream.java:809) ~[protobuf-java-3.13.0.jar:?] hypertrace-ingester | at io.jaegertracing.api_v2.JaegerSpanInternalModel$KeyValue.<init>(JaegerSpanInternalModel.java:352) ~[span-normalizer-api-0.3.4.jar:?] hypertrace-ingester | at io.jaegertracing.api_v2.JaegerSpanInternalModel$KeyValue$1.parsePartialFrom(JaegerSpanInternalModel.java:1261) ~[span-normalizer-api-0.3.4.jar:?] hypertrace-ingester | at io.jaegertracing.api_v2.JaegerSpanInternalModel$KeyValue$1.parsePartialFrom(JaegerSpanInternalModel.java:1255) ~[span-normalizer-api-0.3.4.jar:?] hypertrace-ingester | at com.google.protobuf.CodedInputStream$ArrayDecoder.readMessage(CodedInputStream.java:888) ~[protobuf-java-3.13.0.jar:?] hypertrace-ingester | at io.jaegertracing.api_v2.JaegerSpanInternalModel$Span.<init>(JaegerSpanInternalModel.java:4100) ~[span-normalizer-api-0.3.4.jar:?] hypertrace-ingester | at io.jaegertracing.api_v2.JaegerSpanInternalModel$Span$1.parsePartialFrom(JaegerSpanInternalModel.java:6560) ~[span-normalizer-api-0.3.4.jar:?] hypertrace-ingester | at io.jaegertracing.api_v2.JaegerSpanInternalModel$Span$1.parsePartialFrom(JaegerSpanInternalModel.java:6554) ~[span-normalizer-api-0.3.4.jar:?] hypertrace-ingester | at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:158) ~[protobuf-java-3.13.0.jar:?] hypertrace-ingester | at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:191) ~[protobuf-java-3.13.0.jar:?] hypertrace-ingester | at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:203) ~[protobuf-java-3.13.0.jar:?] hypertrace-ingester | at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:208) ~[protobuf-java-3.13.0.jar:?] hypertrace-ingester | at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:48) ~[protobuf-java-3.13.0.jar:?] hypertrace-ingester | at io.jaegertracing.api_v2.JaegerSpanInternalModel$Span.parseFrom(JaegerSpanInternalModel.java:4713) ~[span-normalizer-api-0.3.4.jar:?] hypertrace-ingester | at org.hypertrace.core.spannormalizer.jaeger.JaegerSpanSerde$De.deserialize(JaegerSpanSerde.java:43) ~[span-normalizer-0.3.4.jar:?] hypertrace-ingester | ... 13 more hypertrace-ingester | 2020-12-04 07:52:44.841 [hypertrace-ingester-job-3badbaeb-656c-40c6-affc-58da86a0133d-StreamThread-1] WARN o.a.k.s.p.i.RecordDeserializer - stream-thread [hypertrace-ingester-job-3badbaeb-656c-40c6-affc-58da86a0133d-StreamThread-1] task [0_0] Skipping record due to deserialization error. topic=[jaeger-spans] partition=[0] offset=[141413]
The text was updated successfully, but these errors were encountered: