Issue with Protobuf Files Configuration when using Kubernetes ConfigMap #4412

kehao95 · 2024-03-19T01:06:32Z

Issue submitter TODO list

I've looked up my issue in FAQ
I've searched for an already existing issues here
I've tried running master-labeled docker image and the issue still persists there
I'm running a supported version of the application which is listed here

Describe the bug (actual behavior)

Hello,

I've encountered an issue with the Kafka-UI configuration when utilizing Protobuf files for serialization/deserialization in a Kubernetes environment.
The Protobuf files are mounted using a Kubernetes ConfigMap. However, Kubernetes creates hidden files and links (e.g., ..data, ..timestamp.file) in the mounted directory. These hidden files and links are inadvertently being processed by Kafka-UI, leading to errors related to multiple enums sharing the same constant due to the presence of duplicate Protobuf files.

Expected behavior

Kafka-UI should ignore hidden files and links created by Kubernetes when mounting ConfigMaps, or provide a mechanism to exclude specific files or patterns from being processed.

Your installation details

The configuration is specified as follows:

yamlApplicationConfig:
  kafka:
    clusters:
      - name: kafka
        bootstrapServers: kafka-cluster-kafka-bootstrap:9092
        serde:
          - name: ProtobufFile
            properties:
              protobufFilesDir: /protofiles/
              protobufMessageName: my.Value

And I was mounting protobuf files using configmap (by ksutomization patch)

helmCharts:
- name: kafka-ui
  releaseName: kafka-ui
  version: 0.7.5
  repo: https://provectus.github.io/kafka-ui-charts
  valuesInline:
    yamlApplicationConfig:
      kafka:
        clusters:
          - name: kafka
            bootstrapServers: kafka-cluster-kafka-bootstrap:9092
            serde:
              - name: ProtobufFile
                properties:
                  protobufFilesDir: /protofiles/mypackage
                  protobufMessageName: mypackage.v1.MyType
      auth:
        type: disabled

patches:
- patch: |-
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: kafka-ui
    spec:
      revisionHistoryLimit: 1
      template:
        spec:
          containers:
          - name: kafka-ui
            volumeMounts:
            - name: protofiles
              mountPath: /protofiles/
          volumes:
          - name: protofiles
            configMap:
              name: proto-mypackage-v1

---
configMapGenerator:
- name: proto-package-v1
  options:
    disableNameSuffixHash: true
  files:
  - my_message.proto
  - my_common.proto

Steps to reproduce

Configure a Kafka cluster in Kafka-UI with Protobuf serialization/deserialization, specifying a directory mounted via Kubernetes ConfigMap for Protobuf files.
Observe the logs/errors in Kafka-UI related to duplicate enums or other Protobuf schema-related errors.

Screenshots

Logs

org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'messagesController' defined in URL [jar:file:/kafka-ui-api.jar!/BOOT-INF/classes!/com/provectus/kafka/ui/controller/MessagesController.class]: Unsatisfied dependency expressed through constructor parameter 0: Error creating bean with name 'messagesService' defined in URL [jar:file:/kafka-ui-api.jar!/BOOT-INF/classes!/com/provectus/kafka/ui/service/MessagesService.class]: Unsatisfied dependency expressed through constructor parameter 1: Error creating bean with name 'deserializationService' defined in URL [jar:file:/kafka-ui-api.jar!/BOOT-INF/classes!/com/provectus/kafka/ui/service/DeserializationService.class]: Failed to instantiate [com.provectus.kafka.ui.service.DeserializationService]: Constructor threw exception
	at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:800)
	at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:245)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1352)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1189)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:560)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:520)
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:326)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:324)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:973)
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:917)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:584)
	at org.springframework.boot.web.reactive.context.ReactiveWebServerApplicationContext.refresh(ReactiveWebServerApplicationContext.java:66)
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:732)
	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:434)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:310)
	at com.provectus.kafka.ui.KafkaUiApplication.startApplication(KafkaUiApplication.java:24)
	at com.provectus.kafka.ui.KafkaUiApplication.main(KafkaUiApplication.java:17)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:95)
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
	at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65)
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'messagesService' defined in URL [jar:file:/kafka-ui-api.jar!/BOOT-INF/classes!/com/provectus/kafka/ui/service/MessagesService.class]: Unsatisfied dependency expressed through constructor parameter 1: Error creating bean with name 'deserializationService' defined in URL [jar:file:/kafka-ui-api.jar!/BOOT-INF/classes!/com/provectus/kafka/ui/service/DeserializationService.class]: Failed to instantiate [com.provectus.kafka.ui.service.DeserializationService]: Constructor threw exception
	at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:800)
	at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:245)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1352)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1189)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:560)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:520)
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:326)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:324)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200)
	at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:254)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1417)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1337)
	at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:887)
	at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791)
	... 26 common frames omitted
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'deserializationService' defined in URL [jar:file:/kafka-ui-api.jar!/BOOT-INF/classes!/com/provectus/kafka/ui/service/DeserializationService.class]: Failed to instantiate [com.provectus.kafka.ui.service.DeserializationService]: Constructor threw exception
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:326)
	at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:314)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1352)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1189)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:560)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:520)
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:326)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:324)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200)
	at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:254)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1417)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1337)
	at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:887)
	at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791)
	... 40 common frames omitted
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.provectus.kafka.ui.service.DeserializationService]: Constructor threw exception
	at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:224)
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:110)
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:323)
	... 54 common frames omitted
Caused by: com.squareup.wire.schema.SchemaException: multiple enums share constant KIND_UNSPECIFIED:
  1. mypackage.v1.Kind.KIND_UNSPECIFIED (/protofiles/core_sdk_message.proto:89:3)
  2. mypackage.v1.Kind.KIND_UNSPECIFIED (/protofiles/..2024_03_18_18_43_09.906008176/core_sdk_message.proto:89:3)
  for file /protofiles/migration_message.proto
multiple enums share constant KIND_SESSION:
  1. mypackage.v1.Kind.KIND_SESSION (/protofiles/core_sdk_message.proto:90:3)
  2. mypackage.v1.Kind.KIND_SESSION (/protofiles/..2024_03_18_18_43_09.906008176/core_sdk_message.proto:90:3)
  for file /protofiles/migration_message.proto
multiple enums share constant KIND_EVENT:
  1. mypackage.v1.Kind.KIND_EVENT (/protofiles/core_sdk_message.proto:91:3)
  2. mypackage.v1.Kind.KIND_EVENT (/protofiles/..2024_03_18_18_43_09.906008176/core_sdk_message.proto:91:3)
  for file /protofiles/migration_message.proto
multiple enums share constant KIND_PAGEVIEW:
  1. mypackage.v1.Kind.KIND_PAGEVIEW (/protofiles/core_sdk_message.proto:92:3)
  2. mypackage.v1.Kind.KIND_PAGEVIEW (/protofiles/..2024_03_18_18_43_09.906008176/core_sdk_message.proto:92:3)
  for file /protofiles/migration_message.proto
multiple enums share constant KIND_USER:
  1. mypackage.v1.Kind.KIND_USER (/protofiles/core_sdk_message.proto:93:3)
  2. mypackage.v1.Kind.KIND_USER (/protofiles/..2024_03_18_18_43_09.906008176/core_sdk_message.proto:93:3)
  for file /protofiles/migration_message.proto
multiple enums share constant ENDPOINT_UNSPECIFIED:
  1. mypackage.v1.Endpoint.ENDPOINT_UNSPECIFIED (/protofiles/..2024_03_18_18_43_09.906008176/pre_enrich_enums.proto:8:3)
  2. mypackage.v1.Endpoint.ENDPOINT_UNSPECIFIED (/protofiles/pre_enrich_enums.proto:8:3)
  for file /protofiles/migration_message.proto
multiple enums share constant ENDPOINT_PIXEL:
  1. mypackage.v1.Endpoint.ENDPOINT_PIXEL (/protofiles/..2024_03_18_18_43_09.906008176/pre_enrich_enums.proto:9:3)
  2. mypackage.v1.Endpoint.ENDPOINT_PIXEL (/protofiles/pre_enrich_enums.proto:9:3)
  for file /protofiles/migration_message.proto
multiple enums share constant ENDPOINT_ANDROID:
  1. mypackage.v1.Endpoint.ENDPOINT_ANDROID (/protofiles/..2024_03_18_18_43_09.906008176/pre_enrich_enums.proto:11:3)
  2. mypackage.v1.Endpoint.ENDPOINT_ANDROID (/protofiles/pre_enrich_enums.proto:11:3)
  for file /protofiles/migration_message.proto
multiple enums share constant ENDPOINT_IOS:
  1. mypackage.v1.Endpoint.ENDPOINT_IOS (/protofiles/..2024_03_18_18_43_09.906008176/pre_enrich_enums.proto:13:3)
  2. mypackage.v1.Endpoint.ENDPOINT_IOS (/protofiles/pre_enrich_enums.proto:13:3)
  for file /protofiles/migration_message.proto
multiple enums share constant ENDPOINT_IDENTIFY_V2:
...

Additional context

Currently I'm using an init containers to copy the content over instead of mounting the configmap directly, which works for me.

patches:
- patch: |-
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: kafka-ui
    spec:
      revisionHistoryLimit: 1
      template:
        spec:
          initContainers:
          - name: copy-protofiles
            image: busybox
            command: ['sh', '-c', 'cp -L -r /proto-src/* /protofiles/ && find /protofiles/ -type d -name ".*" -exec rm -rf {} +']
            volumeMounts:
            - name: proto-mypackage-v1-src
              mountPath: /proto-src/mypackage/v1
            - name: protofiles
              mountPath: /protofiles
          containers:
          - name: kafka-ui
            volumeMounts:
            - name: protofiles
              mountPath: /protofiles/
          volumes:
          - name: proto-mypackage-v1-src
            configMap:
              name: proto-mypackage-v1
          - name: protofiles
            emptyDir: {}

The text was updated successfully, but these errors were encountered:

github-actions · 2024-03-19T01:07:00Z

Hello there kehao95! 👋

Thank you and congratulations 🎉 for opening your very first issue in this project! 💖

In case you want to claim this issue, please comment down below! We will try to get back to you as soon as we can. 👀

kehao95 · 2024-03-19T01:22:28Z

similar issue spring-projects/spring-boot#23232

Not sure, but maybe Files.walk with option FOLLOW_LINKS can resolve this issue.
Otherwise, we may need an extra filter in this function

kafka-ui/kafka-ui-api/src/main/java/com/provectus/kafka/ui/serdes/builtin/ProtobufFileSerde.java

Lines 399 to 415 in 53a6553

    
             private Map<String, ProtoFile> loadFilesWithLocations() { 
        
               Map<String, ProtoFile> filesByLocations = new HashMap<>(); 
        
               try (var files = Files.walk(baseLocation)) { 
        
                 files.filter(p -> !Files.isDirectory(p) && p.toString().endsWith(".proto")) 
        
                     .forEach(path -> { 
        
                       // relative path will be used as "import" statement 
        
                       String relativePath = baseLocation.relativize(path).toString(); 
        
                       var protoFileElement = ProtoParser.Companion.parse( 
        
                           Location.get(baseLocation.toString(), relativePath), 
        
                           readFileAsString(path) 
        
                       ); 
        
                       filesByLocations.put(relativePath, ProtoFile.Companion.get(protoFileElement)); 
        
                     }); 
        
               } 
        
               return filesByLocations; 
        
             } 
        
           }

Haarolean · 2024-04-02T16:08:22Z

@kehao95 hey, as this repo is no longer maintained (see #4255) I'd be happy to help you out here: kafbat/kafka-ui#262

kehao95 · 2024-04-02T18:24:53Z

@Haarolean Thank you for your contribution and fix I'll try it out! I hope there will be a deprecation notice on this project though.

kehao95 added status/triage Issues pending maintainers triage type/bug Something isn't working labels Mar 19, 2024

Haarolean mentioned this issue Mar 25, 2024

BE: Ignore hidden files in protobuf files directories kafbat/kafka-ui#245

Closed

kehao95 closed this as completed Apr 2, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issue with Protobuf Files Configuration when using Kubernetes ConfigMap #4412

Issue with Protobuf Files Configuration when using Kubernetes ConfigMap #4412

kehao95 commented Mar 19, 2024 •

edited

Loading

github-actions bot commented Mar 19, 2024

kehao95 commented Mar 19, 2024 •

edited

Loading

Haarolean commented Apr 2, 2024

kehao95 commented Apr 2, 2024

Issue with Protobuf Files Configuration when using Kubernetes ConfigMap #4412

Issue with Protobuf Files Configuration when using Kubernetes ConfigMap #4412

Comments

kehao95 commented Mar 19, 2024 • edited Loading

Issue submitter TODO list

Describe the bug (actual behavior)

Expected behavior

Your installation details

Steps to reproduce

Screenshots

Logs

Additional context

github-actions bot commented Mar 19, 2024

kehao95 commented Mar 19, 2024 • edited Loading

Haarolean commented Apr 2, 2024

kehao95 commented Apr 2, 2024

kehao95 commented Mar 19, 2024 •

edited

Loading

kehao95 commented Mar 19, 2024 •

edited

Loading