Protobuf Serialization/Deserialization Failure with Redpanda Schema Registry Using Confluent JavaScript Client

[Havret]

Version & Environment

• Redpanda version: redpanda:v24.2.9
• JavaScript libraries:
  • @confluentinc/kafka-javascript: ^1.2.0
  • @confluentinc/schemaregistry: ^1.1.0

What went wrong?

When attempting to use Redpanda's schema registry (redpanda:v24.2.9) with a Protobuf-based producer and consumer setup, messages fail to serialize and deserialize correctly using Confluent's JavaScript client (@confluentinc/schemaregistry and @confluentinc/kafka-javascript).

Steps to reproduce and full setup can be found in this repository:
redpanda-schema-repository-repro

What should have happened instead?

Messages should serialize and deserialize correctly when using the Redpanda schema registry, just as they do when using the Confluent Kafka schema registry.

How to reproduce the issue?

1. Start Redpanda (v24.2.9) with schema registry enabled:
2. Clone and set up the repository:

   git clone https://github.com/Havret/redpanda-schema-repository-repro.git
   cd redpanda-schema-repository-repro
   npm install
   npm run build

3. Run the test:

   npm run dev

4. Observe serialization/deserialization failures in the output.

Additional information

If needed, I can provide further logs or insights.

JIRA Link: CORE-9143

[quophyie]

It seems that this issue is partly caused by the fact that the RedPanda Schema Registry ignores the query param value format=serialized because the redpanda schema registry returns plain text schema instead of a base64 encoded schema when endpoints which accept the format param are called e.g. http://localhost:8081/subjects/my-subject/versions/latest?format=serialized. The confluent schema registry behaves differently i.e. it returns the base64 encoded version of the schema

[dotnwat]

@michael-redpanda @BenPope this issue is new, but I think in Jira we are tracking something similar to this? If it is the same, do we have a different public tracking ticket we could link here?

[BenPope]

this issue is new, but I think in Jira we are tracking something similar to this? If it is the same, do we have a different public tracking ticket we could link here?

The only Jira ticket I could find is the one raised by this issue, and linked above.