Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

magic number mismatch when upgrading docker to 2.34 -> 2.41 #2961

Open
MostFrabjous opened this issue Apr 17, 2024 · 3 comments
Open

magic number mismatch when upgrading docker to 2.34 -> 2.41 #2961

MostFrabjous opened this issue Apr 17, 2024 · 3 comments
Assignees
Labels
bug Bug report

Comments

@MostFrabjous
Copy link

Description

I attempted to upgrade our percona/pmm-server docker installation from tag 2.34 to 2.41

Expected Results

Upgrade finishes and PMM is accepting data from pmm-agent

Actual Results

PMM does not process new data from pmm-agent (v2.39.0) and logs errors (see below).

Note:

  • I could not find any compatibility documentation on pmm-agent version vs pmm-server versions.
  • I did not notice anything about pmm-agent when following the Upgrade Documentation.

Version

PMM Server 2.34.0

PMM Agent 2.39

Steps to reproduce

Upgrade docker container: https://docs.percona.com/percona-monitoring-and-management/setting-up/server/docker.html#upgrade

Relevant logs

2024-04-17T12:14:15.157Z	warn	/home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/app/vminsert/main.go:167	remoteAddr: "127.0.0.1:47654"; requestURI: /prometheus/api/v1/write; cannot decompress zstd-encoded request with length 340: invalid input: magic number mismatch


### Code of Conduct

- [X] I agree to follow Percona Community Code of Conduct
@MostFrabjous MostFrabjous added the bug Bug report label Apr 17, 2024
@BupycHuk
Copy link
Member

Hi, are the logs from PMM Client or PMM Server? Do you see these errors after upgrading to 2.41.0 or with 2.34.0?
Our compatability policy for PMM currently is supporting at least 5 minor releases of PMM Client.
I've found similar issue with VictoriaMetrics, not sure if it will help you VictoriaMetrics/VictoriaMetrics#76 (comment)

@MostFrabjous
Copy link
Author

After searching again, I found this VictriaMetrics Issue: VictoriaMetrics/VictoriaMetrics#5301

@MostFrabjous
Copy link
Author

MostFrabjous commented Apr 19, 2024

Hi, are the logs from PMM Client or PMM Server?

These are server logs

Do you see these errors after upgrading to 2.41.0 or with 2.34.0?

After upgrading to 2.41.0 finished and PMM server starts up again. Errors are visible as soon as pmm-agents are sending data.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Bug report
Projects
None yet
Development

No branches or pull requests

2 participants