Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Loki zstd compression - memory leak due to increasing go-routines #15594

Open
bharath-sadhu opened this issue Jan 5, 2025 · 0 comments
Open
Labels
type/bug Somehing is not working as expected

Comments

@bharath-sadhu
Copy link

bharath-sadhu commented Jan 5, 2025

Hi Team,

zstd compression in loki is causing memory leaks

goroutine profile: total 7031
2211 @ 0x47526e 0x40a05c 0x409c32 0xc0fe7b 0x47dac1
#	0xc0fe7a	github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder.func2+0x9a	/src/loki/vendor/github.com/klauspost/compress/zstd/decoder.go:714

2211 @ 0x47526e 0x40a05c 0x409c32 0xc105b9 0x47dac1
#	0xc105b8	github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder.func1+0x78	/src/loki/vendor/github.com/klauspost/compress/zstd/decoder.go:658

2193 @ 0x47526e 0x450d25 0xc0f88e 0x47dac1
#	0xc0f88d	github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder+0x7ad	/src/loki/vendor/github.com/klauspost/compress/zstd/decoder.go:872

Above is the go-routine profile for one of the querier running in k8s with 44 GB memory utilisation with out any queries running now, and in that querier out of total 7031 goroutines running 95% are from zstd and they are not getting closed properly causing memory to increase in queriers

@bharath-sadhu bharath-sadhu changed the title Loki zstd compression - memory leaks due to increasing go-routines Loki zstd compression - memory leak due to increasing go-routines Jan 5, 2025
@JStickler JStickler added the type/bug Somehing is not working as expected label Jan 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/bug Somehing is not working as expected
Projects
None yet
Development

No branches or pull requests

2 participants