New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reduce heap usage for small record #831

Open

SCNieh opened this issue Dec 14, 2023 · 1 comment

Assignees

Labels

Contributor

SCNieh commented Dec 14, 2023

the heap usage of BlockCache and LogCache will significantly increase due to the large number of StreamRecordBatch when record size is small

SCNieh added the enhancement label

SCNieh self-assigned this

Contributor Author

SCNieh commented Feb 19, 2024

Reduce BlockCache heap usage by directly put data block into cache block and parsing it into StreamRecordBatch on demand
Reduce LogCache heap usage by directly put encoded StreamRecordBatch into StreamCache and decode for indexing on binary search (or cache indices for acceleration by sacrificing more heap memory)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment