Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Range queries is leading to memory growth #324

Open
dremekie opened this issue Jan 22, 2025 · 1 comment
Open

Range queries is leading to memory growth #324

dremekie opened this issue Jan 22, 2025 · 1 comment

Comments

@dremekie
Copy link

dremekie commented Jan 22, 2025

Hi. This is a great library and well-documented. I have a question I hope you can help with and would definitely help others.

The following is a minimal example that demonstrates my issue. The code iterates through batches of 100 keys and removes them from the database. The memory usage grows steadily until all keys are removed. Once all keys are removed, memory usage remains the same. Memory is being retained and not being released. When I execute the code, there is a steady climb from <50M to >490M memory. Once all keys are removed (over 110,000), memory growth stops but memory usage remains the same at > 490M.

Is there anything you can see that I am doing wrong?

import { open } from 'lmdb';
const opts = { path: './testdb', compression: true };

const db = open(opts);
const wait = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
while (true) {
  await wait(50);

  // Collect keys
  let keys = [];
  const iter = db.getKeys({ limit: 100, snapshot: true });
  for (const key of iter) {
    keys.push(key);
  }

  // Remove keys
  keys.forEach((key) => {
    db.remove(key);
    console.log('Key:', key);
  });
  keys = undefined;
}

await db.close();
@dremekie
Copy link
Author

Just to add, when I comment out db.remove(key); (which means it continually fetches the same 100 keys), there is zero memory growth.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant