Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Track energy for persistent database memory #667

Closed
joshua-spacetime opened this issue Dec 13, 2023 · 4 comments · Fixed by #766
Closed

Track energy for persistent database memory #667

joshua-spacetime opened this issue Dec 13, 2023 · 4 comments · Fixed by #766
Assignees
@coolreader18

Comments

@joshua-spacetime
Copy link
Collaborator

joshua-spacetime commented Dec 13, 2023 •
edited
Loading

We should track the persistent memory associated with tables and indexes for energy purposes. This means each table and index needs access to an EnergyMonitor for recording energy use.

If 1 eV is the amount of energy used to store 1 byte on disk for 1 second, then N eV is the amount of energy required to store 1 byte in memory for 1 second for some N. We should pick a suitable N and record, for every insert/delete, the amount of energy used per table, since the previous insert/delete.

As part of this ticket we should also write these energy recordings to prometheus.
@cloutiertyler
Copy link
Contributor

cloutiertyler commented Dec 13, 2023 •
edited
Loading

100 is not so far from a suitable N (for context)

joshua-spacetime added a commit that referenced this issue Jan 12, 2024
@joshua-spacetime
chore: Move table size metrics back into core
e4d3d21 
Table size metrics were previously moved out of core.
This was due to the query planner needing access to them.
However that dependency was ultimately managed differently via #667.
@coolreader18 coolreader18 linked a pull request Jan 26, 2024 that will close this issue
Energy metering for persistent memory usage #766
Merged
@cloutiertyler
Copy link
Contributor

We would like to see the number of bytes per table so that we can track which tables have very large rows.

@gefjon
Copy link
Contributor

gefjon commented Apr 11, 2024

We would like to see the number of bytes per table so that we can track which tables have very large rows.

It would be very easy to compute a conservative overestimate of this by simply tracking the number of pages per table in the committed state. A precise count is more challenging.

@cloutiertyler cloutiertyler changed the title Track energy for persistent database memory Track memory usage for tables in grafana & track energy for persistent database memory Apr 16, 2024
@cloutiertyler cloutiertyler changed the title Track memory usage for tables in grafana & track energy for persistent database memory Track energy for persistent database memory May 28, 2024
@cloutiertyler
Copy link
Contributor

I split this ticket into https://github.com/orgs/clockworklabs/projects/22/views/38?pane=issue&itemId=47389802 as well

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@coolreader18 coolreader18

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Energy metering for persistent memory usage clockworklabs/SpacetimeDB
5 participants
@cloutiertyler @gefjon @coolreader18 @Shubham8287 @joshua-spacetime