Reduce memory usage of serve-testing server in large test suites

[mateenkasim]

Problem Statement

I wasn't sure how to label this as it's more of a concern than a bug or proposal, and it's not high priority for me.
Previous discussions in Discord:

• https://discord.com/channels/844600078504951838/1278749294349910097
• https://discord.com/channels/844600078504951838/1278750115057893418

In many testing paradigms, the DB state is torn down between tests. SpiceDB's serve-testing server mimics this by granting siloed DB states for every unique secret key. However, the server doesn't actually delete any of the data, since it doesn't know when the tests have completed. Because it's a memory datastore, this means memory usage increases monotonically.

Between test suite runs, memory can be cleared by restarting the test SpiceDB instance. However, within a test suite, memory usage may quickly increase if some tests write a lot of relationships. Some mechanism to "tear down" those relationships at the end of the test would be awesome. Doing it by manually tracking writes and running DeleteRelationships would be possible, except for this comment by @vroldanbet:

Deleting won't really reduce memory usage because of the revisioning system.

Solution Brainstorm

No response

[josephschorr]

@mateenkasim The problem is knowing when to "invalidate" a test has completed. We could add a custom API just for serve-testing, but that would mean the clients only having that API for testing. Alternatively, we could have a signal via the existing API, but that would be a hack (something like "this datastore is complete when WriteSchema is called with an empty schema").

What would you prefer?

[mateenkasim]

I'd personally prefer your first solution – seems clearer and easier to debug if someone misuses it, especially if it had a distinctive name or fell under an obvious rpc service like TestService.TearDown. If it did have its own service, you could have it so the service isn't even running (health check to that service would fail) unless the server is serve-testing. Not necessary though, just a thought.

Thanks for your help!

[josephschorr]

@mateenkasim Thoughts on something like this?

service TestingService {
  // Bootstrap bootstraps a test context from a schema and a set of relationships
  rpc Bootstrap(BootstrapRequest) returns (BootstrapResponse) {}

  // TearDown tears down a test context, removing all of its data
  rpc TearDown(TearDownRequest) returns (TearDownResponse) {}

  // Clone clones a test context into another test context
  rpc Clone(CloneRequest) returns (CloneResponse) {}
}

message BootstrapRequest {
   string schema_text = 1;
   repeated core.Relationship relationships = 2;
}

message BootstrapResponse {
  ZedToken written_at = 1;
}

message TearDownRequest {
  string token = 1;
}

message TearDownResponse {}

message CloneRequest {
  string source_token = 1;
  string target_token = 2;
}

message CloneResponse {
    ZedToken written_at = 1;
}

[mateenkasim]

That looks great – Bootstrap and Clone cover all the setup cases I can think of, and TearDown looks as easy as it gets! Although TearDownRequest.token and CloneRequest.target_token are redundant with the token used by the client for auth, right?

[josephschorr]

That looks great – Bootstrap and Clone cover all the setup cases I can think of, and TearDown looks as easy as it gets! Although TearDownRequest.token and CloneRequest.target_token are redundant with the token used by the client for auth, right?

Yeah, they are. Good point