Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding benchmarks for different connectors and refactoring for table states/feature frame/transforms #39

Open
wants to merge 15 commits into
base: main
Choose a base branch
from

Conversation

jeffcheng1234
Copy link
Collaborator

@jeffcheng1234 jeffcheng1234 commented Jun 12, 2022

Adding a benchmark test to get the latencies and throughputs for different connector modes (in memory dictionary, redis, sqlite). Current benchmarking test is running on a single ray actor with different test sizes, suggesting in memory dictionary to be optimal. Requires more robust testing.

Modified the interface for feature frame and transform objects. Feature frame should be responsible for maintaining the different resources it holds (transform object, table state, and scheduler). All query operations called on the transform_object will under the hood query the feature frame that the transform_object belongs to, which will query the table_state object it holds.

Note: This implementation currently assumes a 1:1 mapping of transform objects and feature frame. (For simplicity, we're currently not considering more complex transforms like a join or anything that involves more than one upstream parent)

@jeffcheng1234 jeffcheng1234 changed the title Adding queryable interface for feature frame and benchmarking different connectors Adding benchmarks for different connectors and refactoring for table states/feature frame/transforms Jun 21, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant