Initial pass at a "prepare" step and minibatch extract #15
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mcovarr
reviewed
Sep 15, 2022
Comment on lines
48
to
50
| features = [] | ||
| for row in query: | ||
| features.append(Feature(row["cas_feature_index"], row["feature_name"])) |
There was a problem hiding this comment.
TOL the comprehension version of this logic on line 67 is nice... 🙂
| CAST(FLOOR(rand() * (select count(1) from `{project}.{dataset}.cas_cell_info`) / {extract_bin_size}) as INT) as extract_bin | ||
| FROM `{project}.{dataset}.cas_cell_info` c | ||
| ORDER BY cas_cell_index | ||
| """ |
There was a problem hiding this comment.
This seems to produce a nice set of roughly equally sized bins, except for that last one...
There was a problem hiding this comment.
I can add a comment about this, but there will be a "remainder" bin that is smaller than the rest. There are other approaches, but a reasonable place to start for now and a we learn more about the ML requirements we can optimize. Ie how important is it for the bin to be the exact same size, vs what you asked for, etc.
Accepted suggestions Co-authored-by: Miguel Covarrubias <[email protected]>
mcovarr
approved these changes
Sep 19, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Would like some eyes on the code, and then I'll update the readme/workflow!
Initial scale and performance testing - 4m cell dataset
Prepare: ~1 minute to prepare 4m cells for extract costing ~$1.30.
Extract: ~5 minutes to extract 100k cells to AnnData (700 mb) costing ~$0.003.
Expected linear scaling to 1 billion cells would cost < $355 ( $325 + $30 )