Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

/go/libraries/doltcore/sql/dsess: parallelize sql.NewDatabase work #8740

Merged
merged 10 commits into from
Jan 14, 2025

Conversation

coffeegoddd
Copy link
Contributor

No description provided.

@coffeegoddd coffeegoddd requested review from zachmu and reltuk January 13, 2025 23:46
@coffeegoddd coffeegoddd force-pushed the db/parallelize branch 5 times, most recently from b3ae4da to 4637ccf Compare January 14, 2025 01:03
@coffeegoddd
Copy link
Contributor Author

@coffeegoddd DOLT

comparing_percentages
100.000000 to 100.000000
version result total
fa6c12f ok 5937457
version total_tests
fa6c12f 5937457
correctness_percentage
100.0

@coffeegoddd
Copy link
Contributor Author

@coffeegoddd DOLT

comparing_percentages
100.000000 to 100.000000
version result total
53ba7ca ok 5937458
version total_tests
53ba7ca 5937458
correctness_percentage
100.0

@@ -40,35 +41,61 @@ func NewGlobalStateStoreForDb(ctx context.Context, dbName string, db *doltdb.Dol
rootRefs = append(rootRefs, branches...)
rootRefs = append(rootRefs, remotes...)

var roots []doltdb.Rootish
rootRefsChan := make(chan doltdb.Rootish, len(rootRefs))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of a channel here, just allocate roots here:

roots := make([]doltdb.Rootish, len(rootRefs))

and then have the loop capture the iteration variable and write the root into the right place:

for i, b := range rootRefs {
    eg.Go(func() error {
        ...
            roots[i] = cm
        ...
            roots[i] = ws
        ...
    })
}

Probably best to eg.SetLimit() on this errgroup as well.

Copy link
Contributor

@reltuk reltuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. One suggestion on improving the code.

@coffeegoddd
Copy link
Contributor Author

@coffeegoddd DOLT

comparing_percentages
100.000000 to 100.000000
version result total
a819aa2 ok 5937457
version total_tests
a819aa2 5937457
correctness_percentage
100.0

@coffeegoddd coffeegoddd merged commit 99c7cca into main Jan 14, 2025
21 checks passed
@coffeegoddd
Copy link
Contributor Author

@coffeegoddd DOLT

comparing_percentages
100.000000 to 100.000000
version result total
93355c5 ok 5937457
version total_tests
93355c5 5937457
correctness_percentage
100.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants