go/store/datas/pull: Create a PullChunkTracker for keeping track of what to pull.

[reltuk]

The PullChunkTracker is an optimization which can concurrently call HasMany on the destination database while chunks are being pulled from the source database.

[coffeegoddd]

@reltuk DOLT

comparing_percentages
100.000000 to 100.000000

version	result	total
c5afe61	ok	5937457

version	total_tests
c5afe61	5937457

correctness_percentage
100.0

[coffeegoddd]

@reltuk DOLT

comparing_percentages
100.000000 to 100.000000

version	result	total
1a6bf25	ok	5937457

version	total_tests
1a6bf25	5937457

correctness_percentage
100.0

[coffeegoddd]

@reltuk DOLT

comparing_percentages
100.000000 to 100.000000

version	result	total
383a196	ok	5937457

version	total_tests
383a196	5937457

correctness_percentage
100.0

[macneale4]

A couple comments, and suggested an additional test

[macneale4]

Where is 3 coming from? Were you able to test a few different thread counts and settle on this one using data?

[macneale4]

LGTM!

[coffeegoddd]

@reltuk DOLT

comparing_percentages
100.000000 to 100.000000

version	result	total
dce814d	ok	5937457

version	total_tests
dce814d	5937457

correctness_percentage
100.0

[coffeegoddd]

@coffeegoddd DOLT

comparing_percentages
100.000000 to 100.000000

version	result	total
ccb3dd3	ok	5937457

version	total_tests
ccb3dd3	5937457

correctness_percentage
100.0

[github-actions]

@coffeegoddd DOLT

test_name	detail	row_cnt	sorted	mysql_time	sql_mult	cli_mult
batching	LOAD DATA	10000	1	0.05	0.6
batching	batch sql	10000	1	0.06	1.67
batching	by line sql	10000	1	0.06	1.67
blob	1 blob	200000	1	0.9	3.03	3.44
blob	2 blobs	200000	1	0.89	3.84	4.6
blob	no blob	200000	1	0.9	1.26	1.36
col type	datetime	200000	1	0.8	1.78	1.94
col type	varchar	200000	1	0.66	1.86	1.88
config width	2 cols	200000	1	0.81	1.2	1.25
config width	32 cols	200000	1	1.83	1.39	2.51
config width	8 cols	200000	1	0.94	1.33	1.67
pk type	float	200000	1	0.9	1.28	1.14
pk type	int	200000	1	0.78	1.37	1.32
pk type	varchar	200000	1	1.47	1.03	1.05
row count	1.6mm	1600000	1	5.55	1.48	1.54
row count	400k	400000	1	1.41	1.4	1.46
row count	800k	800000	1	2.87	1.39	1.46
secondary index	four index	200000	1	3.51	1.08	0.94
secondary index	no secondary	200000	1	0.87	1.25	1.41
secondary index	one index	200000	1	1.12	1.43	1.47
secondary index	two index	200000	1	2	1.19	1.1
sorting	shuffled 1mm	1000000	0	4.94	1.77	1.76
sorting	sorted 1mm	1000000	1	4.89	1.79	1.77

[github-actions]

@coffeegoddd DOLT

name	detail	mean_mult
dolt_blame_basic	system table	1.39
dolt_blame_commit_filter	system table	3.58
dolt_commit_ancestors_commit_filter	system table	0.84
dolt_commits_commit_filter	system table	0.91
dolt_diff_log_join_from_commit	system table	2.07
dolt_diff_log_join_to_commit	system table	2.11
dolt_diff_table_from_commit_filter	system table	1.15
dolt_diff_table_to_commit_filter	system table	1.13
dolt_diffs_commit_filter	system table	1.03
dolt_history_commit_filter	system table	1.45
dolt_log_commit_filter	system table	0.84

[github-actions]

@coffeegoddd DOLT

name	add_cnt	delete_cnt	update_cnt	latency
adds_only	60000	0	0	0.73
adds_updates_deletes	60000	60000	60000	3.86
deletes_only	0	60000	0	1.86
updates_only	0	0	60000	2.44