[EXP][C++] Deduplicate schemas when scanning Dataset

[pitrou]

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

[pitrou]

@ursabot please benchmark

[ursabot]

Benchmark runs are scheduled for commit f681035. Watch https://buildkite.com/apache-arrow and https://conbench.ursa.dev for updates. A comment will be posted here when the runs are complete.

[pitrou]

@ursabot please benchmark

[ursabot]

Benchmark runs are scheduled for commit a0503f3. Watch https://buildkite.com/apache-arrow and https://conbench.ursa.dev for updates. A comment will be posted here when the runs are complete.

[pitrou]

@icexelloss This is a quick experiment that you might want to try out on a real-world use case. I don't seem to get any tangible benefits on a synthetic dataset, though it might be due to memory fragmentation.

[conbench-apache-arrow]

Thanks for your patience. Conbench analyzed the 0 benchmarking runs that have been run so far on PR commit f681035.

None of the specified runs were found on the Conbench server.

The full Conbench report has more details.

[conbench-apache-arrow]

Thanks for your patience. Conbench analyzed the 4 benchmarking runs that have been run so far on PR commit a0503f3.

There were 775 benchmark results indicating a performance regression:

• Pull Request Run on test-mac-arm at 2025-01-23 18:46:18Z
  • TakeChunkedChunkedStringFewMonotonicIndices (C++) with params=4194304/1, source=cpp-micro, suite=arrow-compute-vector-selection-benchmark
  • ListSliceStringListViewWithStop (C++) with params=65536/0, source=cpp-micro, suite=arrow-compute-scalar-list-benchmark
• and 773 more (see the report linked below)

The full Conbench report has more details.

[pitrou]

Ok, there are so many unrelated "regressions" in the report above that I'm going to launch another benchmarking run, as it's likely that external factors have influenced that run.

[pitrou]

@ursabot please benchmark

[ursabot]

Commit a0503f3 already has scheduled benchmark runs.

[pitrou]

@ursabot please benchmark

[ursabot]

Benchmark runs are scheduled for commit 335a46b. Watch https://buildkite.com/apache-arrow and https://conbench.ursa.dev for updates. A comment will be posted here when the runs are complete.

[conbench-apache-arrow]

Thanks for your patience. Conbench analyzed the 4 benchmarking runs that have been run so far on PR commit 335a46b.

There were 23 benchmark results indicating a performance regression:

• Pull Request Run on test-mac-arm at 2025-01-24 15:13:21Z
  • BenchmarkTemporal (C++) with params=<Subsecond, non_zoned>/4194304/0, source=cpp-micro, suite=arrow-compute-scalar-temporal-benchmark
  • FilterOverhead (C++) with params=selectivity_benchmark/batch_size:100000/null_prob:100/bool_true_prob:50/real_time, source=cpp-micro, suite=arrow-acero-filter-benchmark
• and 21 more (see the report linked below)

The full Conbench report has more details.

[icexelloss]

@pitrou Thanks! cc @timothydijamco

[timothydijamco]

Thanks for this PR, seems like a good idea to try out.

In response to #45287 (comment): I didn't observe any difference between a version of Arrow with the metadata-clearing patch (#45330) vs a version of Arrow with the metadata-clearing patch plus this patch here in this PR.

Synthetic data

500 files, each with 1 row and 10,000 columns with 200-character-long column names

Peak memory

Performing a "scan" or loading the table into memory

	Only metadata-clearing (#45330)	With metadata-clearing (#45330) and schema deduplication (#45340)
One "scan" (pull batches from scanner->RecordBatchReader() until exhausted)	1.49GB	1.48GB
One scanner->ToTable()	1.57GB	1.55GB

Memory profiles

Performing two "scans"

Only metadata-clearing (#45330)	With metadata-clearing (#45330) and schema deduplication (#45340)

Real data

I ran on a variety of real datasets we have internally (data size varies from <1GB of data to 40GB of data, num columns varies from hundreds to thousands, number of files varies from 1 to hundreds) in the "scan" use case and "load table" use case and did not observe any memory usage difference between the two Arrow versions as well