GH-41816: [C++] Meson Build System Support #45441

WillAyd · 2025-02-06T00:45:37Z

Rationale for this change

The Meson build system may be more user friendly to some developers, and may make it easier to perform tasks like valgrind, coverage, or ASAN/UBSAN coverage. There is also a prior art for using meson in the nanoarrow and arrow-adbc projects.

What changes are included in this PR?

This PR implements a Meson configuration that can support the ARROW_BUILD_TESTS and ARROW_COMPUTE options, as well as the dependencies for those options

Are these changes tested?

Not in CI. You can download and run meson test when built to run the test suite.

Are there any user-facing changes?

No. This is strictly for developers

GitHub Issue: [C++] Meson Support for Arrow #41816

WillAyd · 2025-02-06T00:51:32Z

@kou this is a simplified attempt at adding Meson to address your comment #41816 (comment)

This is currently slower than the CMake configuration by a good deal (pending investigation) and the configuration file is not 100% complete, but this should give us an idea of what a Meson configuration may look like.

To use, from the cpp directory developers can:

meson setup builddir -Dtests=true -Dcompute=true
meson compile -C builddir
meson test -C builddir

For ASAN/UBSAN, users could simply:

meson setup builddir -Dtests=true -Dcompute=true -Db_sanitize=address,undefined

Or if the project is already setup run:

meson configure -C builddir -Db_sanitize=address,undefined

Coverage can be enabled with:

meson configure -C builddir -Db_coverage=true

and tests can be run under valgrind with:

meson test -C builddir --wrap='valgrind --track-origins=yes --leak-check=full' --print-errorlog

kou · 2025-02-06T02:15:28Z

Thanks.

Can we start from more simplified version? For example, we don't need libarrow_testing, tests and so on at the first step.

We want to add a nightly CI for this to detect regression.

We want to update version information automatically in release process. For example:

arrow/dev/release/utils-prepare.sh

Lines 39 to 44 in 0556905

    
           pushd "${ARROW_DIR}/c_glib" 
        
           sed -i.bak -E -e \ 
        
             "s/^version = '.+'/version = '${version}'/" \ 
        
             meson.build 
        
           rm -f meson.build.bak 
        
           git add meson.build

(I can do it in this branch later.)

Anyway, I didn't know that meson format exists! I want to use it in c_glib/ too.

WillAyd · 2025-02-06T16:33:54Z

Sure I can strip down further. So do think just something that builds libarrow is the right starting point?

kou · 2025-02-07T01:39:16Z

Yes. Only minimal libarrow (e.g. no compute, no filesystem and so on) is the right starting point.

Remove extraneous files

WillAyd

Thanks @kou . Hope this is more in line with your expectation.

As far as nightlies go, can you point me to an existing nightly CI setup in the repo? I was able to grep for some R nightlies, but not sure if there is existing infrastructure for C++ nightly jobs where this would be better placed

WillAyd · 2025-02-11T21:08:13Z

cpp/src/arrow/meson.build

+    objects: objlibs,
+    include_directories: [include_dir],
+    install: true,
+    # compute/expression.cc may have undefined IPC symbols in non-IPC builds


Wasn't sure if this was intentional or not in the existing code base

Ah, we should disable features that depend on IPC in cpp/src/arrow/compute/expression.cc like GH-45171.

Could you open an issue for this?

No problem - see #45512

WillAyd · 2025-02-11T21:09:22Z

cpp/src/arrow/meson.build

+    override_options: {'b_lundef': 'false'},
+)
+
+# Meson does not allow you to glob for headers to install. See also


Meson is pretty strict about wildcards for the reasons outlined in their FAQ. Idiomatically, Meson would want you to put the files you want to install in a separate directory and call install_subdir, but that would go beyond the scope of this initial PR I think

WillAyd · 2025-02-11T21:09:48Z

cpp/src/arrow/meson.build

+arrow_so_version = (ver_major.to_int() * 100 + ver_minor.to_int()).to_string()
+arrow_full_so_version = '@0@.@1@.@2@'.format(arrow_so_version, ver_patch, 0)
+
+# TODO: The Meson generated .pc file does not include the Apache license


I did not research this in too much detail yet; figured I'd check if it was a big deal before investing time into a resolution

We don't need it.
arrow.pc.in has the license header because it's in this repository. Files that don't exist in this repository don't need it.

cpp/src/arrow/util/meson.build

kou

Can I push some commits to this branch for nightly CI and auto version update?

kou · 2025-02-12T01:16:35Z

cpp/meson.build

+    'cpp',
+    'c',
+    version: '19.0.0-SNAPSHOT',
+    license: 'Apache 2.0',


Suggested change

license: 'Apache 2.0',

license: 'Apache-2.0',

kou · 2025-02-12T01:26:44Z

cpp/meson.build

+
+git_id = get_option('git_id')
+if git_id == ''
+    git_id = run_command('git', 'log', '-n1', '--format=%H', check: true).stdout().strip()


Can we ignore git log error? This will fail when we use source archive.

Suggested change

git_id = run_command('git', 'log', '-n1', '--format=%H', check: true).stdout().strip()

git_id = run_command('git', 'log', '-n1', '--format=%H').stdout().strip()

kou · 2025-02-12T01:27:03Z

cpp/meson.build

+
+git_description = get_option('git_description')
+if git_description == ''
+    git_description = run_command('git', 'describe', '--tags', check: true).stdout().strip()


Suggested change

git_description = run_command('git', 'describe', '--tags', check: true).stdout().strip()

git_description = run_command('git', 'describe', '--tags').stdout().strip()

kou · 2025-02-12T01:29:14Z

cpp/src/arrow/meson.build

+
+# Meson does not natively support object libraries
+# https://github.com/mesonbuild/meson/issues/13843
+objlib_sources = {


We don't need to emulate objlib in Meson. It's just for faster build.
We can just use library().

I have a pretty poor understanding of how object libraries actually work, but I don't think we can just convert these to using the library call unless we want the linker to allow undefined symbols in these libraries as well (?)

Trying to do what I think you suggest generates a huge amount of errors like:

[7/96] Linking target src/arrow/libarrow_io.so FAILED: src/arrow/libarrow_io.so c++ -o src/arrow/libarrow_io.so src/arrow/libarrow_io.so.p/io_buffered.cc.o src/arrow/libarrow_io.so.p/io_caching.cc.o src/arrow/libarrow_io.so.p/io_compressed.cc.o src/arrow/libarrow_io.so.p/io_file.cc.o src/arrow/libarrow_io.so.p/io_hdfs.cc.o src/arrow/libarrow_io.so.p/io_hdfs_internal.cc.o src/arrow/libarrow_io.so.p/io_interfaces.cc.o src/arrow/libarrow_io.so.p/io_memory.cc.o src/arrow/libarrow_io.so.p/io_slow.cc.o src/arrow/libarrow_io.so.p/io_stdio.cc.o src/arrow/libarrow_io.so.p/io_transform.cc.o -Wl,--as-needed -Wl,--no-undefined -Wl,-O1 -shared -fPIC -Wl,--start-group -Wl,-soname,libarrow_io.so -Wl,--end-group /usr/bin/ld: src/arrow/libarrow_io.so.p/io_buffered.cc.o: in function `arrow::io::BufferedInputStream::SetBufferSize(long)': buffered.cc:(.text+0x13a9): undefined reference to `arrow::Buffer::CheckCPU() const' /usr/bin/ld: buffered.cc:(.text+0x13b1): undefined reference to `arrow::Buffer::CheckMutable() const' /usr/bin/ld: buffered.cc:(.text+0x1407): undefined reference to `arrow::util::detail::StringStreamWrapper::StringStreamWrapper()' /usr/bin/ld: buffered.cc:(.text+0x149e): undefined reference to `arrow::util::detail::StringStreamWrapper::str[abi:cxx11]()' /usr/bin/ld: buffered.cc:(.text+0x14a6): undefined reference to `arrow::util::detail::StringStreamWrapper::~StringStreamWrapper()' /usr/bin/ld: buffered.cc:(.text+0x14b6): undefined reference to `arrow::Status::Status(arrow::StatusCode, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'

Ah, we can remove objlib_sources and static_library() for them entirely.
We can just add sources in objlib_sources to arrow_srcs and use it in one library() (that already exists in this PR).

(We don't need objlib feature entirely with Meson.)

Makes sense. I've still kept the top level dict for parity with CMake, and since some of the sources vary depending on compilation options (like arrow compute). Let me know if this is more in line with what you are thinking

kou · 2025-02-12T01:32:57Z

cpp/src/arrow/meson.build

+    objects: objlibs,
+    include_directories: [include_dir],
+    install: true,
+    # compute/expression.cc may have undefined IPC symbols in non-IPC builds


Ah, we should disable features that depend on IPC in cpp/src/arrow/compute/expression.cc like GH-45171.

Could you open an issue for this?

kou · 2025-02-12T02:21:26Z

cpp/src/arrow/meson.build

+arrow_so_version = (ver_major.to_int() * 100 + ver_minor.to_int()).to_string()
+arrow_full_so_version = '@0@.@1@.@2@'.format(arrow_so_version, ver_patch, 0)
+
+# TODO: The Meson generated .pc file does not include the Apache license


We don't need it.
arrow.pc.in has the license header because it's in this repository. Files that don't exist in this repository don't need it.

cpp/src/arrow/util/meson.build

kou · 2025-02-12T02:26:29Z

cpp/src/arrow/util/meson.build

+    configuration: conf_data,
+    format: 'cmake@',
+    install: true,
+    install_dir: 'arrow',


Suggested change

install_dir: 'arrow',

install_dir: 'arrow/util',

kou · 2025-02-12T02:27:11Z

cpp/src/arrow/util/meson.build

+foreach cmakedefine : [
+    'ARROW_COMPUTE',
+    'ARROW_CSV',
+    'ARROW_CUDA',
+    'ARROW_DATASET',
+    'ARROW_FILESYSTEM',
+    'ARROW_FLIGHT',
+    'ARROW_FLIGHT_SQL',
+    'ARROW_IPC',
+    'ARROW_JEMALLOC',
+    'ARROW_JEMALLOC_VENDORED',
+    'ARROW_JSON',
+    'ARROW_MIMALLOC',
+    'ARROW_ORC',
+    'ARROW_PARQUET',
+    'ARROW_SUBSTRAIT',
+    'ARROW_AZURE',
+    'ARROW_ENABLE_THREADING',
+    'ARROW_GCS',
+    'ARROW_HDFS',
+    'ARROW_S3',
+    'ARROW_USE_GLOG',
+    'ARROW_USE_NATIVE_INT128',
+    'ARROW_WITH_BROTLI',
+    'ARROW_WITH_BZ2',
+    'ARROW_WITH_LZ4',
+    'ARROW_WITH_MUSL',
+    'ARROW_WITH_OPENTELEMETRY',
+    'ARROW_WITH_RE2',
+    'ARROW_WITH_SNAPPY',
+    'ARROW_WITH_UCX',
+    'ARROW_WITH_UTF8PROC',
+    'ARROW_WITH_ZLIB',
+    'ARROW_WITH_ZSTD',
+    'PARQUET_REQUIRE_ENCRYPTION',
+]
+    conf_data.set(cmakedefine, false)
+endforeach


It seems that we don't need to use foreach here:

conf_data.set('ARROW_COMPUTE', false) conf_data.set('ARROW_CSV', false) ...

kou · 2025-02-12T02:27:47Z

cpp/src/arrow/util/meson.build

+    install: true,
+    install_dir: 'arrow',


We don't need to install _internal.h:

Suggested change

install: true,

install_dir: 'arrow',

WillAyd · 2025-02-12T02:47:53Z

Sure no problem

WillAyd marked this pull request as draft February 6, 2025 00:45

github-actions bot added Component: C++ awaiting review Awaiting review labels Feb 6, 2025

WillAyd force-pushed the meson-builds branch from 54ece5a to 2f7048f Compare February 6, 2025 00:53

WillAyd added 2 commits February 11, 2025 14:10

apacheGH-41816: [C++] Meson Build System Support

c3691b2

Build minimal library without options

cbed8e3

Remove extraneous files

WillAyd force-pushed the meson-builds branch from 80c606c to 3760159 Compare February 11, 2025 19:12

WillAyd added 2 commits February 11, 2025 16:03

Complete configuration file

f80bf10

Install headers and add pkgconfig support

61f4bbb

WillAyd force-pushed the meson-builds branch from c492403 to 61f4bbb Compare February 11, 2025 21:03

WillAyd commented Feb 11, 2025

View reviewed changes

github-actions bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Feb 11, 2025

kou reviewed Feb 12, 2025

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Feb 12, 2025

WillAyd mentioned this pull request Feb 12, 2025

Minimal Arrow Library Contains Undefined Symbols from IPC sources #45512

Open

kou feedback

b9743e9

github-actions bot added awaiting change review Awaiting change review awaiting changes Awaiting changes and removed awaiting changes Awaiting changes awaiting change review Awaiting change review labels Feb 12, 2025

Accumulate sources, dependencies, and includes

3ee1e54

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Feb 13, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-41816: [C++] Meson Build System Support #45441

GH-41816: [C++] Meson Build System Support #45441

WillAyd commented Feb 6, 2025 •

edited by github-actions bot

Loading

WillAyd commented Feb 6, 2025

kou commented Feb 6, 2025

WillAyd commented Feb 6, 2025

kou commented Feb 7, 2025

WillAyd left a comment

WillAyd Feb 11, 2025

kou Feb 12, 2025

WillAyd Feb 12, 2025

WillAyd Feb 11, 2025

WillAyd Feb 11, 2025

kou Feb 12, 2025

kou left a comment

kou Feb 12, 2025

kou Feb 12, 2025

kou Feb 12, 2025

kou Feb 12, 2025

WillAyd Feb 12, 2025

kou Feb 13, 2025

kou Feb 13, 2025

WillAyd Feb 13, 2025

kou Feb 12, 2025

kou Feb 12, 2025

kou Feb 12, 2025

kou Feb 12, 2025

kou Feb 12, 2025

WillAyd commented Feb 12, 2025

	git_id = run_command('git', 'log', '-n1', '--format=%H', check: true).stdout().strip()
	git_id = run_command('git', 'log', '-n1', '--format=%H').stdout().strip()

	git_description = run_command('git', 'describe', '--tags', check: true).stdout().strip()
	git_description = run_command('git', 'describe', '--tags').stdout().strip()

GH-41816: [C++] Meson Build System Support #45441

Are you sure you want to change the base?

GH-41816: [C++] Meson Build System Support #45441

Conversation

WillAyd commented Feb 6, 2025 • edited by github-actions bot Loading

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

WillAyd commented Feb 6, 2025

kou commented Feb 6, 2025

WillAyd commented Feb 6, 2025

kou commented Feb 7, 2025

WillAyd left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kou left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WillAyd commented Feb 12, 2025

WillAyd commented Feb 6, 2025 •

edited by github-actions bot

Loading