Distributed request to tables with Object Storage Engines #615

ianton-ru · 2025-02-06T19:17:53Z

Changelog category (leave one):

New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Distributed object storage table engines

Documentation entry for user-facing changes

Before ClickHouse can made distributed table function requests with option object_storage_cluster.

SELECT * FROM icebergS3('http://minio:9000/warehouse/data/', 'minio', 'minio123', 'Parquet') SETTINGS object_storage_cluster='cluster';

This PR expand this to table engined.

CREATE TABLE t (key UInt32, value String) ENGINE=IcebergS3('http://minio:9000/warehouse/data/', 'minio', 'minio123', 'Parquet') SETTINGS object_storage_cluster='cluster';

SELECT * FROM t;

CREATE DATABASE datalake ENGINE = Iceberg('http://rest:8181/v1', 'minio', 'minio123') SETTINGS catalog_type = 'rest', storage_endpoint = 'http://minio:9000/warehouse', warehouse = 'iceberg', object_storage_cluster='cluster';

SELECT * FROM datalake.`iceberg.bids`;

Modify your CI run:

NOTE: If your merge the PR with modified CI you MUST KNOW what you are doing
NOTE: Checked options will be applied if set before CI RunConfig/PrepareRunConfig step

Include tests (required builds will be added automatically):

Exclude tests:

Extra options:

do not test (only style check)
disable merge-commit (no merge from master before tests)
disable CI cache (job reuse)

Only specified batches in multi-batch jobs:

1
2
3
4

altinity-robot · 2025-02-06T19:57:21Z

This is an automated comment for commit 17c53f4 with description of existing statuses. It's updated for the latest CI running

❌ Click here to open a full report in a separate page

Check name	Description	Status
Integration tests	The integration tests report. In parenthesis the package type is given, and in square brackets are the optional part/total tests	❌ failure
Regression aarch64 S3 aws_s3	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	❌ failure
Regression aarch64 S3 azure	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	❌ failure
Regression aarch64 S3 gcs	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	❌ failure
Regression aarch64 kafka	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	❌ failure
Sign aarch64	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	❌ error
Sign release	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	❌ error
Stateless tests	Runs stateless functional tests for ClickHouse binaries built in various configurations -- release, debug, with sanitizers, etc	❌ failure
Stress test	Runs stateless functional tests concurrently from several clients to detect concurrency-related errors	❌ failure

Successful checks

Check name	Description	Status
Builds	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Compatibility check	Checks that clickhouse binary runs on distributions with old libc versions. If it fails, ask a maintainer for help	✅ success
Docker keeper image	The check to build and optionally push the mentioned image to docker hub	✅ success
Docker server image	The check to build and optionally push the mentioned image to docker hub	✅ success
Install packages	Checks that the built packages are installable in a clear environment	✅ success
Ready for release	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 Alter attach partition	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 Alter move partition	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 Alter replace partition	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 Benchmark aws_s3	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 Benchmark gcs	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 Benchmark minio	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 Clickhouse Keeper SSL	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 LDAP authentication	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 LDAP external_user_directory	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 LDAP role_mapping	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 Parquet aws_s3	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 Parquet minio	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 Parquet	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 S3 minio	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 Tiered Storage minio	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 Tiered Storage s3amazon	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 Tiered Storage s3gcs	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 aes_encryption	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 aggregate_functions	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 atomic_insert	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 base_58	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 clickhouse_keeper	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 data_types	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 datetime64_extended_range	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 disk_level_encryption	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 dns	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 engines	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 example	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 extended_precision_data_types	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 kerberos	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 key_value	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 lightweight_delete	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 memory	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 part_moves_between_shards	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 selects	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 session_timezone	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 tiered_storage	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression aarch64 window_functions	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release Alter attach partition	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release Alter move partition	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release Benchmark aws_s3	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release Benchmark gcs	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release Benchmark minio	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release Clickhouse Keeper SSL	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release LDAP authentication	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release LDAP external_user_directory	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release LDAP role_mapping	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release Parquet aws_s3	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release Parquet minio	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release Parquet	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release aes_encryption	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release atomic_insert	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release base_58	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release data_types	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release datetime64_extended_range	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release disk_level_encryption	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release dns	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release engines	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release example	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release extended_precision_data_types	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release kafka	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release kerberos	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release key_value	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release memory	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release part_moves_between_shards	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release session_timezone	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Regression release window_functions	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Stateful tests	Runs stateful functional tests for ClickHouse binaries built in various configurations -- release, debug, with sanitizers, etc	✅ success

arthurpassos

At a first glance, it looks good. This is the first round of review, it includes just a few questions and minor suggestions.

Once these are addressed, I'll dive a bit deeper in the AST creation logic and it should be good to go :).

src/Storages/IStorageCluster.h

src/Storages/ObjectStorage/registerStorageObjectStorage.cpp

src/Databases/Iceberg/DatabaseIceberg.cpp

src/Storages/ObjectStorage/S3/Configuration.cpp

src/Storages/ObjectStorage/StorageObjectStorageCluster.cpp

src/Storages/ObjectStorage/StorageObjectStorageCluster.h

src/Storages/ObjectStorage/StorageObjectStorageCluster.cpp

arthurpassos · 2025-02-10T17:42:45Z

src/Storages/ObjectStorage/StorageObjectStorage.h

@@ -247,6 +247,10 @@ class StorageObjectStorage::Configuration

    virtual void update(ObjectStoragePtr object_storage, ContextPtr local_context);

+    virtual void setFunctionArgs(ASTs & /* args */) const


IMO, this function needs to be either renamed or documented. Usually, when I read a function named void setSomething(Argument argument), I assume the members of the current instance will be modified.

This function is doing the opposite, it is taking the members of the existing instance and populating the argumennt.

Maybe you could construct the arguments object and return it?

ASTPtr getTableFunctionArguments() { auto arguments = std::make_shared<ASTExpressionList>(); // populate it return arguments; }

and then, upon call site:

auto function_arguments = configuration->getTableFunctionArguments();

src/Storages/ObjectStorage/StorageObjectStorageCluster.cpp

arthurpassos

Almost there, couple more changes or discussions and it is good to be merged

src/Storages/ObjectStorage/StorageObjectStorageCluster.cpp

arthurpassos · 2025-02-13T19:30:07Z

src/Storages/ObjectStorage/StorageObjectStorageCluster.cpp

@@ -153,7 +160,7 @@ void StorageObjectStorageCluster::updateQueryForDistributedEngineIfNeeded(ASTPtr
            queryToString(query));
    }

-    configuration->setFunctionArgs(arguments->children);
+    configuration->getTableFunctionArguments(arguments->children);


In #615 (comment), I suggested the arguments to be created inside getTableFunctionArguments instead of populating a parameter. Any specific reason you did it this way?

If there is one, I am fine with this approach

Just to avoid copying after function call. children is a vector, not a pointer.

I am not sure I follow.

On line 155, you do: auto arguments = std::make_shared<ASTExpressionList>();

Then you are passing children as a reference on 167 and populating it inside the getTableFunctionArguments function.

On line 169, you read the values.

What I am suggesting is that you create the arguments list inside the getTableFunctionArguments function, populate and simply return it. It'll be a shared_ptr, there is no deep copy in this process. Then, the rest is the same.

What am I missing?

Changes not in arguments but in arguments->children
children has type ASTs
https://github.com/ClickHouse/ClickHouse/blob/master/src/Parsers/IAST.h#L37
It's a variant of vector, not a single pointer
https://github.com/ClickHouse/ClickHouse/blob/master/src/Parsers/IAST_fwd.h#L14

As discussed on slack, let's make the function return it as suggested in the original comment

src/Databases/Iceberg/DatabaseIceberg.cpp

arthurpassos · 2025-02-13T19:34:38Z

src/Storages/ObjectStorage/Azure/Configuration.cpp

+void StorageAzureConfiguration::getTableFunctionArguments(ASTs & args) const
+{
+    if (!args.empty())
+    { /// Just check


I would just remove this comment

Actually, if you implement this function in the way I suggested in #615 (comment), perhaps this whole if statement is no longer necessary?

src/Storages/ObjectStorage/Azure/Configuration.cpp

arthurpassos · 2025-02-13T19:42:49Z

src/Storages/ObjectStorage/S3/Configuration.cpp

+            args.push_back(std::make_shared<ASTLiteral>(auth_settings[S3AuthSetting::session_token].value));
+        if (format != "auto")
+            args.push_back(std::make_shared<ASTLiteral>(format));
+        if (!compression_method.empty())


Is it ok not to add some other arguments like structure and headers as the docs suggest? https://clickhouse.com/docs/en/sql-reference/table-functions/s3

Yes, this fields are already added in addStructureAndFormatToArgsIfNeeded methods.
https://github.com/ClickHouse/ClickHouse/blob/master/src/Storages/ObjectStorage/S3/Configuration.cpp#L399
These field required for table functions too, when access parameters need to add only for table engine case.

Nitpick: perhaps add a comment saying this is already added somewhere else?

I think I rename method to addPathAndAccessKeysToArgs, to consistency with addStructureAndFormatToArgsIfNeeded

src/Storages/ObjectStorage/StorageObjectStorageSource.cpp

arthurpassos

I posted two more comments which are not mandatory, although I am curious about #615 (comment).

In any case, it can be merged as it is. LGTM

arthurpassos · 2025-02-24T20:01:03Z

src/Storages/IStorageCluster.cpp

 {
+    auto cluster_name_ = getClusterName(context);


Why the underscore in the variable name?

Ah, I assume it's because IStorageCluster already has a cluster_name member that comes from engines like S3Cluster and not the setting, right?

If that's so, what happens if the user has both of them set?

PLus, I would rename it to be something like cluster_name_from_settings

Warning that local variable masked class member, and option "warning as error" (-Werror).

arthurpassos

Tomorrow I'll have annother look on the pure_storage member and its auxiliary function getPureStorage

arthurpassos · 2025-02-24T20:04:51Z

src/Storages/IStorageCluster.h

+        ContextPtr context,
+        bool async_insert) override;
+
+    ClusterPtr getCluster(ContextPtr context) const { return getClusterImpl(context, cluster_name); }


Why move the implementation to the header file?

Single line implementation. In my opinion single-line implementation in header is more readable and simple to understanding than separate implementation in cpp-file, when declaration and implementation in two different places.

arthurpassos · 2025-02-24T20:08:50Z

src/Storages/ObjectStorage/StorageObjectStorageCluster.cpp

@@ -136,4 +260,87 @@ RemoteQueryExecutor::Extension StorageObjectStorageCluster::getTaskIteratorExten
    return RemoteQueryExecutor::Extension{ .task_iterator = std::move(callback) };
 }

+std::shared_ptr<StorageObjectStorage> StorageObjectStorageCluster::getPureStorage(ContextPtr context)


nitpick: nonClusteredStorage instead of pureStorage

Just remove method - pure_storage created in constructor anyway, so it always exists.

arthurpassos · 2025-02-24T20:11:45Z

src/Storages/ObjectStorage/StorageObjectStorageCluster.cpp

+
+String StorageObjectStorageCluster::getClusterName(ContextPtr context) const
+{
+    auto cluster_name_ = context->getSettingsRef()[Setting::object_storage_cluster].value;


This is confusing. variable needs to be renamed and some comments need to be written explaining why there are two sources of cluster names.

I feel like there is an easier way to handle this, perhaps at construction time you could decide which cluster name to use and stick to that one?

Variable name: rename the variable. One must be named cluster_name_from_settings, while the other remains intact.

Comment/ Docs: This piece of code includes just one if statement checking two sources of cluster name, it is not hard to understand that. The "tricky" part to understand is WHY this is being done. I would like to see something along these lines: "StorageObjectStorageCluster is always being instantiated, even for non clustered table definitions. But the user can specify the cluster in the query settings, and that must be honored. The xyz setting takes precedence over the table definition one because of abc."

arthurpassos · 2025-02-24T20:15:21Z

src/Storages/ObjectStorage/StorageObjectStorageCluster.h

    const String engine_name;
    const StorageObjectStorage::ConfigurationPtr configuration;
    const ObjectStoragePtr object_storage;
    bool cluster_name_in_settings;
+


Add a comment explaining why these new members are needed

arthurpassos · 2025-02-24T20:26:11Z

src/Storages/ObjectStorage/StorageObjectStorageCluster.h

+        if (pure_storage)
+            pure_storage->setInMemoryMetadata(metadata_);
+        IStorageCluster::setInMemoryMetadata(metadata_);


I assume you are calling the method for both simply because it is more convenient and there are no side effects

src/Storages/ObjectStorage/StorageObjectStorageCluster.cpp

arthurpassos

Correct me if I am wrong, but one thing you are trying to achieve here that wasn't in the original plans is to be able to switch cluster on and off using query settings. Is that correct? For example:

create table test ... engine=IcebergS3 -- no cluster settings here
select * from test -- single node call
select * from test settings object_storage_cluster="acluster" -- clustered call

If that is correct, could you please add a test that cover this use case?

arthurpassos · 2025-02-27T01:03:33Z

src/Storages/ObjectStorage/StorageObjectStorageCluster.cpp

+
+String StorageObjectStorageCluster::getClusterName(ContextPtr context) const
+{
+    auto cluster_name_ = context->getSettingsRef()[Setting::object_storage_cluster].value;


Variable name: rename the variable. One must be named cluster_name_from_settings, while the other remains intact.

Comment/ Docs: This piece of code includes just one if statement checking two sources of cluster name, it is not hard to understand that. The "tricky" part to understand is WHY this is being done. I would like to see something along these lines: "StorageObjectStorageCluster is always being instantiated, even for non clustered table definitions. But the user can specify the cluster in the query settings, and that must be honored. The xyz setting takes precedence over the table definition one because of abc."

ianton-ru · 2025-02-27T07:46:29Z

Correct me if I am wrong, but one thing you are trying to achieve here that wasn't in the original plans is to be able to switch cluster on and off using query settings. Is that correct? For example:
create table test ... engine=IcebergS3 -- no cluster settings here
select * from test -- single node call
select * from test settings object_storage_cluster="acluster" -- clustered call
If that is correct, could you please add a test that cover this use case?

Correct. Test is here.
create_iceberg_table creates table without setting. select_pure_table_engine - select query without any additional settings. select_pure_table_engine_cluster - query witn object_storage_cluster setting. And below in count_secondary_subqueries checks that the first one has no secondary queries, and the second has secondary query on each cluster host.

arthurpassos · 2025-02-27T12:52:04Z

Correct me if I am wrong, but one thing you are trying to achieve here that wasn't in the original plans is to be able to switch cluster on and off using query settings. Is that correct? For example:
create table test ... engine=IcebergS3 -- no cluster settings here
select * from test -- single node call
select * from test settings object_storage_cluster="acluster" -- clustered call
If that is correct, could you please add a test that cover this use case?
Correct. Test is here. create_iceberg_table creates table without setting. select_pure_table_engine - select query without any additional settings. select_pure_table_engine_cluster - query witn object_storage_cluster setting. And below in count_secondary_subqueries checks that the first one has no secondary queries, and the second has secondary query on each cluster host.

Ok. So am I right that the reason you can't pick one cluster name during the instantiation of StorageObjectStorageCluster is because for table engines that make change at runtime?

Initially, I wanted you to do something like the below:

std::string resolved_cluster_name = cluster_name;
if (auto cluster_name_from_settings = context->get...)
{
    resolved_cluster_name = cluster_name_from_settings;
}

return std::make_shared<StorageObjectStorageCluster>(resolved_cluster_name, ...);

This would remove the branching from inside the class, making it simpler and more idiomatic. But, if my above statement is correct, then this is not possible since you must be able to switch between clustered and non clustered implementation after the class has been instantiated.

If all of this is correct, then the only thing left for me to approve the PR is the comment / docs I asked for in: #615 (comment)

arthurpassos · 2025-02-27T12:53:30Z

src/Core/SettingsChangesHistory.cpp

@@ -73,6 +73,7 @@ static std::initializer_list<std::pair<ClickHouseVersion, SettingsChangesHistory
            {"least_greatest_legacy_null_behavior", true, false, "New setting"},
            {"object_storage_cluster", "", "", "New setting"},
            {"object_storage_max_nodes", 0, 0, "New setting"},
+            {"input_format_parquet_use_metadata_cache", false, false, "New setting"},


Oh, one more thing. I believe you added this because of a CI/CD failure. @Enmk has merged a PR that fixes this. Perhaps you want to update your branch?

arthurpassos

LGTM

…-engine

Enmk · 2025-03-01T01:31:52Z

src/Storages/ObjectStorage/StorageObjectStorageCluster.cpp

@@ -75,19 +94,132 @@ StorageObjectStorageCluster::StorageObjectStorageCluster(

    setVirtuals(VirtualColumnUtils::getVirtualsForFileLikeStorage(metadata.columns, context_, sample_path));
    setInMemoryMetadata(metadata);
+
+    pure_storage = std::make_shared<StorageObjectStorage>(


setVirtuals() tries to access pure_storage, which is nullptr...

https://s3.amazonaws.com/altinity-build-artifacts/615/17c53f4b34f3d1de0d46781d014825d061b24897/stateless_tests__release_.html

2025-02-28 20:35:49 [389fdafdb833] 2025.02.28 09:35:33.789714 [ 15427 ] <Fatal> BaseDaemon: ######################################## 2025-02-28 20:35:49 [389fdafdb833] 2025.02.28 09:35:33.789754 [ 15427 ] <Fatal> BaseDaemon: (version 24.12.2.20024.altinitytest (altinity build), build id: 1C0EBCF3C00E43841FC84CD01D5EB270B4C8F268, git hash: 664518871e6afcfdfb63964c7c38aa0b14854d25) (from thread 1444) (query_id: 69ae0cff-14c5-4f44-a31a-0c4d86618c26) (query: SELECT *, column0 FROM s3Cluster(test_cluster_one_shard_three_replicas_localhost, 'http://localhost:11111/test/hive_partitioning/column0=Elizabeth/sample.parquet') LIMIT 10;) Received signal Segmentation fault (11) 2025-02-28 20:35:49 [389fdafdb833] 2025.02.28 09:35:33.789784 [ 15427 ] <Fatal> BaseDaemon: Address: NULL pointer. Access: read. Address not mapped to object. 2025-02-28 20:35:49 [389fdafdb833] 2025.02.28 09:35:33.789801 [ 15427 ] <Fatal> BaseDaemon: Stack trace: 0x000000000dae2ad5 0x00007fd65efc3520 0x0000000010afc85e 0x0000000010af9135 0x000000000f53f1c1 0x000000000f53e3dd 0x0000000010a0afd6 0x000000001199809f 0x00000000117a0abe 0x00000000117cd5bc 0x0000000011790236 0x000000001178e66c 0x000000001178df40 0x0000000011e3d036 0x000000001209961d 0x000000001209757c 0x000000001209b322 0x0000000012035bcc 0x000000001241541f 0x0000000012410a1d 0x00000000136ee19e 0x0000000013709418 0x000000001661cc07 0x000000001661d099 0x00000000165e99bc 0x00000000165e7f5d 0x00007fd65f015ac3 0x00007fd65f0a7850 2025-02-28 20:35:49 [389fdafdb833] 2025.02.28 09:35:33.820039 [ 15427 ] <Fatal> BaseDaemon: 0. ./build_docker/./src/Common/SignalHandlers.cpp:105: signalHandler(int, siginfo_t*, void*) @ 0x000000000dae2ad5 2025-02-28 20:35:49 [389fdafdb833] 2025.02.28 09:35:33.820109 [ 15427 ] <Fatal> BaseDaemon: 1. ? @ 0x00007fd65efc3520 2025-02-28 20:35:49 [389fdafdb833] 2025.02.28 09:35:33.851621 [ 15427 ] <Fatal> BaseDaemon: 2. ./src/Storages/ObjectStorage/StorageObjectStorageCluster.h:49: DB::StorageObjectStorageCluster::setVirtuals(DB::VirtualColumnsDescription) @ 0x0000000010afc85e 2025-02-28 20:35:49 [389fdafdb833] 2025.02.28 09:35:33.882793 [ 15427 ] <Fatal> BaseDaemon: 3. ./build_docker/./src/Storages/ObjectStorage/StorageObjectStorageCluster.cpp:95: DB::StorageObjectStorageCluster::StorageObjectStorageCluster(String const&, std::shared_ptr<DB::StorageObjectStorage::Configuration>, std::shared_ptr<DB::IObjectStorage>, std::shared_ptr<DB::Context const>, DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::optional<DB::FormatSettings>, DB::LoadingStrictnessLevel, std::shared_ptr<DB::IAST>) @ 0x0000000010af9135 2025-02-28 20:35:49 [389fdafdb833] 2025.02.28 09:35:33.907551 [ 15427 ] <Fatal> BaseDaemon: 4.0. inlined from ./contrib/llvm-project/libcxx/include/__memory/construct_at.h:35: DB::StorageObjectStorageCluster* std::construct_at[abi:v15007]<DB::StorageObjectStorageCluster, String const&, std::shared_ptr<DB::StorageObjectStorage::Configuration>&, std::shared_ptr<DB::IObjectStorage>&, std::shared_ptr<DB::Context const>&, DB::StorageID, DB::ColumnsDescription&, DB::ConstraintsDescription, String, std::nullopt_t const&, DB::LoadingStrictnessLevel, std::nullptr_t, DB::StorageObjectStorageCluster*>(DB::StorageObjectStorageCluster*, String const&, std::shared_ptr<DB::StorageObjectStorage::Configuration>&, std::shared_ptr<DB::IObjectStorage>&, std::shared_ptr<DB::Context const>&, DB::StorageID&&, DB::ColumnsDescription&, DB::ConstraintsDescription&&, String&&, std::nullopt_t const&, DB::LoadingStrictnessLevel&&, std::nullptr_t&&) 2025-02-28 20:35:49 [389fdafdb833] 2025.02.28 09:35:33.907617 [ 15427 ] <Fatal> BaseDaemon: 4.1. inlined from ./contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:298: void std::allocator_traits<std::allocator<DB::StorageObjectStorageCluster>>::construct[abi:v15007]<DB::StorageObjectStorageCluster, String const&, std::shared_ptr<DB::StorageObjectStorage::Configuration>&, std::shared_ptr<DB::IObjectStorage>&, std::shared_ptr<DB::Context const>&, DB::StorageID, DB::ColumnsDescription&, DB::ConstraintsDescription, String, std::nullopt_t const&, DB::LoadingStrictnessLevel, std::nullptr_t, void, void>(std::allocator<DB::StorageObjectStorageCluster>&, DB::StorageObjectStorageCluster*, String const&, std::shared_ptr<DB::StorageObjectStorage::Configuration>&, std::shared_ptr<DB::IObjectStorage>&, std::shared_ptr<DB::Context const>&, DB::StorageID&&, DB::ColumnsDescription&, DB::ConstraintsDescription&&, String&&, std::nullopt_t const&, DB::LoadingStrictnessLevel&&, std::nullptr_t&&) 2025-02-28 20:35:49 [389fdafdb833] 2025.02.28 09:35:33.907645 [ 15427 ] <Fatal> BaseDaemon: 4.2. inlined from ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:292: __shared_ptr_emplace<const std::basic_string<char, std::char_traits<char>, std::allocator<char> > &, std::shared_ptr<DB::StorageObjectStorage::Configuration> &, std::shared_ptr<DB::IObjectStorage> &, std::shared_ptr<const DB::Context> &, DB::StorageID, DB::ColumnsDescription &, DB::ConstraintsDescription, std::basic_string<char, std::char_traits<char>, std::allocator<char> >, const std::nullopt_t &, DB::LoadingStrictnessLevel, std::nullptr_t> 2025-02-28 20:35:49 [389fdafdb833] 2025.02.28 09:35:33.907663 [ 15427 ] <Fatal> BaseDaemon: 4. ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:953: std::shared_ptr<DB::StorageObjectStorageCluster> std::allocate_shared[abi:v15007]<DB::StorageObjectStorageCluster, std::allocator<DB::StorageObjectStorageCluster>, String const&, std::shared_ptr<DB::StorageObjectStorage::Configuration>&, std::shared_ptr<DB::IObjectStorage>&, std::shared_ptr<DB::Context const>&, DB::StorageID, DB::ColumnsDescription&, DB::ConstraintsDescription, String, std::nullopt_t const&, DB::LoadingStrictnessLevel, std::nullptr_t, void>(std::allocator<DB::StorageObjectStorageCluster> const&, String const&, std::shared_ptr<DB::StorageObjectStorage::Configuration>&, std::shared_ptr<DB::IObjectStorage>&, std::shared_ptr<DB::Context const>&, DB::StorageID&&, DB::ColumnsDescription&, DB::ConstraintsDescription&&, String&&, std::nullopt_t const&, DB::LoadingStrictnessLevel&&, std::nullptr_t&&) @ 0x000000000f53f1c1 2025-02-28 20:35:49 [389fdafdb833] 2025.02.28 09:35:33.930592 [ 15427 ] <Fatal> BaseDaemon: 5.0. inlined from ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:962: std::shared_ptr<DB::StorageObjectStorageCluster> std::make_shared[abi:v15007]<DB::StorageObjectStorageCluster, String const&, std::shared_ptr<DB::StorageObjectStorage::Configuration>&, std::shared_ptr<DB::IObjectStorage>&, std::shared_ptr<DB::Context const>&, DB::StorageID, DB::ColumnsDescription&, DB::ConstraintsDescription, String, std::nullopt_t const&, DB::LoadingStrictnessLevel, std::nullptr_t, void>(String const&, std::shared_ptr<DB::StorageObjectStorage::Configuration>&, std::shared_ptr<DB::IObjectStorage>&, std::shared_ptr<DB::Context const>&, DB::StorageID&&, DB::ColumnsDescription&, DB::ConstraintsDescription&&, String&&, std::nullopt_t const&, DB::LoadingStrictnessLevel&&, std::nullptr_t&&) 2025-02-28 20:35:49 [389fdafdb833] 2025.02.28 09:35:33.930655 [ 15427 ] <Fatal> BaseDaemon: 5. ./build_docker/./src/TableFunctions/TableFunctionObjectStorageCluster.cpp:53: DB::TableFunctionObjectStorageCluster<DB::S3ClusterDefinition, DB::StorageS3Configuration>::executeImpl(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const>, String const&, DB::ColumnsDescription, bool) const @ 0x000000000f53e3dd

Enmk

stateless test 03203_hive_style_partitioning crashes the server, please take a look and fix it
https://s3.amazonaws.com/altinity-build-artifacts/615/17c53f4b34f3d1de0d46781d014825d061b24897/stateless_tests__release_.html

Distributed request to tables with Object Storage Engines

c22bf24

ianton-ru added 2 commits February 7, 2025 10:21

Fix tests

3a11374

Merge antalya

dfd14d0

arthurpassos reviewed Feb 10, 2025

View reviewed changes

Fixes after review

fb3e1b6

arthurpassos reviewed Feb 13, 2025

View reviewed changes

ianton-ru added 2 commits February 14, 2025 10:47

Merge base Antalya branch

28d6c5c

More fixes after review

78261d3

arthurpassos previously approved these changes Feb 14, 2025

View reviewed changes

ianton-ru added 3 commits February 14, 2025 13:41

Rename getTableFunctionArguments to addPathAndAccessKeysToArgs

ac37da6

More refactoring

db44166

Add ability to choose object storage cluster in select query

3fafe6f

ianton-ru force-pushed the project-antalya-24.12.2-distributed-table-engine branch from c7b1ad3 to 3fafe6f Compare February 18, 2025 15:43

ianton-ru added 3 commits February 18, 2025 20:16

Fix write to pure engine

cfc74ec

Fix virtual columns for pure storage

df462de

More fixes

508e4ba

Enmk changed the base branch from project-antalya-24.12.2 to antalya February 19, 2025 21:28

ianton-ru dismissed arthurpassos’s stale review via 508e4ba February 19, 2025 21:35

ianton-ru added 2 commits February 20, 2025 14:34

Do not update configuration twice

9dbd209

Merge antalya branch

5b923c9

ianton-ru mentioned this pull request Feb 20, 2025

Distributed table engine #643

Closed

29 tasks

Create pure storage on create

293ac83

arthurpassos reviewed Feb 24, 2025

View reviewed changes

More simple

5bc11ee

ianton-ru force-pushed the project-antalya-24.12.2-distributed-table-engine branch from 942c5e1 to 5bc11ee Compare February 26, 2025 11:51

arthurpassos reviewed Feb 27, 2025

View reviewed changes

Rename cluster_name_ variable to cluster_name_from_settings

a4943e2

arthurpassos reviewed Feb 27, 2025

View reviewed changes

ianton-ru added 2 commits February 27, 2025 15:02

Merge antalya branch

d5d3073

Changes after review

09321d3

arthurpassos approved these changes Feb 27, 2025

View reviewed changes

Merge branch 'antalya' into project-antalya-24.12.2-distributed-table…

17c53f4

…-engine

Enmk reviewed Mar 1, 2025

View reviewed changes

Enmk requested changes Mar 1, 2025

View reviewed changes

		@@ -247,6 +247,10 @@ class StorageObjectStorage::Configuration

		virtual void update(ObjectStoragePtr object_storage, ContextPtr local_context);

		virtual void setFunctionArgs(ASTs & /* args */) const

Distributed request to tables with Object Storage Engines #615

Are you sure you want to change the base?

Distributed request to tables with Object Storage Engines #615

Conversation

ianton-ru commented Feb 6, 2025

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Documentation entry for user-facing changes

Modify your CI run:

Include tests (required builds will be added automatically):

Exclude tests:

Extra options:

Only specified batches in multi-batch jobs:

altinity-robot commented Feb 6, 2025 • edited Loading

arthurpassos left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arthurpassos left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ianton-ru Feb 14, 2025 • edited Loading

Choose a reason for hiding this comment

arthurpassos left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arthurpassos left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arthurpassos Feb 24, 2025 • edited Loading

Choose a reason for hiding this comment

arthurpassos Feb 27, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arthurpassos left a comment

Choose a reason for hiding this comment

arthurpassos Feb 27, 2025 • edited Loading

Choose a reason for hiding this comment

ianton-ru commented Feb 27, 2025

arthurpassos commented Feb 27, 2025

Choose a reason for hiding this comment

arthurpassos left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Enmk left a comment

Choose a reason for hiding this comment

altinity-robot commented Feb 6, 2025 •

edited

Loading

ianton-ru Feb 14, 2025 •

edited

Loading

arthurpassos Feb 24, 2025 •

edited

Loading

arthurpassos Feb 27, 2025 •

edited

Loading

arthurpassos Feb 27, 2025 •

edited

Loading