GH-44800: [C#] Implement Flight SQL Client #44783

HackPoint · 2024-11-18T17:41:14Z

GH-44800: [C#] Implement Flight SQL Client

Rationale for this Change

This pull request introduces a new implementation of FlightSqlClient and PreparedStatement in C#. Previously, there was no C# client for Flight SQL, leaving a significant gap for .NET developers who wished to interact with Flight SQL servers.

The implementation aligns with the existing C++ Flight SQL client API, ensuring consistent and familiar behavior across languages and providing a robust client for the Apache Arrow ecosystem in .NET.

What's Included in this PR?

Key Features

FlightSqlClient:
- Provides query execution (ExecuteAsync, ExecuteUpdateAsync) and schema retrieval (GetCatalogsAsync, GetDbSchemasAsync, etc.).
- Implements metadata operations for catalogs, schemas, tables, and more.
- Fully integrated with gRPC and Apache Arrow ecosystems.
- Supports extensibility for advanced features like transactions.
PreparedStatement:
- Implements parameterized query execution (SetParameters, ExecuteAsync, and ExecuteUpdateAsync).
- Supports lifecycle management (CloseAsync) for effective resource handling.
- Aligns with the prepared statement design in the C++ client.

API Consistency

This implementation mirrors the C++ Flight SQL client to ensure API alignment across supported languages:

Consistent naming conventions and parameter semantics.
Ensures .NET developers can work seamlessly with existing Flight SQL servers.

Are These Changes Tested?

Testing Overview

Unit Tests:
- Added tests for query execution and parameter binding in PreparedStatement.
- Verified schema retrieval methods like GetCatalogsAsync and GetDbSchemasAsync.
Integration Tests:
- Tested against a live Flight SQL server to validate query execution, schema retrieval, and metadata operations.
End-to-End Tests:
- Covered real-world scenarios for parameterized updates and queries, ensuring robustness.

Example Test Cases

Verify that parameterized queries return correct results with valid input.
Ensure schema retrieval throws appropriate exceptions for invalid descriptors.
Validate row counts after ExecuteUpdateAsync.

Are There Any Breaking Changes?

This PR introduces new functionality and does not affect any existing features. There are no breaking changes.

Are There Any User-Facing Changes?

New Capabilities

FlightSqlClient:
- Query execution and schema retrieval for SQL queries on Flight SQL servers.
- Metadata retrieval for catalogs, schemas, tables, and more.
PreparedStatement:
- Supports parameterized queries with proper parameter binding.
- Provides robust lifecycle management and execution.

API Consistency

Aligns with the C++ Flight SQL client for interoperability and familiar API design.

Additional Notes

This PR does not include transaction support at this stage, as it requires additional server-side capabilities.
All methods follow idiomatic C# practices, including async/await for non-blocking operations.
Extensible for future enhancements, including advanced features like savepoints.

Resources

C++ Flight SQL Client Reference: Apache Arrow Flight SQL Documentation
Apache Arrow Contribution Guide: Contributing to Apache Arrow

Feedback and Suggestions

Thank you for reviewing this contribution! Suggestions and feedback are welcome to ensure the implementation meets the project's standards and requirements.

GitHub Issue: [C#] Implement Flight SQL Client #44800

**GetExportedKeys** - Retrieves a description about the foreign key columns that reference the primary key columns of the given table. - **GetExportedKeysSchema** - Get the exported keys schema from the server. - **GetImportedKeys** - Retrieves the foreign key columns for the given table. - **GetImportedKeysSchema** - Get the imported keys schema from the server. - **GetCrossReference** - Retrieves a description of the foreign key columns in the given foreign key table that reference the primary key or the columns representing a unique constraint of the parent table. - **GetCrossReferenceSchema** - Get the cross reference schema from the server. - **GetTableTypes** - Request a list of table types. - **GetTableTypesSchema** - Get the table types schema from the server. - **GetXdbcTypeInfo** - Request the information about all the data types supported. - **GetXdbcTypeInfo (with data_type parameter)** - Request the information about a specific data type supported. - **GetXdbcTypeInfoSchema** - Get the type info schema from the server. - **GetSqlInfo** - Request a list of SQL information. - **GetSqlInfoSchema**

…t/flight-sql-client

# Conflicts: # csharp/src/Apache.Arrow.Flight/Client/FlightClient.cs # csharp/test/Apache.Arrow.Flight.Tests/FlightTests.cs

…htSqlServer.cs csharp/Apache.Arrow.Flight.Sql.TestWeb/TestFlightSqlServer.cs var batches = flightHolder.GetRecordBatches(); foreach (var batch in batches) { await responseStream.WriteAsync(batch.RecordBatch, batch.Metadata); Add ConfigureAwait(false)

- PreparedStatement implementation - SqlActions adding command requests -- Commit -- Rollback -- BeginTransaction -- CancelFlightInfo

…t was forked

github-actions · 2024-11-20T17:35:02Z

⚠️ GitHub issue #44800 has been automatically assigned in GitHub to PR creator.

assignUser · 2024-11-20T17:51:57Z

I approved the workflows.

I don't work on C# sharp but wanted to thank you @HackPoint for such a big first-time contribution!

HackPoint · 2024-11-25T15:22:52Z

I approved the workflows.

I don't work on C# sharp but wanted to thank you @HackPoint for such a big first-time contribution!

Can you please review the code and allow me to close the PR please?

CurtHagenlocher · 2024-11-25T16:11:52Z

Can you please review the code and allow me to close the PR please?

I will review this tonight.

kou · 2024-11-26T05:27:48Z

Could you revert all cpp/ changes?

HackPoint · 2024-11-26T05:30:32Z

I’ve haven’t made any changes in c++, but I’ll re-fork and cherry-pick my code into a new fork and push all the changes.

…

On Tue, 26 Nov 2024 at 7:28 Sutou Kouhei ***@***.***> wrote: Could you revert all cpp/ changes? — Reply to this email directly, view it on GitHub <#44783 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABTSBM7DBABI7X26GBOCS7D2CQBGVAVCNFSM6AAAAABSAGSJF6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOJZGY4TEMRQGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

HackPoint · 2024-11-28T10:43:02Z

Could you revert all cpp/ changes?

seems that it's done

# Conflicts: # csharp/src/Apache.Arrow.Flight.Sql/Apache.Arrow.Flight.Sql.csproj

jeremyosterhoudt · 2025-01-06T17:54:07Z

Any updates on this? We are also looking to use .NET with a Flight SQL client and are eager to get this functionality.

CurtHagenlocher · 2025-01-06T18:39:31Z

Any updates on this? We are also looking to use .NET with a Flight SQL client and are eager to get this functionality.

I'm sorry about the delay; I was very much wanting to get the review done before today's fork but first work, then vacation, then a lengthy internet outage at home got in the way. I will definitely make time for a review later this week.

HackPoint · 2025-01-27T16:17:20Z

Any updates on this? We are also looking to use .NET with a Flight SQL client and are eager to get this functionality.

I'm sorry about the delay; I was very much wanting to get the review done before today's fork but first work, then vacation, then a lengthy internet outage at home got in the way. I will definitely make time for a review later this week.

Hi Curt,
Is it possible to review and if approved merge my PR, it's already waiting for more than 2 months.

CurtHagenlocher

Thanks again for such a comprehensive change! I'm very sorry it took me this long to get to it. I've left a bunch of feedback of various degrees of seriousness. You'll also likely need to rebase given the amount of time that has passed (and the presence of a merge conflict).

CurtHagenlocher · 2025-02-08T22:56:48Z

csharp/examples/Examples.sln

@@ -7,6 +7,8 @@ Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "FluentBuilderExample", "Flu
 EndProject
 Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Apache.Arrow", "..\src\Apache.Arrow\Apache.Arrow.csproj", "{1FE1DE95-FF6E-4895-82E7-909713C53524}"
 EndProject
+Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "FlightClientExample", "FlightClientExample\FlightClientExample.csproj", "{CBB46C39-530D-465A-9367-4E771595209A}"
+EndProject


FlightClientExample was added to Examples.sln but its files are not part of this pull request.

CurtHagenlocher · 2025-02-08T22:58:44Z

csharp/src/Apache.Arrow.Flight/FlightInfoCancelRequest.cs

@@ -0,0 +1,38 @@
+using System;


Added files must have the Apache header. It's missing from this file as well as FlightInfoCancelResult.cs.

CurtHagenlocher · 2025-02-09T02:58:20Z

csharp/src/Apache.Arrow.Flight.Sql/TableRef.cs

+public class TableRef
+{
+    public string? Catalog { get; set; }
+    public string DbSchema { get; set; } = null!;


The null! seems dodgy. As these are required to be set, consider using a pair of constructors TableRef(string, string) and TableRef(string, string, string) instead.

CurtHagenlocher · 2025-02-09T02:58:57Z

csharp/src/Apache.Arrow.Flight.Sql/Transaction.cs

+        _transactionId = ByteString.CopyFromUtf8(transactionId);
+    }
+
+    public bool IsValid() => TransactionId.Length > 0;


Should this be a property?

CurtHagenlocher · 2025-02-09T03:00:35Z

csharp/src/Apache.Arrow.Flight.Sql/Transaction.cs

+    }
+
+    public bool IsValid() => TransactionId.Length > 0;
+    public void ResetTransaction()


ResetTransaction doesnt appear to be used.

CurtHagenlocher · 2025-02-09T03:44:59Z

csharp/test/Apache.Arrow.Flight.Sql.Tests/FlightSqlClientTests.cs

+            new StringArray.Builder().Append("John Doe").Build()
+        }, 1);
+
+


nit: extra blank line

CurtHagenlocher · 2025-02-09T03:45:19Z

csharp/test/Apache.Arrow.Flight.Sql.Tests/FlightSqlClientTests.cs

+        Assert.IsType<FlightInfo>(flightInfo);
+    }
+
+


nit: extra blank line

CurtHagenlocher · 2025-02-09T03:45:38Z

csharp/test/Apache.Arrow.Flight.Sql.Tests/FlightSqlClientTests.cs

+        Assert.Equal(expectedFlightInfo.Endpoints.Count, flightInfo.Endpoints.Count);
+    }
+
+


nit: extra blank line

CurtHagenlocher · 2025-02-09T03:46:16Z

csharp/test/Apache.Arrow.Flight.Sql.Tests/FlightSqlPreparedStatementTests.cs

+                }, 3);
+        }
+
+


nit: extra blank line

CurtHagenlocher · 2025-02-09T03:47:48Z

csharp/test/Apache.Arrow.Flight.Sql.Tests/FlightSqlTestUtils.cs

+        return batchBuilder.Build();
+    }
+
+


nit: extra blank line

Genady Shmunik and others added 30 commits August 5, 2024 19:14

initial commit

96e686a

Merge branch 'apache:main' into feat/flight-sql-client

61afbc4

Merge branch 'apache:main' into feat/flight-sql-client

b838643

Merge branch 'refs/heads/main' into feat/flight-sql-client

b191116

feat: cancel functionality

d7a535c

feat: cancel flight info

b6c3e9e

feat: cancel query

d43aa76

Merge branch 'apache:main' into feat/flight-sql-client

f3d713d

test(FlightSqlClient): infra and initial testing

728957e

Merge branch 'apache:main' into feat/flight-sql-client

9bee3e4

Merge remote-tracking branch 'origin/feat/flight-sql-client' into fea…

f469bac

…t/flight-sql-client

refactor: FlightSqlClient

a3c0c09

Merge branch 'main' into feat/flight-sql-client

0aa9353

feat(test): GetFlightInfo

4d48ab1

refactor: Flight sql testing

4bfc2a9

testing: flightsql

291670d

rollback: FlightTests

2df0baa

Merge branch 'main' into feat/flight-sql-client

6e822ce

# Conflicts: # csharp/src/Apache.Arrow.Flight/Client/FlightClient.cs # csharp/test/Apache.Arrow.Flight.Tests/FlightTests.cs

test: flight client

6c6cbf7

testing: FlightSqlClient

89b6661

testing: CancelQuery

ac23dba

test: DoPut

0263c44

Merge branch 'apache:main' into feat/flight-sql-client

dea81b0

Merge branch 'main' into feat/flight-sql-client

a326326

chore: protobuf version fix

c5eb5d6

chore: remove commented out commands in CMakeLists.txt

c4abd00

chore: attending issues in github

8907913

chore: adding features

861df24

- PreparedStatement implementation - SqlActions adding command requests -- Commit -- Rollback -- BeginTransaction -- CancelFlightInfo

HackPoint added 4 commits November 20, 2024 08:52

chore: appended not existing project which was added after the projec…

842e1ab

…t was forked

chore: reverted version to existing one 2.66.0 -> 2.67.0

9f166c2

chore: removed empty line

be65271

Merge branch 'main' into feat/flight-sql-client

59cc8e7

HackPoint requested review from CurtHagenlocher and lidavidm November 20, 2024 07:44

HackPoint changed the title ~~Feat/flight sql client~~ GH-44800: [C#] Implement Flight SQL Client Nov 20, 2024

github-actions bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Nov 20, 2024

HackPoint added 3 commits November 24, 2024 12:20

fix: conversion from long to int and default value

2b615d6

chore: clear out the empty line

96768cd

chore: remove redundant change

02a9cb1

HackPoint added 2 commits November 28, 2024 11:55

Merge branch 'main' into feat/flight-sql-client

dc9cf53

chore: revert changes

d4afd4e

github-actions bot removed Component: C++ Component: FlightRPC labels Nov 28, 2024

HackPoint added 2 commits December 23, 2024 13:23

Merge branch 'main' into feat/flight-sql-client

589fb12

# Conflicts: # csharp/src/Apache.Arrow.Flight.Sql/Apache.Arrow.Flight.Sql.csproj

chore: clean up redundant namespaces

e124151

CurtHagenlocher self-assigned this Feb 8, 2025

CurtHagenlocher requested changes Feb 9, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-44800: [C#] Implement Flight SQL Client #44783

GH-44800: [C#] Implement Flight SQL Client #44783

HackPoint commented Nov 18, 2024 •

edited by github-actions bot

Loading

github-actions bot commented Nov 20, 2024

assignUser commented Nov 20, 2024

HackPoint commented Nov 25, 2024

CurtHagenlocher commented Nov 25, 2024

kou commented Nov 26, 2024

HackPoint commented Nov 26, 2024 via email

HackPoint commented Nov 28, 2024

jeremyosterhoudt commented Jan 6, 2025

CurtHagenlocher commented Jan 6, 2025

HackPoint commented Jan 27, 2025

CurtHagenlocher left a comment •

edited

Loading

CurtHagenlocher Feb 8, 2025

CurtHagenlocher Feb 8, 2025

CurtHagenlocher Feb 9, 2025

CurtHagenlocher Feb 9, 2025

CurtHagenlocher Feb 9, 2025

CurtHagenlocher Feb 9, 2025

CurtHagenlocher Feb 9, 2025

CurtHagenlocher Feb 9, 2025

CurtHagenlocher Feb 9, 2025

CurtHagenlocher Feb 9, 2025

		Assert.Equal(expectedFlightInfo.Endpoints.Count, flightInfo.Endpoints.Count);
		}

GH-44800: [C#] Implement Flight SQL Client #44783

Are you sure you want to change the base?

GH-44800: [C#] Implement Flight SQL Client #44783

Conversation

HackPoint commented Nov 18, 2024 • edited by github-actions bot Loading

Rationale for this Change

What's Included in this PR?

Key Features

API Consistency

Are These Changes Tested?

Testing Overview

Example Test Cases

Are There Any Breaking Changes?

Are There Any User-Facing Changes?

New Capabilities

API Consistency

Additional Notes

Resources

Feedback and Suggestions

github-actions bot commented Nov 20, 2024

assignUser commented Nov 20, 2024

HackPoint commented Nov 25, 2024

CurtHagenlocher commented Nov 25, 2024

kou commented Nov 26, 2024

HackPoint commented Nov 26, 2024 via email

HackPoint commented Nov 28, 2024

jeremyosterhoudt commented Jan 6, 2025

CurtHagenlocher commented Jan 6, 2025

HackPoint commented Jan 27, 2025

CurtHagenlocher left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HackPoint commented Nov 18, 2024 •

edited by github-actions bot

Loading

CurtHagenlocher left a comment •

edited

Loading