Donate Ray SQL to the DataFusion Project (apache#1)

* Initial commit

* Basic project structure

* gitignore

* Add protobuf plumbing (apache#2)

* Implement protobuf codec

* Wire up query execution (apache#4)

* query runs end to end (apache#5)

* re-organize python code (apache#6)

* Implement shuffle more fully (apache#7)

* update README (apache#8)

* Bug fix (apache#9)

* Support multiple shuffle partitions (apache#10)

* More shuffle fixes (apache#11)

* fix readme (apache#12)

* add perf chart (apache#13)

* Remove hard-coded temp dir (apache#14)

* bug fix (apache#15)

* New results (apache#16)

* Upgrade to DataFusion 17, fix a couple of bugs, add some tests (apache#18)

* Remove debug logging (apache#19)

* update README (apache#21)

* Make better use of futures (apache#23)

* Documentation & bug fixes (apache#24)

* Update README.md

* [WIP] Use Ray object store for shuffle exchange (apache#28)

* Fixes for Ray-based shuffle (apache#29)

* Small fixes for Context (apache#30)

* Make distributed execution work (apache#33)

* Make distributed execution work

* fix tips.py

* fixes; incorporate changes from apache#32

* Upgrade to DataFusion 20 (apache#31)

* Add support for DDL statements, such as `CREATE VIEW` (apache#35)

* Experimenting with supporting DDL

* update docs

* Use PyArrow for zero-copy interaction with the Ray Object Store (apache#36)

* Optimize Ray shuffle with zero-copy object store

* remove more clones

* change bytes to pyarrow.array

* revert /tmp

* remove empty_result_set

* remove empty_result_set

* Fix input partition count bug

* Add Frank as author (apache#37)

* fix hyperlink of issue 22 in docs/README.txt (apache#40)

Co-authored-by: ivanfan <[email protected]>

* delta lake and iceberg table support (apache#43)

* delta support

* imports

* Update DataFusion version to 28.0.0 (apache#41)

* Update DataFusion version

* update example

* Upgrade to DataFusion 33 (apache#45)

* Upgrade to DataFusion 33

* undo release profile change

* Add basic GitHub workflow to compile code (apache#47)

* Create rust.yml

* install protobuf

* fix

* fix

* fix

* fix

* fix

* fix

* Add ASF license header

Signed-off-by: Austin Liu <[email protected]>

* Remove ASF header for generated code

Signed-off-by: Austin Liu <[email protected]>

---------

Signed-off-by: Austin Liu <[email protected]>
Co-authored-by: Andy Grove <[email protected]>
Co-authored-by: Frank Luan <[email protected]>
Co-authored-by: Frank Luan <[email protected]>
Co-authored-by: Ivankings <[email protected]>
Co-authored-by: ivanfan <[email protected]>
Co-authored-by: raviranak <[email protected]>

.github/workflows/rust.yml

@@ -0,0 +1,25 @@
+name: Rust
+
+on:
+  push:
+    branches: [ "main" ]
+  pull_request:
+    branches: [ "main" ]
+
+env:
+  CARGO_TERM_COLOR: always
+
+jobs:
+  build:
+
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v3
+    - name: Install protobuf compiler
+      shell: bash
+      run: sudo apt-get install protobuf-compiler
+    - name: Build Rust code
+      run: cargo build --verbose
+    - name: Run tests
+      run: cargo test --verbose

.gitignore

@@ -0,0 +1,7 @@
+.idea
+target
+__pycache__
+venv
+*.so
+*.log
+results-sf*