Tracking issues of 0.1.0 version for Apache Paimon Rust

[Xuanwo]

Hello everyone, this is the tracking update for the paimon rust 0.1.0 release.

Goal

Before we outline the tasks for the 0.1.0 release, let me clarify the project's goal:

Developing a complete implementation of Paimon in pure Rust.

• Users can read/write paimon table like they do in java API.
• Users can read/write paimon table in arrow format.
• Native support for DataFusion
  • But also enable users to implement their own query engines based on this project.
• Native WASM support (a.k.a paimon-wasm)
• Native Python binding based on rust core (a.k.a paimon-py)
• Hive catalog support

Tasks

This will be our initial release, and I aim to include basic read support in it.

• Spec: Implement types that needed by paimon.
  • feat(spec): Add schema types #5
  • Datatypes
  • spec: Implement Snapshot #6
  • spec: Implement Manifest List #14
  • spec: Implement Manifest #7
  • spec: Implement Datafile #10
  • Global Index
  • Data File Index
• Catalog
  • feat(catalog): add catalog API #62
  • catalog: Implement the filesystem catalog #70 (leave hive catalog in next release)
• Arrow Integration:
  • reader: Allow reading paimon data into arrow format. #71
• Basic Read Process: No schema evolution, no merging, no deletion.
  • scan: Implement scan to generate data split from the Snapshot #72 (filter & project push down are not required in v0.1)
• IO: Integrate with Apache OpenDAL for IO.
• Release Utils: Impelment scripts to help generate and verify ASF releases.

After all those tasks, I expect users can read an existing paimon table from storage services.

[QuakeWang]

@Xuanwo I will do the Snapshot task.

[Xuanwo]

@Xuanwo I will do the Snapshot task.

Welcome, have fun!

[dharanad]

hello @Xuanwo , I would like to work on implementing Manifest.

[Xuanwo]

hello @Xuanwo , I would like to work on implementing Manifest.

Great, have fun!

[crrow]

Hey, I'm new around here and this project sounds interesting. Can I jump in and help out?

[Xuanwo]

Manifest

Thanks a lot, welcome to pick up a part. I'm willing to update tracking issues for you.

[crrow]

Manifest

Thanks a lot, welcome to pick up a part. I'm willing to update tracking issues for you.

okay, let me try data file

[Asura7969]

I want to try the manifest list

[thexiay]

Cool, it's good project to be familiar with rust and big data infrastructure😆

[QuakeWang]

I will do the Changelog task, and the corresponding issue have been created #16.

[Xuanwo]

Cool, it's good project to be familiar with rust and big data infrastructure😆

Welcome!

I will do the Changelog task, and the corresponding issue have been created #16.

Updated.

[suxiaogang223]

Hi, I am a big fan of rust, and I usually do data lake-related development. Be happy to join the development of paimon-rust in the future.

[Aitozi]

Does the FileSystemCatalog rely on the Integrate with Apache OpenDAL ? There are some IO operation in the FileSystemCatalog. @Xuanwo

[Xuanwo]

Does the FileSystemCatalog rely on the Integrate with Apache OpenDAL ?

Yes, my plan is using opendal for those IO operations.

[devillove084]

I will try to implement the File index part and try to follow the implementation logic of the Java version and use the Rust style as much as possible. @Xuanwo

[Xuanwo]

I will try to implement the File index part and try to follow the implementation logic of the Java version and use the Rust style as much as possible. @Xuanwo

Thanks!

[ForeverAngry]

Hi all, I'd like to help out with the project. I'm a bit new to rust. Let me know where I can lend a hand at!