Skip to content

A native Rust library for Delta Lake, with bindings into Python and Ruby.

License

Notifications You must be signed in to change notification settings

jslorrma/delta-rs

 
 

Repository files navigation

delta-rs

About

This library provides low level access to Delta tables in Rust, which can be used with data processing frameworks like datafusion, ballista, polars, vega, etc. It also provides bindings to other higher level languages such as Python or Ruby.

Features

Supported backends:

  • Local file system

  • AWS S3

  • Azure Data Lake Storage Gen 2 (HOW-TO)

  • Google Cloud Storage

Table 1. Support features
Operation/Feature Rust Python Ruby

Read table

✔️

✔️

✔️

Stream table update

✔️

✔️

Filter files with partitions

✔️

✔️

Vacuum (delete stale files)

#97

#97

History

✔️

✔️

Write transactions

✔️

Checkpoint creation

✔️

High-level file writer

Optimize

#98

Get Involved

Development Meeting

We have a standing development sync meeting for those that are interested. The meeting is held every two weeks at 9am PST on Tuesday mornings. The direct meeting URL is shared in the Slack channel above ☝️ before the meeting.

These meetings are also streamed live via YouTube if you just want to listen in.

Development

delta-rs requires the Rust compiler, which can be installed with the rustup command.

Running tests can be done with cargo test in the root directory, or one of the directories below:

Rust

The rust/ directory contains core Rust APIs for accessing Delta Lake from Rust, or for higher-level language bindings.

Python

The python/ directory contains the deltalake Python package built on top of delta-rs

Ruby

The ruby/ directory contains an early prototype of a Ruby library built on top of delta-rs

About

A native Rust library for Delta Lake, with bindings into Python and Ruby.

Resources

License

Code of conduct

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Rust 82.9%
  • Python 9.7%
  • TLA 5.7%
  • Shell 0.8%
  • Makefile 0.5%
  • Ruby 0.3%
  • Batchfile 0.1%