Skip to content

selvavm/clickhouse.rs

 
 

Repository files navigation

clickhouse.rs

A typed client for ClickHouse.

Crates.io Documentation MIT licensed Build Status

Features

  • Uses serde for encoding/decoding rows.
  • Uses RowBinary encoding.
  • Provides API for selecting.
  • Provides API for infinite transactional (see below) inserting.
  • Provides API for watching live views.
  • Decompression (lz4, gzip, zlib and brotli). lz4 by default.

Plans

  • Compression for INSERTs.
  • One-line configuration (schema://user:password@host[:port]/database?p=v).
  • The builder pattern.
  • Move to RowBinaryWithNamesAndTypes to support type conversions and better errors.
  • Alternatives for hyper.

Usage

To use the crate, add this to your Cargo.toml:

[dependencies]
clickhouse = "0.9"

[dev-dependencies]
clickhouse = { version = "0.9", features = ["test-util"] }

Examples

See more examples.

Create Client

use clickhouse::Client;

let client = Client::default()
    .with_url("http://localhost:8123")
    .with_user("name")
    .with_password("123")
    .with_database("test");
  • Reuse created clients or clone them in order to reuse a connection pool.

Select rows

use serde::Deserialize;
use clickhouse::Row;

#[derive(Row, Deserialize)]
struct MyRow<'a> {
    no: u32,
    name: &'a str,
}

let mut cursor = client
    .query("SELECT ?fields FROM some WHERE no BETWEEN ? AND ?")
    .bind(500)
    .bind(504)
    .fetch::<MyRow<'_>>()?;

while let Some(row) = cursor.next().await? { .. }
  • Placeholder ?fields is replaced with no, name (fields of Row).
  • Placeholder ? is replaced with values in following bind() calls.
  • Convenient fetch_one::<Row>() and fetch_all::<Row>() can be used to get a first row or all rows correspondingly.
  • sql::Identifier can be used to bind table names.
  • soa-derive is useful for Nested types.

Insert a batch

let mut insert = client.insert("some")?;
insert.write(&Row { no: 0, name: "foo" }).await?;
insert.write(&Row { no: 1, name: "bar" }).await?;
insert.end().await?;
  • If end() isn't called the insertion will be aborted.
  • Rows are being sent progressively to spread network load.
  • ClickHouse inserts batches atomically only if all rows fit in the same partition and their number is less max_insert_block_size.
  • ch2rs is useful to generate a row type from ClickHouse.

Infinite inserting

let mut inserter = client.inserter("some")?
    .with_max_entries(150_000) // `250_000` by default
    .with_max_duration(Duration::from_secs(15)); // `10s` by default

inserter.write(&Row { no: 0, name: "foo" }).await?;
inserter.write(&Row { no: 1, name: "bar" }).await?;
let stats = inserter.commit().await?;
if stats.entries > 0 {
    println!("{} entries ({} transactions) have been inserted",
        stats.entries, stats.transactions);
}
  • Inserter ends an active insert in commit() if thresholds (max_entries, max_duration) are reached.
  • The interval between ending active inserts is biased (±10% of max_duration) to avoid load spikes by parallel inserters.
  • All rows between commit() calls are inserted in the same INSERT statement.
  • Do not forget to flush if you want to terminate inserting:
inserter.end().await?;

Perform DDL

client.query("DROP TABLE IF EXISTS some").execute().await?;

Live views

Requires the watch feature.

let mut cursor = client
    .watch("SELECT max(no), argMax(name, no) FROM some")
    .fetch::<Row<'_>>()?;

let (version, row) = cursor.next().await?.unwrap();
println!("live view updated: version={}, row={:?}", version, row);

// Use `only_events()` to iterate over versions only.
let mut cursor = client.watch("some_live_view").limit(20).only_events().fetch()?;
println!("live view updated: version={:?}", cursor.next().await?);
  • Use carefully.
  • This code uses or creates if not exists a temporary live view named lv_{sha1(query)} to reuse the same live view by parallel watchers.
  • You can specify a name instead of a query.
  • This API uses JSONEachRowWithProgress under the hood because of the issue.
  • Only struct rows can be used. Avoid fetch::<u64>() and other without specified names.

Mocking

The crate provides utils for mocking CH server and testing DDL, SELECT, INSERT and WATCH queries.

The functionality can be enabled with the test-util feature. Use it only in dev-dependencies.

See the example.

About

A typed client for ClickHouse

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Rust 100.0%