Skip to content

Commit

Permalink
add(clickhouse): credentials integration (#4)
Browse files Browse the repository at this point in the history
  • Loading branch information
wojcik-dorota authored Aug 21, 2024
1 parent 1ce0286 commit cd9d5aa
Show file tree
Hide file tree
Showing 5 changed files with 342 additions and 109 deletions.
114 changes: 114 additions & 0 deletions docs/products/clickhouse/concepts/data-integration-overview.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
---
title: Aiven for ClickHouse® data service integration
sidebar_label: Data integration
keywords: [data service integration, data source integration, managed credentials integration, managed databases integration, named collections]
---

import {ConsoleIcon} from "@site/src/components/ConsoleIcons";

Aiven for ClickHouse® supports different types of integration allowing you to efficiently connect with other services or data sources and access the data to be processed.

There are a few ways of classifying integration types supported in Aiven for ClickHouse:

- [By purpose](/docs/products/clickhouse/concepts/data-integration-overview#observability-integrations-vs-data-source-integrations):
observability integration vs data source integration
- [By location](/docs/products/clickhouse/concepts/data-integration-overview#integrations-between-aiven-managed-services-vs-external-integrations):
integration between Aiven-managed services vs external integration (between an
Aiven-managed service and an external service)
- By scope: [managed databases integration](/docs/products/clickhouse/concepts/data-integration-overview#managed-databases-integration) and
[managed credentials integration](/docs/products/clickhouse/concepts/data-integration-overview#managed-credentials-integration)

## Observability integrations vs data source integrations

Aiven for ClickHouse supports observability integrations and data source integrations,
which have different purposes:

- [Observability integration](/docs/products/clickhouse/howto/list-integrations) is
connecting to other services (either Aiven-managed or external ones) to expose and
process logs and metrics.
- Data service integration is connecting to other services (either Aiven-managed or external)
to use them as data sources. In Aiven for ClickHouse, data service integration is
possible with the following data source types:

- Apache Kafka®
- PostgreSQL®
- MySQL®
- ClickHouse®
- Amazon S3®

## Integrations between Aiven-managed services vs external integrations

By enabling data service integrations, you create streaming data pipelines across
services. Depending on where the services are located, you can have either
integrations between Aiven-managed services or external integrations (between an Aiven service
and an external data source or application).

For integrating with external data sources, Aiven for ClickHouse provides two types of
data service integrations:

- [Managed databases](/docs/products/clickhouse/concepts/data-integration-overview#managed-databases-integration)
- [Managed credentials](/docs/products/clickhouse/concepts/data-integration-overview#managed-credentials-integration)

## Managed credentials integration

The managed credentials integration uses the
[ClickHouse named collections](https://clickhouse.com/docs/en/operations/named-collections)
logic. It allows storing key-value pairs that are to be used as credentials when
connecting to external data sources. To integrate data, you create tables using table
engines. With the managed credentials integration enabled,
querying the data is easier and quicker since you no longer need connections parameters in
each query. They are stored and available from credential storage in Aiven.

Managed credentials integration in Aiven for ClickHouse is supported with the following
data source types:

- PostgreSQL®
- MySQL®
- ClickHouse®
- Amazon S3®

:::important
The managed credentials integration works one-way: It allows to integrate with data source
and access your data from Aiven for ClickHouse, but it doesn’t allow to access Aiven for
ClickHouse data from the integrated data source.
:::

For information on how table engines work in Aiven for ClickHouse services, preview
[Engines: database and table](/docs/products/clickhouse/concepts/service-architecture#engines-database-and-table).
For the list of table engines available in Aiven for ClickHouse, check
[Supported table engines](/docs/products/clickhouse/reference/supported-table-engines).

## Managed databases integration

The managed databases integration allows using a database engine for handling your
external data. When enabled, this type of integration provides you with an automatically
created database, where the remote data is exposed.

Managed databases integration in Aiven for ClickHouse is supported with the following
data source types:

- PostgreSQL®
- Apache Kafka®

For information on how database engines work in Aiven for ClickHouse services, preview
[Engines: database and table](/docs/products/clickhouse/concepts/service-architecture#engines-database-and-table).
For more information on ClickHouse database engines, see
[Database engines](https://clickhouse.com/docs/en/engines/database-engines).

## Supported data source types

Depending on a data source type, Aiven for ClickHouse supports different integration modes.

| Data source type | Data source integration<br/>(with Aiven service <br/>or external source)| Managed databases integration| Managed credentials integration |
|------------------|----------------------------|------------------------------|---------------------------------|
| PostgreSQL | <ConsoleIcon name="tick"/> | <ConsoleIcon name="tick"/> | <ConsoleIcon name="tick"/> |
| MySQL | <ConsoleIcon name="tick"/> | <ConsoleIcon name="cross"/> | <ConsoleIcon name="tick"/> |
| Apache Kafka | <ConsoleIcon name="tick"/> | <ConsoleIcon name="tick"/> | <ConsoleIcon name="cross"/> |
| ClickHouse | <ConsoleIcon name="tick"/> | <ConsoleIcon name="cross"/> | <ConsoleIcon name="tick"/> |
| Amazon S3 | <ConsoleIcon name="tick"/> | <ConsoleIcon name="cross"/> | <ConsoleIcon name="tick"/> |

## Related pages

- [Set up Aiven for ClickHouse® data service integration](/docs/products/clickhouse/howto/data-service-integration)
- [Manage Aiven for ClickHouse® integration databases](/docs/products/clickhouse/howto/integration-databases)
- [Integrate your Aiven for ClickHouse® service](/docs/products/clickhouse/howto/list-integrations)
18 changes: 15 additions & 3 deletions docs/products/clickhouse/howto/copy-data-across-instances.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,16 +15,28 @@ Details of the source (remote) server

## Copy data

1. From your target servers, use the `remoteSecure()`
1. From your target server, use the `remoteSecure()`
function to select data from the source server.

```shell
```sql
SELECT * FROM remoteSecure('HOSTNAME:PORT', db.remote_engine_table, 'USERNAME', 'PASSWORD') LIMIT 3;
```

:::tip
If you have the
[managed credentials integration](/docs/products/clickhouse/concepts/data-integration-overview#managed-credentials-integration)
enabled, you can use instead:

```sql
SELECT * FROM remoteSecure('service_YOUR_REMOTE_CLUSTER', db.remote_engine_table) LIMIT 3;
```

See how to [enable the managed credentials integration](/docs/products/clickhouse/howto/data-service-integration#integrate-with-external-data-sources).
:::

1. Insert the selected data into the target server.

```shell
```sql
INSERT INTO [db.]table [(c1, c2, c3)] SELECT ...
```

Expand Down
Loading

0 comments on commit cd9d5aa

Please sign in to comment.