Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Postgres ClickPipe: Add docs for pausing, table modification #3226

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
---
title: "Adding specific tables to a ClickPipe"
slug: /en/integrations/clickpipes/postgres/add_table
---

There are scenarios where it would be useful to add specific tables to a pipe. This becomes a common necessity as your transactional or analytical workload scales.

## Steps to add specific tables
This can be done by the following steps:
1. [Pause](./pause_and_resume#steps-to-pause-a-postgres-clickpipe) the pipe.
2. Click on Edit Table settings.
3. Locate your table - this can be done by searching it in the search bar.
4. Select the table by clicking on the checkbox.
<br/>
![Add table](./images/add_table.png)

5. Click update.
6. Upon successful update, the pipe will have statuses Setup, Snapshot and Running in that order. The table's initial load can be tracked in the **Tables** tab.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
---
title: "Pausing and Resuming a Postgres ClickPipe"
slug: /en/integrations/clickpipes/postgres/pause_and_resume
---

There are scenarios where it would be useful to pause a Postgres ClickPipe. For example, you may want to run some analytics on existing data in a static state. Or, you might be performing upgrades on Postgres. Here is how you can pause and resume a Postgres ClickPipe.

## Steps to pause a Postgres ClickPipe
1. In the Data Sources tab, click on the Postgres ClickPipe you wish to pause.
2. Head over to the **Settings** tab.
3. Click on the **Pause** button.
<br/>
![Pause button](./images/pause_button.png)

4. A dialog box should appear for confirmation. Click on Pause again.
<br/>
![Pause dialog](./images/pause_dialog.png)

4. Head over to the **Metrics** tab.
5. In around 5 seconds (and also on page refresh), the status of the pipe should be **Paused**.
<br/>
![Pause status](./images/pause_status.png)


## Steps to resume a Postgres ClickPipe
1. In the Data Sources tab, click on the Postgres ClickPipe you wish to resume. The status of the mirror should be **Paused** initially.
2. Head over to the **Settings** tab.
3. Click on the **Resume** button.
<br/>
![Resume button](./images/resume_button.png)

4. A dialog box should appear for confirmation. Click on Resume again.
<br/>
![Resume dialog](./images/resume_dialog.png)

5. Head over to the **Metrics** tab.
6. In around 5 seconds (and also on page refresh), the status of the pipe should be **Running**.
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
---
title: "Removing specific tables from a ClickPipe"
slug: /en/integrations/clickpipes/postgres/removing_tables
---

There are scenarios where it would be useful to remove specific tables of a pipe. For example, your analytics workload need to require this table on ClickHouse, and to save costs it makes sense to not have it be replicated.

## Steps to remove specific tables
The first step is to remove the table from the pipe. This can be done by the following steps:
1. [Pause](./pause_and_resume#steps-to-pause-a-postgres-clickpipe) the pipe.
2. Click on Edit Table Settings.
3. Locate your table - this can be done by searching it in the search bar.
4. Deselect the table by clicking on the selected checkbox.
<br/>
![Remove table](./images/remove_table.png)

5. Click update.
6. Upon successful update, in the **Metrics** tab the status will be **Running**. This table will no longer be replicated by this ClickPipe.
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
---
title: Resyncing Specific Tables
description: Resyncing specific tables in a Postgres ClickPipe
slug: /en/integrations/clickpipes/postgres/table_resync
---

# Resyncing specific tables

There are scenarios where it would be useful to have specific tables of a pipe be re-synced. Some sample use-cases could be major schema changes on Postgres, or maybe some data re-modelling on the ClickHouse.

While resyncing individual tables with a button click is a work-in-progress, this guide will share steps on how you can achieve this today in the Postgres ClickPipe.

### 1. Remove the table from the pipe

This can be followed by following the [table removal guide](./removing_tables).

### 2. Truncate or drop the table on ClickHouse

This step is to avoid data duplication when we add this table again in the next step. You can do this by heading over to the **SQL Console** tab in ClickHouse Cloud and running a query.
Note that since PeerDB creates ReplacingMergeTree tables by default, if your table is small enough where temporary duplicates is harmless, this step can be skipped.

### 3. Add the table to the ClickPipe again
This can be followed by following the [table addition guide](./add_table).
Loading