Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature](partial update) Support flexible partial update in stream load with json files #39756

Open
wants to merge 22 commits into
base: master
Choose a base branch
from

Conversation

bobhan1
Copy link
Contributor

@bobhan1 bobhan1 commented Aug 22, 2024

This PR add the ability to update different columns for each row in one stream load
Doc: apache/doris-website#1140

Example

MySQL root@127.1:d1> CREATE TABLE t1 (
                  -> `k` int(11) NULL, 
                  -> `v1` BIGINT NULL,
                  -> `v2` BIGINT NULL DEFAULT "9876",
                  -> `v3` BIGINT NOT NULL,
                  -> `v4` BIGINT NOT NULL DEFAULT "1234",
                  -> `v5` BIGINT NULL
                  -> ) UNIQUE KEY(`k`) DISTRIBUTED BY HASH(`k`) BUCKETS 1
                  -> PROPERTIES(
                  -> "replication_num" = "1",
                  -> "enable_unique_key_merge_on_write" = "true"); 
Query OK, 0 rows affected
Time: 0.013s
MySQL root@127.1:d1> insert into t1 select number, number, number, number, number, number from numbers("number" = "6");
Query OK, 6 rows affected
Time: 0.107s
MySQL root@127.1:d1> select * from t1;
+---+----+----+----+----+----+
| k | v1 | v2 | v3 | v4 | v5 |
+---+----+----+----+----+----+
| 0 | 0  | 0  | 0  | 0  | 0  |
| 1 | 1  | 1  | 1  | 1  | 1  |
| 2 | 2  | 2  | 2  | 2  | 2  |
| 3 | 3  | 3  | 3  | 3  | 3  |
| 4 | 4  | 4  | 4  | 4  | 4  |
| 5 | 5  | 5  | 5  | 5  | 5  |
+---+----+----+----+----+----+

test1.json:

{"k": 1, "v1": 10}
{"k": 2, "v2": 20, "v5": 25}
{"k": 3, "v3": 30}
{"k": 4, "v4": 20, "v1": 43, "v3": 99}
{"k": 5, "v5": null}
{"k": 6, "v1": 999, "v3": 777}
{"k": 2, "v4": 222}
{"k": 1, "v2": 111, "v3": 111}
curl --location-trusted -u root: \
-H "strict_mode:false" \
-H "format:json" \
-H "read_json_by_line:true" \
-H "unique_key_update_mode:UPDATE_FLEXIBLE_COLUMNS" \
-T test1.json \
-XPUT http://<host>:<http_port>/api/d1/t1/_stream_load
MySQL root@127.1:d1> select * from t1;
+---+-----+------+-----+------+--------+
| k | v1  | v2   | v3  | v4   | v5     |
+---+-----+------+-----+------+--------+
| 0 | 0   | 0    | 0   | 0    | 0      |
| 1 | 10  | 111  | 111 | 1    | 1      |
| 2 | 2   | 20   | 2   | 222  | 25     |
| 3 | 3   | 3    | 30  | 3    | 3      |
| 4 | 43  | 4    | 99  | 20   | 4      |
| 5 | 5   | 5    | 5   | 5    | <null> |
| 6 | 999 | 9876 | 777 | 1234 | <null> |
+---+-----+------+-----+------+--------+

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

@bobhan1 bobhan1 force-pushed the mow-flexible-partial-update branch from bd91d77 to 40a5580 Compare August 22, 2024 04:06
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

@bobhan1 bobhan1 changed the title [Draft](partial update) Support flexible partial update in stream load with json files [Feature](partial update) Support flexible partial update in stream load with json files Aug 22, 2024
@bobhan1 bobhan1 force-pushed the mow-flexible-partial-update branch from 21b47f7 to edb8bf3 Compare August 22, 2024 07:50
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/olap/base_tablet.cpp Show resolved Hide resolved
@bobhan1 bobhan1 force-pushed the mow-flexible-partial-update branch from edb8bf3 to a82fc76 Compare August 22, 2024 10:59
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/olap/base_tablet.cpp Show resolved Hide resolved
@bobhan1 bobhan1 force-pushed the mow-flexible-partial-update branch 3 times, most recently from 001ea2a to 830813c Compare August 23, 2024 02:54
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/exec/tablet_info.h Show resolved Hide resolved
@bobhan1 bobhan1 force-pushed the mow-flexible-partial-update branch from 830813c to 3e7e9f5 Compare August 23, 2024 03:02
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/olap/partial_update_info.h Show resolved Hide resolved
@bobhan1 bobhan1 force-pushed the mow-flexible-partial-update branch 12 times, most recently from 92da59e to ccec48f Compare August 23, 2024 06:59
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

@bobhan1 bobhan1 force-pushed the mow-flexible-partial-update branch from ccec48f to 139660e Compare August 23, 2024 07:17
@bobhan1
Copy link
Contributor Author

bobhan1 commented Sep 29, 2024

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.23% (9630/25863)
Line Coverage: 28.57% (79732/279115)
Region Coverage: 28.01% (41213/147131)
Branch Coverage: 24.61% (20977/85230)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8ebeb14cc1bcbe0646f8f6da889cd443852c8c56_8ebeb14cc1bcbe0646f8f6da889cd443852c8c56/report/index.html

@bobhan1 bobhan1 force-pushed the mow-flexible-partial-update branch 3 times, most recently from 6fadcda to 9b533de Compare October 5, 2024 03:47
@bobhan1 bobhan1 force-pushed the mow-flexible-partial-update branch from 9b533de to 2627c19 Compare October 5, 2024 03:48
@bobhan1 bobhan1 force-pushed the mow-flexible-partial-update branch from 2627c19 to c905dbe Compare October 5, 2024 04:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants