add English doc

docs/data-operate/import/import-way/stream-load-manual.md

@@ -336,6 +336,7 @@ Determines whether to enable the Pipeline engine to execute Streamload tasks. Se
 | enclose                      | Specify the enclosure character. When a CSV data field contains a row delimiter or column delimiter, to prevent unexpected truncation, you can specify a single-byte character as the enclosure for protection. For example, if the column delimiter is "," and the enclosure is "'", the data "a,'b,c'" will have "b,c" parsed as a single field. Note: When the enclosure is set to a double quote ("), make sure to set `trim_double_quotes` to true. |
 | escape                       | Specify the escape character. It is used to escape characters that are the same as the enclosure character within a field. For example, if the data is "a,'b,'c'", and the enclosure is "'", and you want "b,'c" to be parsed as a single field, you need to specify a single-byte escape character, such as "", and modify the data to "a,'b','c'". |
 | memtable_on_sink_node        | Whether to enable MemTable on DataSink node when loading data, default is false. |
+|unique_key_update_mode        | The update modes on Unique tables, currently are only effective for Merge-On-Write Unique tables. Supporting three types: `UPSERT`, `UPDATE_FIXED_COLUMNS`, and `UPDATE_FLEXIBLE_COLUMNS`. `UPSERT`: Indicates that data is loaded with upsert semantics; `UPDATE_FIXED_COLUMNS`: Indicates that data is loaded through partial updates; `UPDATE_FLEXIBLE_COLUMNS`: Indicates that data is loaded through flexible partial updates.|
 
 ### Load return value
 
@@ -1064,7 +1065,7 @@ Doris supports a very rich set of column transformations and filtering operation
 
 The strict_mode attribute is used to set whether the import task runs in strict mode. This attribute affects the results of column mapping, transformation, and filtering, and it also controls the behavior of partial column updates. For specific instructions on strict mode, please refer to the [Strict Mode](../../../data-operate/import/load-strict-mode) documentation.
 
-### Perform partial column updates during import
+### Perform partial column updates/flexible partial update during import
 
 For how to express partial column updates during import, please refer to the Data Manipulation/Data Update documentation.
 

docs/data-operate/update/update-of-unique-model.md

@@ -155,4 +155,281 @@ Suggestions for improving load performance:
 
 Now, all rows in a batch write task (whether it is an load task or `INSERT INTO`) can only update the same columns. If you need to update different columns, you will need to perform separate batch writes.
 
-In the future, flexible column updates will be supported, allowing users to update different columns for each row within the same batch load.
+## Flexible Partial Column Updates
+
+Before version x.x.x, Doris's partial update feature required that every row in an import update the same columns. Starting from version x.x.x, Doris supports a more flexible partial update method that allows each row in a single import to update different columns.
+
+:::caution
+Note
+
+1. The flexible partial update feature is supported since version x.x.x.
+2. Currently, only the Stream Load import method and tools using Stream Load (e.g. Doris-Flink-Connector) support this feature.
+3. The import file must be in JSON format when using flexible column updates.
+:::
+
+### Applicable Scenarios
+
+When using CDC to synchronize data from a database system to Doris in real-time, the records output by the source system may not contain complete row data, but only the values of the primary keys and the updated columns. In such cases, the columns updated in a batch of data within a time window may differ. Flexible column updates can be used to import data into Doris.
+
+### Usage
+
+**Enabling Flexible Column Updates for Existing Tables**
+
+For existing Merge-On-Write tables created in old versions of Doris, after upgrading, you can enable flexible partial updates using the command: `ALTER TABLE db1.tbl1 ENABLE FEATURE "UPDATE_FLEXIBLE_COLUMNS";`. After executing this command, if the result of `show create table db1.tbl1` includes `"enable_unique_key_skip_bitmap_column" = "true"`, the feature has been successfully enabled. Ensure that the target table has the light-schema-change feature enabled beforehand.
+
+**Using Flexible Column Updates for New Tables**
+
+For new tables, to use the flexible column update feature, specify the following table properties when creating the table to enable Merge-on-Write, light-schema-change, and include the required hidden bitmap column for flexible column updates (the default value for these properties is `true`):
+
+```Plain
+"enable_unique_key_merge_on_write" = "true"
+"enable_unique_key_skip_bitmap_column" = "true"
+"enable_light_schema_change" = "true"
+```
+
+**StreamLoad**
+
+When using Stream Load, add the following header:
+
+```Plain
+unique_key_update_mode:UPDATE_FLEXIBLE_COLUMNS
+```
+
+**Flink Doris Connector**
+
+If using the Flink Doris Connector, add the following configuration:
+
+```Plain
+'sink.properties.unique_key_update_mode' = 'UPDATE_FLEXIBLE_COLUMNS'
+```
+
+### Example
+
+Assuming the following table:
+
+```sql
+CREATE TABLE t1 (
+  `k` int(11) NULL, 
+  `v1` BIGINT NULL,
+  `v2` BIGINT NULL DEFAULT "9876",
+  `v3` BIGINT NOT NULL,
+  `v4` BIGINT NOT NULL DEFAULT "1234",
+  `v5` BIGINT NULL
+) UNIQUE KEY(`k`) DISTRIBUTED BY HASH(`k`) BUCKETS 1
+PROPERTIES(
+"replication_num" = "3",
+"enable_unique_key_merge_on_write" = "true",
+"enable_light_schema_change" = "true",
+"enable_unique_key_skip_bitmap_column" = "true");
+```
+
+The original data in the table is:
+
+```sql
+MySQL root@127.1:d1> select * from t1;
++---+----+----+----+----+----+
+| k | v1 | v2 | v3 | v4 | v5 |
++---+----+----+----+----+----+
+| 0 | 0  | 0  | 0  | 0  | 0  |
+| 1 | 1  | 1  | 1  | 1  | 1  |
+| 2 | 2  | 2  | 2  | 2  | 2  |
+| 3 | 3  | 3  | 3  | 3  | 3  |
+| 4 | 4  | 4  | 4  | 4  | 4  |
+| 5 | 5  | 5  | 5  | 5  | 5  |
++---+----+----+----+----+----+
+```
+
+Now, updating some fields using flexible column updates:
+
+```shell
+$ cat test1.json
+```
+```json
+{"k": 0, "__DORIS_DELETE_SIGN__": 1}
+{"k": 1, "v1": 10}
+{"k": 2, "v2": 20, "v5": 25}
+{"k": 3, "v3": 30}
+{"k": 4, "v4": 20, "v1": 43, "v3": 99}
+{"k": 5, "v5": null}
+{"k": 6, "v1": 999, "v3": 777}
+{"k": 2, "v4": 222}
+{"k": 1, "v2": 111, "v3": 111}
+```
+```shell
+curl --location-trusted -u root: \
+-H "strict_mode:false" \
+-H "format:json" \
+-H "read_json_by_line:true" \
+-H "unique_key_update_mode:UPDATE_FLEXIBLE_COLUMNS" \
+-T test1.json \
+-XPUT http://<host>:<http_port>/api/d1/t1/_stream_load
+```
+
+After the update, the data in the table is:
+
+```sql
+MySQL root@127.1:d1> select * from t1;
++---+-----+------+-----+------+--------+
+| k | v1  | v2   | v3  | v4   | v5     |
++---+-----+------+-----+------+--------+
+| 1 | 10  | 111  | 111 | 1    | 1      |
+| 2 | 2   | 20   | 2   | 222  | 25     |
+| 3 | 3   | 3    | 30  | 3    | 3      |
+| 4 | 43  | 4    | 99  | 20   | 4      |
+| 5 | 5   | 5    | 5   | 5    | <null> |
+| 6 | 999 | 9876 | 777 | 1234 | <null> |
++---+-----+------+-----+------+--------+
+```
+
+### Limitations and Considerations
+
+1. Similar to previous partial updates, flexible column updates require that each row of imported data include all key columns. Rows not meeting this requirement will be filtered out and counted in filter rows. If the number of filtered rows exceeds the `max_filter_ratio` threshold for this import, the entire import will fail, and filtered data will generate an error log.
+
+2. In flexible partial update loads, key-value pairs in each JSON object are only valid if the key matches a column name in the target table. Key-value pairs that do not meet this requirement will be ignored. Pairs with keys `__DORIS_VERSION_COL__`, `__DORIS_ROW_STORE_COL__`, or `__DORIS_SKIP_BITMAP_COL__` will also be ignored.
+
+3. If the table properties of the target table include `function_column.sequence_type`, the import can specify the value for the `__DORIS_SEQUENCE_COL__` column by including a key-value pair in the JSON object with key `__DORIS_SEQUENCE_COL__`. For rows that do not specify a value for the `__DORIS_SEQUENCE_COL__`, if the key exists in the original table, the value will be filled from the old row; otherwise, it will be set to null.
+
+For example, for the following table:
+
+```sql
+CREATE TABLE t2 (
+  `k` int(11) NULL, 
+  `v1` BIGINT NULL,
+  `v2` BIGINT NULL DEFAULT "9876",
+  `v3` BIGINT NOT NULL,
+  `v4` BIGINT NOT NULL DEFAULT "1234",
+  `v5` BIGINT NULL
+) UNIQUE KEY(`k`) DISTRIBUTED BY HASH(`k`) BUCKETS 1
+PROPERTIES(
+"replication_num" = "3",
+"enable_unique_key_merge_on_write" = "true",
+"enable_light_schema_change" = "true",
+"enable_unique_key_skip_bitmap_column" = "true",
+"function_column.sequence_type" = "int");
+```
+
+The original data in the table is:
+
+```sql
++---+----+----+----+----+----+----------------------+
+| k | v1 | v2 | v3 | v4 | v5 |__DORIS_SEQUENCE_COL__|
++---+----+----+----+----+----+----------------------+
+| 0 | 0  | 0  | 0  | 0  | 0  | 0                    |
+| 1 | 1  | 1  | 1  | 1  | 10 | 10                   |
+| 2 | 2  | 2  | 2  | 2  | 20 | 20                   |
+| 3 | 3  | 3  | 3  | 3  | 30 | 30                   |
+| 4 | 4  | 4  | 4  | 4  | 40 | 40                   |
+| 5 | 5  | 5  | 5  | 5  | 50 | 50                   |
++---+----+----+----+----+----+----------------------+
+```
+
+Importing data using flexible partial column updates:
+
+```json
+{"k": 1, "v1": 111, "v5": 9, "__DORIS_SEQUENCE_COL__": 9}
+{"k": 2, "v2": 222, "v5": 25, "__DORIS_SEQUENCE_COL__": 25}
+{"k": 3, "v3": 333}
+{"k": 4, "v4": 444, "v5": 50, "v1": 411, "v3": 433, "v2": null, "__DORIS_SEQUENCE_COL__": 50}
+{"k": 5, "v5": null, "__DORIS_SEQUENCE_COL__": null}
+{"k": 6, "v1": 611, "v3": 633}
+{"k": 7, "v3": 733, "v5": 300, "__DORIS_SEQUENCE_COL__": 300}
+```
+
+The final data in the table is as follows:
+
+```sql
++---+--------+--------+-----+------+--------+
+| k | v1     | v2     | v3  | v4   | v5     |
++---+--------+--------+-----+------+--------+
+| 0 | 0      | 0      | 0   | 0    | 0      |
+| 1 | 1      | 1      | 1   | 1    | 1      |
+| 5 | 5      | 5      | 5   | 5    | 5      |
+| 2 | 2      | 222    | 2   | 2    | 25     |
+| 3 | 3      | 3      | 333 | 3    | 3      |
+| 4 | 411    | <null> | 433 | 444  | 50     |
+| 6 | 611    | 9876   | 633 | 1234 | <null> |
+| 7 | <null> | 9876   | 733 | 1234 | 300    |
++---+--------+--------+-----+------+--------+
+```
+
+4. When the table property `function_column.sequence_col` is set in the target table, the key-value pair in the JSON object for flexible column updates with the key `__DORIS_SEQUENCE_COL__` will be ignored. The value of the `__DORIS_SEQUENCE_COL__` column in a specific row during import will be the same of the final value of the column specified by the table property `function_column.sequence_col` for that row.
+
+For example, for the following table:
+
+```sql
+CREATE TABLE t3 (
+  `k` int(11) NULL, 
+  `v1` BIGINT NULL,
+  `v2` BIGINT NULL DEFAULT "9876",
+  `v3` BIGINT NOT NULL,
+  `v4` BIGINT NOT NULL DEFAULT "1234",
+  `v5` BIGINT NULL DEFAULT "31"
+) UNIQUE KEY(`k`) DISTRIBUTED BY HASH(`k`) BUCKETS 1
+PROPERTIES(
+"replication_num" = "3",
+"enable_unique_key_merge_on_write" = "true",
+"enable_light_schema_change" = "true",
+"enable_unique_key_skip_bitmap_column" = "true",
+"function_column.sequence_col" = "v5");
+```
+
+The original data in the table is as follows:
+
+```sql
++---+----+----+----+----+----+----------------------+
+| k | v1 | v2 | v3 | v4 | v5 |__DORIS_SEQUENCE_COL__|
++---+----+----+----+----+----+----------------------+
+| 0 | 0  | 0  | 0  | 0  | 0  | 0                    |
+| 1 | 1  | 1  | 1  | 1  | 10 | 10                   |
+| 2 | 2  | 2  | 2  | 2  | 20 | 20                   |
+| 3 | 3  | 3  | 3  | 3  | 30 | 30                   |
+| 4 | 4  | 4  | 4  | 4  | 40 | 40                   |
+| 5 | 5  | 5  | 5  | 5  | 50 | 50                   |
++---+----+----+----+----+----+----------------------+
+```
+
+Using flexible partial updates, the following data is imported:
+
+```json
+{"k": 1, "v1": 111, "v5": 9}
+{"k": 2, "v2": 222, "v5": 25}
+{"k": 3, "v3": 333}
+{"k": 4, "v4": 444, "v5": 50, "v1": 411, "v3": 433, "v2": null}
+{"k": 5, "v5": null}
+{"k": 6, "v1": 611, "v3": 633}
+{"k": 7, "v3": 733, "v5": 300}
+```
+
+The final data in the table is as follows:
+
+```sql
++---+--------+--------+-----+------+-----+
+| k | v1     | v2     | v3  | v4   | v5  |
++---+--------+--------+-----+------+-----+
+| 0 | 0      | 0      | 0   | 0    | 0   |
+| 1 | 1      | 1      | 1   | 1    | 10  |
+| 5 | 5      | 5      | 5   | 5    | 50  |
+| 2 | 2      | 222    | 2   | 2    | 25  |
+| 3 | 3      | 3      | 333 | 3    | 30  |
+| 4 | 411    | <null> | 433 | 444  | 50  |
+| 6 | 611    | 9876   | 633 | 1234 | 31  |
+| 7 | <null> | 9876   | 733 | 1234 | 300 |
++---+--------+--------+-----+------+-----+
+```
+
+5. When using flexible partial updates, the following import parameters cannot be specified or enabled:
+    - The `merge_type` parameter cannot be specified.
+    - The `delete` parameter cannot be specified.
+    - The `fuzzy_parse` parameter cannot be enabled.
+    - The `columns` parameter cannot be specified.
+    - The `jsonpaths` parameter cannot be specified.
+    - The `hidden_columns` parameter cannot be specified.
+    - The `function_column.sequence_col` parameter cannot be specified.
+    - The `sql` parameter cannot be specified.
+    - The `memtable_on_sink_node` option cannot be enabled.
+    - The `group_commit` parameter cannot be specified.
+    - The `where` parameter cannot be specified.
+
+6. Flexible partial updates are not supported on tables with Variant columns.
+
+7. Flexible partial updates are not supported on tables with synchronous materialized views.