Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add tree document #608

Merged
merged 11 commits into from
Aug 25, 2023
Merged

Add tree document #608

Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
6d86852
Create template
MoonGyu1 Aug 16, 2023
285c11f
Create template
MoonGyu1 Aug 16, 2023
b376d03
Add draft docs
MoonGyu1 Aug 25, 2023
ffed1e7
Merge conflict
MoonGyu1 Aug 25, 2023
9af9634
Add images
MoonGyu1 Aug 25, 2023
105bead
Add path example
MoonGyu1 Aug 25, 2023
9bcfa93
Add code lines
MoonGyu1 Aug 25, 2023
a3170de
Update data structure related to Tree
MoonGyu1 Aug 25, 2023
2691c5f
Replace JSON-like tree to XML-like tree
MoonGyu1 Aug 25, 2023
2fa0a56
Rename docs concurrent-tree-editing to tree
MoonGyu1 Aug 25, 2023
0293c8d
Remove link at Clone
MoonGyu1 Aug 25, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 10 additions & 5 deletions design/data-structure.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: data-structure
target-version: 0.3.1
target-version: 0.4.6
---

# Data Structures
Expand Down Expand Up @@ -28,7 +28,9 @@ The `json` and `crdt` package has data structures for representing the contents

Below is the dependency graph of data structures used in a JSON-like document.

![data-structure](./media/data-structure.png)

<img src="https://github.com/yorkie-team/yorkie/assets/78714820/d91aebad-4637-4873-803e-cda025ade1f6" width="700" />


The data structures can be divided into three groups:

Expand All @@ -45,7 +47,9 @@ JSON-like data strucutres are used when editing JSON-like documents.
- `Primitive`: represents primitive data like `string`, `number`, `boolean`, `null`, etc.
- `Object`: represents [object type](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object) of JavaScript. Just like JavaScript, you can use `Object` as [hash table](https://en.wikipedia.org/wiki/Hash_table).
- `Array`: represents [array type](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array) of JavaScript. You can also use `Array` as [list](https://en.wikipedia.org/wiki/List_(abstract_data_type)).
- `Text`: represents text with style attributes in rich text editors such as [Quill](https://github.com/yorkie-team/yorkie-js-sdk/blob/main/examples/quill.html). Users can express styles such as bold, italic, and underline to text content. Of course, it can represent just a plain text in text-based editors such as [CodeMirror](https://github.com/yorkie-team/yorkie-js-sdk/blob/main/examples/index.html). It supports collaborative editing; multiple users can modify parts of the contents without conflict.
- `Text`: represents text with style attributes in rich text editors such as [Quill](https://quilljs.com/). Users can express styles such as bold, italic, and underline to text content. Of course, it can represent just a plain text in text-based editors such as [CodeMirror](https://codemirror.net). It supports collaborative editing; multiple users can modify parts of the contents without conflict.
- `Counter`: represents a counter in the document. As a proxy for the CRDT counter, it is used when the user manipulates the counter from the outside.
- `Tree`: represents CRDT-based tree structure that is used to represent the document tree of text-based editor such as [ProseMirror](https://prosemirror.net/).

JSON-like data structures can be edited through proxies. For example:

Expand All @@ -72,15 +76,16 @@ CRDT data structures are used by JSON-like group to resolve conflicts in concurr
- `ElementRHT`: similar to `RHT`, but has elements as values.
- `RGATreeList`: extended `RGA(Replicated Growable Array)` with an additional index tree. The index tree manages the indices of elements and provides faster access to elements at the int-based index.
- `RGATreeSplit`: extended `RGATreeList` allowing characters to be represented as blocks rather than each single character.

- `CRDTTree`: represents the CRDT tree with an index tree structure'. It resolves conflicts arising from concurrent editing.
### Common Group

Common data structures can be used for general purposes.

- [`SplayTree`](https://en.wikipedia.org/wiki/Splay_tree): A tree that moves nodes to the root by splaying. This is effective when user frequently access the same location, such as text editing. We use `SplayTree` as an index tree to give each node a weight, and to quickly access the node based on the index.
- [`LLRBTree`](https://en.wikipedia.org/wiki/Left-leaning_red%E2%80%93black_tree): A tree simpler than Red-Black Tree. Newly added `floor` method finds the node of the largest key less than or equal to the given key.
- [`Trie`](https://en.wikipedia.org/wiki/Trie): A data structure that can quickly search for prefixes of sequence data such as strings. We use `Trie` to remove nested events when the contents of the `Document` are modified at once.

- `IndexTree`: A tree implementation to represent a document of text-based editors.

### Risks and Mitigation

We can replace the data structures with better ones for some reason, such as performance. For example, `SplayTree` used in `RGATreeList` can be replaced with [TreeList](https://commons.apache.org/proper/commons-collections/apidocs/org/apache/commons/collections4/list/TreeList.html).
166 changes: 166 additions & 0 deletions design/tree.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,166 @@
---
title: tree
target-version: 0.4.6
---

# Tree

## Summary

In Yorkie, users can create and edit JSON-like documents using JSON-like data structures such as `Primitive`, `Object`, `Array`, `Text`, and `Tree`. Among these, the `Tree` structure is used to represent the document model of a tree-based text editor, similar to XML.

This document introduces the `Tree` data structure, and explains the operations provided by `Tree`, focusing on the `Tree` coordinate system and the logic of the `Tree.Edit` operation. Furthermore, it explains how this logic ensures eventual consistency in concurrent document editing scenarios.

### Goals

This document aims to help new SDK contributors understand the overall `Tree` data structure and explain how Yorkie ensures consistency when multiple clients are editing concurrently.

### Non-Goals

This document focuses on `Tree.Edit` operations rather than `Tree.Style`.

## Proposal Details

### XML-like Tree

In yorkie, a XML-like `Tree` is used to represent the document model of a tree-based text editor.

This tree-based document model resembles XML tree and consists of element nodes and text nodes. element nodes can have attributes, and text nodes contain a string as their value. For example:

<img src="https://github.com/yorkie-team/yorkie/assets/78714820/b5500d8b-43db-4d89-983d-5708b7041cc4" width="550" />

**Operation**

The XML-like `Tree` provides specialized operations tailored for text editing rather than typical operations of a general tree. To specify the operation's range, an `index` or `path` is used. For example:

<img src="https://github.com/yorkie-team/yorkie/assets/78714820/9b3420e9-6c18-4114-9f4f-1c8e02324dd6" width="550" />

These `index`es are assigned in order at positions where the user's cursor can reach. These `index`es draw inspiration from ProseMirror's index and share a similar structural concept.

In the case of a `path`, it contains `offset`s of each node from the root node as elements except the last. The last element of the `path` represents the position in the parent node. For example, the `path` of the position between '`k`' and '`i`' is `[1, 4]`. The first element of the `path` is the `offset` of the `<b>` in `<p>` and the second element represents the position between '`k`' and '`i`' in `<b>`.

1. `Tree.Edit`

Users can use the `Edit` operation to insert or delete nodes within the `Tree`.

https://github.com/yorkie-team/yorkie/blob/fd3b15c7d2c482464b6c8470339bcc497204114e/pkg/document/json/tree.go#L115-L131

Where `fromIdx` is the starting position of editing, `toIdx` is the ending position, and `contents` represent the nodes to be inserted. If `contents` are omitted, the operation only deletes nodes between `fromIdx` and `toIdx`.

<img src="https://github.com/yorkie-team/yorkie/assets/78714820/c1184839-3e50-41fa-b558-8c0677285660" width="450" />

Similarly, users can specify the editing range using a `path` that leads to the `Tree`'s node in the type of `[]int`.

https://github.com/yorkie-team/yorkie/blob/fd3b15c7d2c482464b6c8470339bcc497204114e/pkg/document/json/tree.go#L217-L237

2. `Tree.Style`

Users can use the `Style` operation to specify attributes for the element nodes in the `Tree`.

https://github.com/yorkie-team/yorkie/blob/fd3b15c7d2c482464b6c8470339bcc497204114e/pkg/document/json/tree.go#L239-L268

### Implementation of Edit Operation

**Tree Coordinate System**

<img src="https://github.com/yorkie-team/yorkie/assets/78714820/08c1e917-08cf-492c-84c2-cf72b98c38f3" width="600" />

Yorkie implements the above [data structure](https://github.com/yorkie-team/yorkie/blob/main/design/data-structure.md) to create a JSON-like `Document`, which consists of different layers, each with its own coordinate system. The dependency graph above can be divided into three main groups. The **JSON-like** group directly used by users to edit JSON-like `Document`s. The **CRDT** Group is utilized from the JSON-like group to resolve conflicts in concurrent editing situations. Finally, the **common** group is used for the detailed implementation of CRDT group and serves general purposes.

Thus, the `Tree`, introduced in this document, has dependencies such as '`Tree` → `CRDTTree` → `IndexTree`', and each layer has its own coordinate system:

<img src="https://github.com/yorkie-team/yorkie/assets/78714820/33519a1e-c8cb-4b4d-9d0e-d2fcc2052013" width="450" />

These coordinate systems transform in the order of '`index(path)` → `IndexTree.TreePos` → `CRDTTree.TreeNodeID` → `CRDTTree.TreePos`'.

<img src="https://github.com/yorkie-team/yorkie/assets/78714820/af339dc7-5c03-4cae-a1cb-f5879bfce3be" />

1. `index` → `IndexTree.TreePos`

The `index` is the coordinate system used by users for local editing. This `index` is received from the user, and is converted to `IndexTree.TreePos`. This `IndexTree.TreePos` represents the physical position within the local tree and is used for actual tree editing.

2. `IndexTree.TreePos` → (`CRDTTree.TreeNodeID`) → `CRDTTree.TreePos`

Next, the obtained `IndexTree.TreePos` is transformed into the logical coordinate system of the distributed tree, represented by `CRDTTree.TreePos`. To achieve this, the given physical position, `IndexTree.TreePos`, is used to find the parent node and left sibling node. Then, a `CRDTTree.TreePos` is created using the unique IDs of the parent node and left sibling node, which are `CRDTTree.TreeNodeID`. This coordinate system is used in subsequent `Tree.Edit` and `Tree.Style` operations.

In the case of remote editing, where the local coordinate system is received from the user in local editing, there is no need for Step 1 since changes are pulled from the server using `ChangePack` to synchronize the changes. Refer to the [document-editing](https://github.com/yorkie-team/yorkie/blob/main/design/document-editing.md) for more details.

**Tree.Edit Logic**

The core process of the `Tree.Edit` operation is as follows:

1. Find `CRDTTree.TreePos` from the given `fromIdx` and `toIdx` (local editing only).
2. Find the corresponding left sibling node and parent node within the `IndexTree` based on `CRDTTree.TreePos`.
3. Delete nodes in the range of `fromTreePos` to `toTreePos`.
4. Insert the given nodes at the appropriate positions (insert operation only).

**[[STEP 1]](https://github.com/yorkie-team/yorkie/blob/fd3b15c7d2c482464b6c8470339bcc497204114e/pkg/document/json/tree.go#L121C1-L128)** Find `CRDTTree.TreePos` from the given `fromIdx` and `toIdx` (local editing only)

In the case of local editing, the given `index`es are converted to `CRDTTree.TreePos`. The detailed process is the same as described in the 'Tree Coordinate System' above.

**[[STEP 2]](https://github.com/yorkie-team/yorkie/blob/fd3b15c7d2c482464b6c8470339bcc497204114e/pkg/document/crdt/tree.go#L572C1-L580C3)** Find the corresponding left sibling node and parent node within the `IndexTree` based on `CRDTTree.TreePos`

2-1. For text nodes, if necessary, split nodes at the appropriate positions to find the left sibling node.

2-2. Determine the sequence of nodes and find the appropriate position. Since `Clone`s[링크] of each client might exist in different states, the `findFloorNode` function is used to find the closest node (lower bound).
hackerwins marked this conversation as resolved.
Show resolved Hide resolved

**[[STEP 3]](https://github.com/yorkie-team/yorkie/blob/fd3b15c7d2c482464b6c8470339bcc497204114e/pkg/document/crdt/tree.go#L582-L640)** Delete nodes in the range of `fromTreePos` to `toTreePos`

3-1. Traverse the range and identify nodes to be removed. If a node is an element node and doesn't include both opening and closing tags, it is excluded from removal.

3-2. Update the `latestCreatedAtMapByActor` information for each node and mark nodes with tombstones in the `IndexTree` to indicate removal.

**[[STEP 4]](https://github.com/yorkie-team/yorkie/blob/fd3b15c7d2c482464b6c8470339bcc497204114e/pkg/document/crdt/tree.go#L642-L681)** Insert the given nodes at the appropriate positions (insert operation only)

4-1. If the left sibling node at the insertion position is the same as the parent node, it means the node will be inserted as the leftmost child of the parent. Hence, the node is inserted at the leftmost position of the parent's children list.

4-2. Otherwise, the new node is inserted to the right of the left sibling node.

### How to Guarantee Eventual Consistency

**Coverage**

<img src="https://github.com/yorkie-team/yorkie/assets/78714820/c911ffb4-9021-4a1f-9a11-b8e28fb41435" width="750" >

Using conditions such as range type, node type, and edit type, 27 possible cases of concurrent editing can be represented.

<img src="https://github.com/yorkie-team/yorkie/assets/78714820/d1054938-b701-4e90-bcbb-8ff5d62b19d4" width="400">

Eventual consistency is guaranteed for these [27 cases](https://github.com/yorkie-team/yorkie/blob/fd3b15c7d2c482464b6c8470339bcc497204114e/test/integration/tree_test.go#L736-L2094). In addition, eventual consistency is ensured for the following edge cases:

- Selecting multiple nodes in a multi-level range
- Selecting only a part of nodes (e.g., selecting only the opening tag or closing tag of the node)

**How does it work?**

- `lastCreatedAtMapByActor`

https://github.com/yorkie-team/yorkie/blob/81137b32d0d1d3d36be5b63652e5ab0273f536de/pkg/document/operations/tree_edit.go#L36-L38

`latestCreatedAtMapByActor` is a map that stores the latest creation time by actor for the nodes included in the editing range. However, relying solely on the typical `lamport` clocks that represent local clock of clients, it's not possible to determine if two events are causally related or concurrent. For instance:

<img src="https://github.com/yorkie-team/yorkie/assets/78714820/cc025542-2c85-40ef-b846-157f38177487" width="450" />

In the case of the example above, during the process of synchronizing operations between clients A and B, client A is unaware of the existence of '`c`' when client B performs `Edit(0,2)`. As a result, an issue arises where the element '`c`', which is within the contained range, gets deleted together.

To address this, the `lastCreatedAtMapByActor` is utilized during operation execution to store final timestamp information for each actor. Subsequently, this information allows us to ascertain the causal relationship between the two events.

- Restricted to only `insertAfter`

https://github.com/yorkie-team/yorkie/blob/422901861aedbd3a86fdcb9cf3b5740d6daf38eb/pkg/index/tree.go#L552-L570

To ensure consistency in concurrent editing scenarios, only the `insertAfter` operation is allowed, rather than `insertBefore`, similar to conventional CRDT algorithms. To achieve this, `CRDTTree.TreePos` takes a form that includes `LeftSiblingID`, thus always maintaining a reference to the left sibling node.

If the left sibling node is the same as the parent node, it indicates that the node is positioned at the far left of the parent's children list.

- `FindOffset`

https://github.com/yorkie-team/yorkie/blob/422901861aedbd3a86fdcb9cf3b5740d6daf38eb/pkg/index/tree.go#L393-L412

During the traversal of the given range in `traverseInPosRange` (STEP3), the process of converting the provided `CRDTTree.TreePos` to an `IndexTree.TreePos` is executed. To determine the `offset` for this conversion, the `FindOffset` function is utilized. In doing so, calculating the `offset` excluding the removed nodes prevents potential issues that can arise in concurrent editing scenarios.

### Risks and Mitigation

- In the current conflict resolution policy of Yorkie, when both insert and delete operations occur simultaneously, even if the insert range is included in the delete range, the inserted node remains after synchronization. This might not always reflect the user's intention accurately.

- The `Tree.Edit` logic uses index-based traversal instead of node-based traversal for a clearer implementation. This might lead to a performance impact. If this becomes a concern, switching to node-based traversal can be considered.