Skip to content

Commit 319c8a0

Browse files
zxch3nclaude
andcommitted
docs: extract 11 key Loro concepts into dedicated concept documents
Extracted and reorganized key Loro concepts for better accessibility: High Priority Concepts: - Frontiers: Compact version representation using operation IDs - OpLog and DocState Separation: Architecture split between state and history - Attached/Detached States: Document version state and container association - Operations and Changes: How operations group and form changes Medium Priority Concepts: - Event Graph Walker: Novel CRDT algorithm using simple indices - Cursor and Stable Positions: Position references surviving concurrent edits - Import Status: Tracking successful imports and pending operations - Version Representations: Comparison between Version Vectors and Frontiers Low Priority Concepts: - Shallow Snapshots: Partial history snapshots with content redaction - PeerID Management: ID assignment and conflict avoidance strategies - Transaction Model: Operation bundling for event emission Changes made: - Created 11 new concept documents with beginner-friendly explanations - Set up redirects from original locations in advanced section - Updated navigation structure in _meta.js - Migrated relevant images and diagrams - Made documentation more progressive, starting simple and building complexity 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
1 parent c00665a commit 319c8a0

22 files changed

+5112
-395
lines changed
Lines changed: 1 addition & 88 deletions
Original file line numberDiff line numberDiff line change
@@ -1,88 +1 @@
1-
---
2-
keywords: "crdt, oplog, snapshot, doc state, checkout, version"
3-
description: "Introducing Loro's DocState and OpLog concepts"
4-
---
5-
6-
# DocState and OpLog
7-
8-
Although not explicitly exposed in the WASM interface, internally in Loro, we
9-
distinctly differentiate between:
10-
11-
- The current state of the document: DocState
12-
- The edit history of the document: OpLog
13-
14-
During local operations, we update the DocState and record the operations in
15-
OpLog. When merging remote updates, we add the new Ops to OpLog and compute a
16-
Delta. This Delta is applied to DocState and also emitted as an event.
17-
18-
DocState can switch between different versions, similar to Git's checkout. In
19-
this case, we calculate the Delta based on the edit history. The same mechanism
20-
applies: the Delta is emitted as an event and applied to DocState.
21-
22-
Impact on the encoding schema:
23-
24-
- When calling `doc.export({ mode: "update" })` or
25-
`doc.export({ mode: "update-in-range" })`, we only encode the operations that
26-
occurred after the specified version.
27-
- When calling `doc.export({ mode: "snapshot" })` or
28-
`doc.export({ mode: "shallow-snapshot" })`, we encode both OpLog and DocState,
29-
providing rapid loading speed (as it doesn't require recalculating the state
30-
of DocState).
31-
32-
## Attached/Detached LoroDoc Status
33-
34-
As we aim to support version control and the ability to load OpLog without
35-
state, the version of DocState and the latest version recorded in OpLog may not
36-
always match. When they align, it is in an _attached_ state; otherwise, it's in
37-
a _detached_ state.
38-
39-
```ts twoslash
40-
import { LoroDoc } from "loro-crdt";
41-
// ---cut---
42-
const doc = new LoroDoc();
43-
doc.setPeerId(1);
44-
doc.getText("text").insert(0, "Hello");
45-
const doc2 = doc.fork(); // create a fork of the doc
46-
console.log(doc.version().toJSON());
47-
// Map(1) { "1" => 5 }
48-
console.log(doc.oplogVersion().toJSON());
49-
// Map(1) { "1" => 5 }
50-
51-
doc.checkout([{ peer: "1", counter: 1 }]);
52-
console.log(doc.version().toJSON());
53-
// Map(1) { "1" => 2 }
54-
console.log(doc.oplogVersion().toJSON());
55-
// Map(1) { "1" => 5 }
56-
57-
doc2.setPeerId(2);
58-
doc2.getText("text").insert(5, "!");
59-
doc.import(doc2.export({ mode: "update" }));
60-
console.log(doc.version().toJSON());
61-
// Map(1) { "1" => 2 }
62-
console.log(doc.oplogVersion().toJSON());
63-
// Map(2) { "1" => 5, "2" => 1 }
64-
65-
console.log(doc.isDetached()); // true
66-
doc.attach();
67-
console.log(doc.version().toJSON());
68-
// Map(2) { "1" => 5, "2" => 1 }
69-
console.log(doc.oplogVersion().toJSON());
70-
// Map(2) { "1" => 5, "2" => 1 }
71-
72-
```
73-
74-
![DocState and OpLog Detached Example](./images/version-4.png)
75-
76-
The doc cannot be edited in the detached mode. Users must use `attach()` to
77-
return to the latest version to continue editing.
78-
79-
### Attached/Detached Container Status
80-
81-
This refers to whether a container is associated with a document. If not, it's a
82-
detached container created by methods like `new LoroText()`. The `.isAttached()`
83-
state never changes for an instance of a container.
84-
85-
When you insert a detached container into an attached one, you get a new
86-
attached container that has the same content as the detached one.
87-
88-
> This is different from the LoroDoc's attached/detached status
1+
export const redirect = '/docs/concepts/oplog_docstate'
Lines changed: 1 addition & 59 deletions
Original file line numberDiff line numberDiff line change
@@ -1,59 +1 @@
1-
---
2-
keywords: "crdt, event graph walker, eg-walker, synchronization, collaboration"
3-
description: "introduction to Event Graph Walker, a crdt algorithm for real-time collaboration and synchronization."
4-
---
5-
6-
# Brief Introduction to Event Graph Walker (Eg-Walker)
7-
8-
Eg-walker is a novel CRDT algorithm introduced in:
9-
10-
> [Collaborative Text Editing with Eg-walker: Better, Faster, Smaller](https://arxiv.org/abs/2409.14252)
11-
> By: Joseph Gentle, Martin Kleppmann
12-
13-
import { ReactPlayer } from "../../../components/video";
14-
15-
<ReactPlayer
16-
url="/static/REG.mp4"
17-
width={512}
18-
style={{maxWidth: "calc(100vw - 40px)"}}
19-
height={512}
20-
muted={true}
21-
loop={true}
22-
controls={true}
23-
playing={true}
24-
/>
25-
26-
Whether dealing with real-time collaboration or multi-end synchronization, a
27-
directed acyclic graph (DAG) forms over the history of these parallel edits,
28-
similar to Git's history. The Eg-walker algorithm records the history of user edits
29-
on the DAG. Unlike conventional CRDTs, Eg-walker can record just the original description
30-
of operations, not the metadata of CRDTs.
31-
32-
For instance, in text editing scenarios, the [RGA algorithm] needs the op ID and
33-
[Lamport timestamp][Lamport] of the character to the left to determine the
34-
insertion point. [Yjs]/Fugue, however, requires the op ID of both the left and
35-
right characters at insertion. In contrast, Eg-walker simplifies this by only
36-
recording the index at the time of insertion. Loro, which uses [Fugue] upon Eg-walker,
37-
inherits these advantages.
38-
39-
An index is not a stable position descriptor, as the index of an operation can
40-
be affected by other operations. For example, if you highlight content from
41-
`index=x` to `index=y`, and concurrently someone inserts n characters at
42-
`index=n` where `n<x`, then your highlighted range should shift to cover from
43-
`x+n` to `y+n`. However, Eg-walker can determine the exact position of this index and
44-
reconstruct the corresponding CRDT structure by replaying history.
45-
46-
Reconstructing history might seem time-consuming, but Eg-walker can backtrack only
47-
some. When merging updates from remote, it only needs to replay the operations
48-
between the current version and the remote version up to their lowest common ancestor,
49-
constructing a temporary CRDTs to calculate the effect of the remote update.
50-
51-
The Eg-walker algorithm excels with its fast local update speeds and eliminates
52-
concerns about tombstone collection in CRDTs. For instance, if an operation has
53-
been synchronized across all endpoints, no new operations will occur
54-
concurrently with it, allowing it to be safely removed from the history.
55-
56-
[Lamport]: https://en.wikipedia.org/wiki/Lamport_timestamp
57-
[Fugue]: https://arxiv.org/abs/2305.00583
58-
[RGA algorithm]: https://www.sciencedirect.com/science/article/abs/pii/S0743731510002716
59-
[Yjs]: https://github.com/yjs/yjs
1+
export const redirect = '/docs/concepts/event_graph_walker'
Lines changed: 1 addition & 144 deletions
Original file line numberDiff line numberDiff line change
@@ -1,144 +1 @@
1-
# Operations and Change
2-
3-
In Loro, every basic operation such as setting a key-value pair on a Map, adding
4-
a list item, or inserting/deleting a character in text is considered an
5-
individual op. (Don't worry about the cost, in Loro's internal memory
6-
representation and export format, consecutive ops are merged into a larger op,
7-
such as consecutive text insertions and deletions.)
8-
9-
One or more local consecutive `Op`s constitute a `Change`, which includes the
10-
following information:
11-
12-
- ID: ID of the Change is essentially the first op's ID
13-
- Timestamp: An optional timestamp, which can be enabled with
14-
`setRecordTimestamp(true)`. If not enabled, there is no extra storage
15-
overhead.
16-
- Dependency IDs: Used to represent the causal order, the Op IDs that the
17-
current Change directly depends on.
18-
- Commit Message: An optional commit message (WIP not yet released); when not
19-
enabled, there is no extra storage overhead.
20-
21-
Each time `doc.commit()` is called, a new `Change` is generated, which will be
22-
merged with the previous local `Change` as much as possible to reduce the amount
23-
of metadata that needs to be stored.
24-
25-
> Note: Each time you export, a `doc.commit()` is implicitly performed by the
26-
> Loro Doc.
27-
28-
Unlike a Git commit, Loro's Change can be merged; it is neither atomic nor
29-
indivisible. This design allows Loro to better accommodate real-time
30-
collaboration scenarios (where each keystroke would have its own `doc.commit()`,
31-
which would be hugely costly if not merged) and asynchronous collaboration
32-
scenarios (like Git, which combines many modifications to form one).
33-
34-
## When a New Change is Formed
35-
36-
> Note: You may not need to understand the content of this section, and the
37-
> content may change in future versions. Unless you want to understand Loro's
38-
> internal implementation or want to achieve more extreme performance
39-
> optimization.
40-
41-
By default, each commit-generated `Change` will merge with the previous local
42-
`Change`. However, there are exceptions in several cases:
43-
44-
- The current Change depends on a Change from a different peer. This occurs when
45-
local operations build upon recently applied remote operations. For example,
46-
deleting a character sequence that was just inserted by a remote peer. These
47-
causal relationships form a DAG (Directed Acyclic Graph). After importing
48-
remote updates, the next local Change will have new dependency IDs,
49-
necessitating a separate Change.
50-
- When `setRecordTimestamp(true)` is set, if the time interval between
51-
successive Changes exceeds the "change merge interval" (default duration
52-
1000s).
53-
- When the current Change has a different commit message from the previous
54-
Change by the same peer.
55-
56-
## Example
57-
58-
```ts twoslash
59-
import { Change, LoroDoc } from "loro-crdt";
60-
// ---cut---
61-
const docA = new LoroDoc();
62-
docA.setPeerId("0");
63-
const textA = docA.getText("text");
64-
// This create 3 operations
65-
textA.insert(0, "123");
66-
// This create a new Change
67-
docA.commit();
68-
// This create 2 operations
69-
textA.insert(0, "ab");
70-
// This will NOT create a new Change
71-
docA.commit();
72-
73-
{
74-
const changeMap: Map<`${number}`, Change[]> = docA.getAllChanges();
75-
console.log(changeMap);
76-
// Output:
77-
//
78-
// Map(1) {
79-
// "0" => [
80-
// {
81-
// lamport: 0,
82-
// length: 5,
83-
// peer: "0",
84-
// counter: 0,
85-
// deps: [],
86-
// timestamp: 0
87-
// }
88-
// ]
89-
// }
90-
}
91-
92-
// Create docB from doc
93-
const docB = LoroDoc.fromSnapshot(docA.export({ mode: "snapshot" }));
94-
docB.setPeerId("1");
95-
const textB = docB.getText("text");
96-
// This create 2 operations
97-
textB.insert(0, "cd");
98-
99-
// Import the Change from docB to doc
100-
const bytes = docB.export({ mode: "update" }); // Exporting has implicit commit
101-
docA.import(bytes);
102-
103-
// This create 1 operations
104-
textA.insert(0, "1");
105-
// Because doc import a Change from docB, it will create a new Change for
106-
// new commit to record this causal order
107-
docA.commit();
108-
{
109-
const changeMap: Map<`${number}`, Change[]> = docA.getAllChanges();
110-
console.log(changeMap);
111-
// Output:
112-
//
113-
// Map(2) {
114-
// "0" => [
115-
// {
116-
// lamport: 0,
117-
// length: 5,
118-
// peer: "0",
119-
// counter: 0,
120-
// deps: [],
121-
// timestamp: 0
122-
// },
123-
// {
124-
// lamport: 7,
125-
// length: 1,
126-
// peer: "0",
127-
// counter: 5,
128-
// deps: [ { peer: "1", counter: 1 } ],
129-
// timestamp: 0
130-
// }
131-
// ],
132-
// "1" => [
133-
// {
134-
// lamport: 5,
135-
// length: 2,
136-
// peer: "1",
137-
// counter: 0,
138-
// deps: [ { peer: "0", counter: 4 } ],
139-
// timestamp: 0
140-
// }
141-
// ]
142-
// }
143-
}
144-
```
1+
export const redirect = '/docs/concepts/operations_changes'

0 commit comments

Comments
 (0)