Movable List

[zxch3n]

Background

Loro's List supports insert and delete operations but lacks built-in methods for set and move. To simulate set and move, developers might combine delete and insert. However, this approach can lead to issues during concurrent operations on the same element, often resulting in duplicate entries upon merging.

For instance, consider a list [0, 1, 2]. If user A moves the element '0' to position 1, while user B moves it to position 2, the ideal merged outcome should be either [1, 0, 2] or [1, 2, 0]. However, using the delete-insert method to simulate a move results in [1, 0, 2, 0], as both users delete '0' from its original position and insert it independently at new positions.

To address this, we introduce a MovableList container. This new container type directly supports move and set operations, aligning more closely with user expectations and preventing the issues associated with simulated moves.

Design

Algorithm Introduction

The algorithm is based on Moving Elements in List CRDTs by Martin Kleppmann.

It uses a List CRDT to provide "stable position representation." Then, it records the position and value with an additional LWW Map. When users move an item, it creates a new list item on the List CRDT and removes the old one, and update the LWW Map with the new position ID.

Thus, it's implemented by 3 CRDTs:

• Modeling Move with a List CRDT, where each move involves deleting the item from its original position and inserting a new item at a new position.
• Modeling the mappings Element ID → List Item ID and Element ID → Value with two LWW Maps. When a Move occurs, it updates the mapping to List Item ID; when a set op occurs, it updates the mapping to Value.

List Item and Element

We use three CRDTs in the algorithm: one CRDT maintains the order of the List, one maintains the mapping from ElementID to the List, and one maintains the mapping from ElementID to Value.

We refer to elements on the List as List Items; each item only has ID information without a specific value.

For the two LWW Maps, we store them together in an Element object.

Two Indexing Methods

In the case of concurrent moving, movable list items are not directly removed from the document, so there will be extra list Items on the document with no Elements pointing to them. These are imperceptible to the users. However, for the List Diff Calculator, whether an Element points to a List Item is imperceptible, so the Diff it produces cannot determine whether a List Item is needless. Therefore, in the following scenario,

Origin:                 [x, y, z]
Concurrent A: move 0->2 [y, z, x]
Concurrent B: move 0->1 [y, x, z]

• In the merged version, from the user's perspective, the List length is 3, but for the DiffCalculator, the length is 4.
• In the merged version, the index of C for the user is 1, but for the DiffCalculator, it is 2.

We refer to the index from the user's perspective as UserIndex, and the one stored internally for the DiffCalculator as OpIndex.

Design of Internal Event & External Event

Internal Event is used by the DiffCalculator to pass state changes to the State, while External Event is used by the State to communicate its changes to the outside world.

The design of the Internal Event includes the following:

• Changes in value and pos on the Element
• Changes in the elements of the List Items list

External Event continues to use a format similar to Quill Delta but adds Meta information to mark whether the creation is a result of Move behavior.

Implementation of DiffCalculator

The calculation of Internal Events is the task of the DiffCalculator.

• The change calculation of the List Items list follows the original logic of the List CRDT, essentially consistent internally
• The changes in the Elements' Registers follow the design of the Registers on the Map

Structure of MovableList State in Memory

• Uses BTree for efficient querying and random position insertion/deletion
  • It also supports querying using two different indexing methods, reusing the data structure we have for Text
    • Text also needed to support efficient indexing of specific positions through different encoding methods
• Allows lookup of an element's position in the BTree by id

Encoding of Operations

• Information to be encoded for Move operations includes:
  • from_index: usize
  • to_index: usize
  • elem_id: IdLp
• Information to be encoded for Set operations includes:
  • elem_id: IdLp
  • value: LoroValue

Storage of Snapshot

The Snapshot needs to encode all List Items, including those not visible to the user, into a separate vec

#[columnar(vec, ser, de, iterable)]
#[derive(Debug, Clone, Copy)]
struct EncodedItem {
    #[columnar(strategy = "DeltaRle")]
    invisible_list_item: usize,
    #[columnar(strategy = "BoolRle")]
    pos_id_eq_elem_id: bool,
}

#[columnar(vec, ser, de, iterable)]
#[derive(Debug, Clone)]
struct EncodedId {
    #[columnar(strategy = "DeltaRle")]
    peer_idx: usize,
    #[columnar(strategy = "DeltaRle")]
    lamport: u32,
}

#[columnar(ser, de)]
struct EncodedSnapshot {
    #[columnar(class = "vec", iter = "EncodedItem")]
    items: Vec<EncodedItem>,
    #[columnar(class = "vec", iter = "EncodedId")]
    ids: Vec<EncodedId>,
}

Implementation of Handler

Due to the "Two Indexing Method," the index information exposed to users is different from the index information used for the op. Because the Handler is where users directly interact with Loro, we let index conversions happen at the Handler.

Examples

import { describe, expect, it } from "vitest";
import { Delta, ListDiff, Loro, TextDiff } from "loro-crdt";
import {
  Cursor,
  LoroList,
  LoroMovableList,
  OpId,
  PeerID,
  setDebug,
} from "loro-crdt";

describe("movable list", () => {
  it("should work like list", () => {
    const doc = new Loro();
    const list = doc.getMovableList("list");
    expect(list.length).toBe(0);
    list.push("a");
    expect(list.length).toBe(1);
    expect(list.get(0)).toBe("a");
    let v = list.pop();
    expect(list.length).toBe(0);
    expect(v).toBe("a");
  });

  it("can be synced", () => {
    const doc = new Loro();
    const list = doc.getMovableList("list");
    list.push("a");
    list.push("b");
    list.push("c");
    expect(list.toArray()).toEqual(["a", "b", "c"]);
    list.set(2, "d");
    list.move(0, 1);
    const doc2 = new Loro();
    const list2 = doc2.getMovableList("list");
    expect(list2.length).toBe(0);
    doc2.import(doc.exportFrom());
    expect(list2.length).toBe(3);
    expect(list2.get(0)).toBe("b");
    expect(list2.get(1)).toBe("a");
    expect(list2.get(2)).toBe("d");
  });

  it("should support move", () => {
    const doc = new Loro();
    const list = doc.getMovableList("list");
    list.push("a");
    list.push("b");
    list.push("c");
    expect(list.toArray()).toEqual(["a", "b", "c"]);
    list.move(0, 1);
    expect(list.toArray()).toEqual(["b", "a", "c"]);
  });

  it("should support set", () => {
    const doc = new Loro();
    const list = doc.getMovableList("list");
    list.push("a");
    list.push("b");
    list.push("c");
    expect(list.toArray()).toEqual(["a", "b", "c"]);
    list.set(1, "d");
    expect(list.toArray()).toEqual(["a", "d", "c"]);
  });

  it.todo("should support get cursor", () => {
    const doc = new Loro();
    doc.setPeerId(1);
    const list = doc.getMovableList("list");
    list.push("a");
    list.push("b");
    list.push("c");
    expect(list.toArray()).toEqual(["a", "b", "c"]);
    const cursor = list.getCursor(1)!;
    const ans = doc.getCursorPos(cursor);
    expect(ans.offset).toBe(1);
    expect(ans.update).toBeFalsy();

    // cursor position should not be affected by set and move
    list.set(1, "d");
    list.move(1, 2);
    const ans2 = doc.getCursorPos(cursor);
    expect(ans2.offset).toBe(1);
    expect(ans2.update).toBeTruthy();
    const pos = ans2.update?.pos();
    expect(pos).toStrictEqual({ peer: "1", counter: 4 });
  });

  it("inserts sub-container", () => {
    const doc = new Loro();
    const list = doc.getMovableList("list");
    list.push("a");
    list.push("b");
    list.push("c");
    const subList = list.insertContainer(1, new LoroList());
    subList.push("d");
    subList.push("e");
    subList.push("f");
    expect(list.toJson()).toEqual(["a", ["d", "e", "f"], "b", "c"]);
    list.move(1, 0);
    expect(list.toJson()).toEqual([["d", "e", "f"], "a", "b", "c"]);
    list.move(0, 3);
    expect(list.toJson()).toEqual(["a", "b", "c", ["d", "e", "f"]]);
  });

  it("can be inserted into a list as an attached container", () => {
    const doc = new Loro();
    const list = doc.getMovableList("list");
    list.push("a");
    list.push("b");
    list.push("c");
    const blist = doc.getList("blist");
    const newList: LoroMovableList = blist.insertContainer(0, list);
    expect(blist.toJson()).toEqual([["a", "b", "c"]]);
    newList.move(0, 1);
    expect(blist.toJson()).toEqual([["b", "a", "c"]]);
    list.move(0, 2);
    // change on list should not affect blist
    expect(blist.toJson()).toEqual([["b", "a", "c"]]);
  });

  it("length should be correct when there are concurrent move", () => {
    const docA = new Loro();
    const list = docA.getMovableList("list");
    list.push("a");
    list.push("b");
    list.push("c");
    const docB = new Loro();
    const listB = docB.getMovableList("list");
    docB.import(docA.exportFrom());
    listB.move(0, 1);
    list.move(0, 1);
    docB.import(docA.exportFrom());
    expect(listB.toJson()).toEqual(["b", "a", "c"]);
    expect(listB.length).toBe(3);
  });

  it("concurrent set the one with larger peer id win", () => {
    const docA = new Loro();
    docA.setPeerId(0);
    const listA = docA.getMovableList("list");
    listA.push("a");
    listA.push("b");
    listA.push("c");
    const docB = new Loro();
    docB.setPeerId(1);
    const listB = docB.getMovableList("list");
    docB.import(docA.exportFrom());
    listA.set(1, "fromA");
    listB.set(1, "fromB");
    docB.import(docA.exportFrom());
    docA.import(docB.exportFrom());
    expect(listA.toJson()).toEqual(["a", "fromB", "c"]);
    expect(listA.length).toBe(3);
    expect(listB.toJson()).toEqual(["a", "fromB", "c"]);
    expect(listB.length).toBe(3);
  });

  it("can be subscribe", async () => {
    const doc = new Loro();
    const list = doc.getMovableList("list");
    list.push("a");
    list.push("b");
    list.push("c");
    let called = false;
    const id = list.subscribe((event) => {
      expect(event.by).toBe("local");
      for (const e of event.events) {
        expect(e.target).toBe(list.id);
        if (e.diff.type === "list") {
          expect(e.diff).toStrictEqual(
            {
              "type": "list",
              "diff": [{ insert: ["a", "b", "c"] }],
            } as ListDiff,
          );
        } else {
          throw new Error("unknown diff type");
        }
      }

      called = true;
    });
    await new Promise((r) => setTimeout(r, 1));
    expect(called).toBeFalsy();
    doc.commit();
    await new Promise((r) => setTimeout(r, 1));
    expect(called).toBeTruthy();
  });
});

Performance

Tested on M1. The duration is recorded in ms.

The cost of MovableList is close to List.

task	snapshot_size	updates_size	apply_duration	encode_snapshot_duration	encode_udpate_duration	decode_snapshot_duration	decode_update_duration	doc_json_size
[Movable List] Append x 10_000	41886	31850	5.942708	4.215083	0.182833	6.108375	23.030208	48900
[Movable List] Prepend x 10_000	41886	41862	5.806541	4.8026670000000005	1.014	6.647958	31.083790999999998	48900
[Movable List] Random Insert x 10_000	81453	61292	7.189042	6.26075	1.209084	7.1939589999999995	32.932207999999996	48900
[Movable List] Random Insert&Delete x 10_000	100935	100929	24.972082999999998	5.406916000000001	2.238042	6.055124999999999	26.851375	11
[Movable List] Random Insert&Delete x 100_000	1291077	1291071	361.367875	71.357083	27.313750000000002	78.415541	418.25437500000004	11
[Movable List] Collab Insert x 100_000	1754015	1224751	1634.7663340000001	167.83241700000002	39.300084	255.554666	1344.622417	580010
[List] Append x 10_000	41872	31850	4.209541	1.812583	0.177	4.4605	2.80225	48900
[List] Prepend x 10_000	41872	41862	3.0690410000000004	2.060625	1.107958	4.623875	14.678292	48900
[List] Random Insert x 10_000	81422	61283	3.7324580000000003	3.3357080000000003	1.259166	5.297458	15.530209	48900
[List] Random Insert&Delete x 10_000	100935	100929	10.380709	4.1217500000000005	2.313084	3.897375	20.73425	11
[List] Random Insert&Delete x 100_000	1291077	1291071	137.486417	37.102167	28.856832999999998	51.214957999999996	310.83558300000004	11
[List] Collab Insert x 100_000	1754000	1224751	816.7858329999999	130.287958	37.529832999999996	150.925792	657.4695409999999	580010
[Movable List] Collab Set x 100_000	999436	992780	314.677916	41.915833	28.737083	62.689792	71.34575	4010
[Movable List] Collab Move x 100_000	1817009	1608197	2426.7491250000003	55.0175	34.364459	100.102042	1316.108833	7790