Real time collaborative text editing, by Miroslav Hettes, Smarkup
1 like633 views
The document discusses the architecture and implementation of Smarkup, a collaborative text editing tool that combines Conflict-free Replicated Data Types (CRDT) and Operational Transformation (OT) for efficient real-time collaboration. It details how changes are managed, synchronized, and resolved among multiple users to maintain strong consistency, while also addressing challenges like identifier generation and conflict resolution. The paper references previous research and provides a foundation for developing a seamless collaborative editing experience.
![Set (orderer)
• Set contains unique values
• Associa'vity
• Set(1) + Set(3) = Set(3) + Set(1) = Set(1, 3)
• Idempotence
• Set(1) + Set(1) = Set(1)
• adding numbers to set => sorted set
• 1, 5, 6, 8 => [1, 5, 6, 8]
• 8, 6, 5, 1 => [1, 5, 6, 8]
• 8, 1, 6, 5 => [1, 5, 6, 8]](https://image.slidesharecdn.com/smarkupkeynote2018-06-27-real-time-collaborative-text-editingslideshare-180628124040/85/Real-time-collaborative-text-editing-by-Miroslav-Hettes-Smarkup-28-320.jpg)
Real time collaborative text editing, by Miroslav Hettes, Smarkup
- 1. Miroslav He,eš, Smarkup Real 5me collabora5ve text edi5ng
- 2. Smarkup
- 3. Smarkup The ul5mate text editor Let’s collaborate…
- 5. Collabora5on - sending file
- 6. Collabora5on - sending file
- 7. Collabora5on - over file on server
- 8. Collabora5on - over file on server
- 15. Git is file
- 16. But the conflicts
- 17. Defining problem • Real 'me collabora'on • Allow mul'ple users edit the same document at the same 'me • Edi'ng experience - notepad • High responsiveness • Local replica'on • Offline edits • Op'mis'c replica'on • Document can diverge -> auto-resolving conflicts
- 18. Text edi5ng • Text edit • change opera'on on document • basically one keystroke • Document • result of text edits / opera'ons sequences • Basic opera'ons • insert • delete
- 19. Text edi5ng insert 0, “H” insert 1, “e” insert 3, “llo” delete 2 insert 2, “y”
- 20. Collabora5ve text edi5ng • Every user (site) has a local copy (replica) of the document • When user changes document he generates text change • Change is immediately applied to his local replica • And then asynchronously (non-blocking) broadcasted to other users (sites) • When all changes are delivered, all users contains the same document (all replicas are equal)
- 21. Collabora5ve text edi5ng Distributed Collabora/ve Edi/ng Peer 2 Peer
- 25. Opera5on Transforma5on • Every opera/on has a context (state of document when it was generated) • Opera/on needs to be executed in the same context as it was generated • Or it has to be transformed based on changed context • Very complicated and not very intui/ve • Effec/ve for implementa/on • used in Google Docs
- 26. Order ma,ers! insert 2, “e” + insert 5, “!” =/= insert 5, “!” + insert 2, “e” Ok. But maybe we can look at this a liPle differently.
- 27. Perfect world • Opera'ons proper)es • Associa)vity • Opera'ons can be applied in any order • 1 + 5 = 5 + 1 • Idempotence • Opera'on can be applied mul'ple 'mes • 8 * 1 * 1 * 1 = 8
- 28. Set (orderer) • Set contains unique values • Associa'vity • Set(1) + Set(3) = Set(3) + Set(1) = Set(1, 3) • Idempotence • Set(1) + Set(1) = Set(1) • adding numbers to set => sorted set • 1, 5, 6, 8 => [1, 5, 6, 8] • 8, 6, 5, 1 => [1, 5, 6, 8] • 8, 1, 6, 5 => [1, 5, 6, 8]
- 29. Perfect world • If we can model text opera'on like this • Edits would not conflict with each others • We do not need to care about the order or transforming opera'ons • Building final document would be as easy as crea'ng sorted set
- 30. Design structure • Basic idea • Each character with have and iden'fier • All iden'fiers are globally uniq and creates total order • Opera'ons specifica'on • insert ID, CHAR • delete ID • There is special number for beginning and end of document, all characters are in between • Document = sorted characters by iden'fiers without deleted ones
- 32. Insert opera5on • Every 'me we wants to add new character as given posi'on we generate (random) number within interval borders • Interval borders • _begin_, H = (0, 12) • H, e = (12, 102) • e, l = (102, 133) • l, l = (133, 502) • l, o = (502, 810) • o, _end_ = (810, 1 000)
- 33. Insert opera5on • Inser'ng “!” aber “o” = rand(810, 1 000) = 920 • insert 920, “!”
- 34. Delete opera5on • you can delete only by ID • dele'ng the second “L” character • delete 502 • Helo!
- 35. Commuta5vity • All possible combina'ons • insert + insert • delete + delete • insert + delete • delete + insert
- 36. Commuta5vity • insert ID1 + insert ID1 • each 'me we generate new insert, new uniq iden'fier is created (bear with me) • there is a total order from uniq iden'fier = commutes • delete ID1 + delete ID2 • ID1 = ID2 • deleted the same character mul'ple 'mes, does not make it any more deleted • ID1 != ID2 • two different character are deleted, this commutes
- 37. Commuta5vity • insert ID1 + delete ID2 • ID1 = ID2 • character will be deleted • ID1 != ID2 • ok • delete ID1 + insert ID2 • ID1 = ID2 • character will be never showed, character is deleted right aber it arrives • ID1 != ID2 • ok
- 38. Idempotence • insert ID, CHAR • ID is globally uniq, so character will be inserted only once • delete ID • deleted character cannot be deleted more
- 39. Problems • Two clients can generate the same iden'fier for different character - remember it randomly generated • add each client number and use it it posi'on • (client_id, posi'on) • Limited iden'fier space • if two characters use closest iden'fier next to each other, we cannot insert character in between • Memory overhead • Iden'fiers • Deleted characters - tombstones
- 40. Results • We are able to model structure where its opera'on commutes and sa'sfy idempotency • We are able to create structure where are no conflicts • If there would always be space between any two characters, we can create endlessly long document • It is easy to reason about • We are limited number of iden'fiers • Storing iden'fiers and deleted characters creates overhead
- 41. CRDT • Conflict-free replicated data type • Strong Eventual Consistency • Total order, par'al order • Collabora've edi'ng approaches • WOOT • Logoot • LSEQ • RGA • Treedoc
- 42. CRDT / OT • Think in CRDT, implement in OT • Easy to think about, effec've for implementa'on • XI Editor • Each character has a uniq iden'fier • It is its index • Iden'fier are ‘changed’ on insert • Insert is transforma'on map for iden'fiers (indexes) • We do not show deleted character but we keep them • solves tombstone problem in peer 2 peer
- 46. Smarkup What is?
- 48. Confluence Google Docs Evernote One note Notepad Wiki Markdown (Sublime, etc.) Notes … = But simple!
- 49. Collabora5ve edi5ng • Heavily inspired in XI Editor • Combina'on of CRDT with OT • OT for memory efficiency • CRDT proper'es faster merging • Client Server • We keep deleted character (easy undo) • text is separated from tombstones in Smarkup
- 50. Sources (I) • First paper on collabora'on, 1989, Concurrency Control in Groupware Systems hpps://dl.acm.org/cita'on.cfm?id=66963 • Jupiter paper, High-latency, low-bandwidth windowing in the Jupiter collabora'on system hpps://dl.acm.org/cita'on.cfm?id=215706 • Tombstone Transforma'on Func'ons for Ensuring Consistency in Collabora've Edi'ng Systems hpps://ieeexplore.ieee.org/document/4207539/ • Talk, Strong Eventual Consistency and Conflict-free Replicated Data Types. Marc Shapiro - hpps://www.youtube.com/watch?v=oyUHd894w18 • Research paper, Conflict-free Replicated Data Types - hpps://pages.lip6.fr/ Marc.Shapiro/papers/RR-7687.pdf • hpps://en.wikipedia.org/wiki/Conflict-free_replicated_data_type • WOOT paper, Real 'me group editors without Opera'onal transforma'on hpps://hal.inria.fr/inria-00071240/document • Logoot paper, A Scalable Op'mis'c Replica'on Algorithm for Collabora've Edi'ng on P2P Networks hpps://hal.inria.fr/inria-00432368/document
- 51. Sources (II) • LSEQ: an adap've structure for sequences in distributed collabora've edi'ng hpps:// dl.acm.org/cita'on.cfm?id=2494278 • Towards a unified theory of Opera'onal Transforma'on and CRDT hpps:// medium.com/@raphlinus/towards-a-unified-theory-of-opera'onal-transforma'on-and- crdt-70485876f72f • Working code for Opera'onal Transforma'on/CRDT hybrid hpps://medium.com/ @raphlinus/working-code-for-opera'onal-transforma'on-crdt-hybrid-9d04a57309da • A simple approach to building a real-'me collabora've text editor hpp:// digitalfreepen.com/2017/10/06/simple-real-'me-collabora've-text-editor.html • Xi Editor CRDT concept hpps://google.github.io/xi-editor/docs/crdt.html • Xi Editor CRDT details hpps://google.github.io/xi-editor/docs/crdt-details.html • Understanding and Applying Opera'onal Transforma'on hpp://www.codecommit.com/ blog/java/understanding-and-applying-opera'onal-transforma'on • hpp://archagon.net/blog/2018/03/24/data-laced-with-history/ • hpps://mar'n.kleppmann.com/2017/04/24/json-crdt.html
- 52. Smarkup
- 53. Smarkup The ul5mate text editor
- 54. Smarkup Looking for a Front-End Developer smarkup.com [email protected] @smarkup