Resource Orchestration and Fault-Tolerant State Synchronization in Distributed and Edge Operating Systems
Amul Info ·
Failure models, consensus, CRDTs, scheduling and edge platforms such as KubeEdge, OpenYurt and K3s: how distributed and edge systems place work and keep state consistent when links fail.
An overview of how distributed and edge operating systems place work on scarce, scattered hardware and keep replicated state consistent when links fail: the theory (failure models, impossibility results, clocks, consensus, CRDTs, scheduling and fairness), the systems that put it into practice (Kubernetes, Borg, KubeEdge, OpenYurt, K3s, Spanner, Dynamo and others), the challenges specific to the edge, and the open research problems.
In practice today, the "distributed OS" role is mostly played by cluster managers plus a distributed key-value/coordination store : Kubernetes + etcd, Borg + Chubby, Mesos + ZooKeeper, Nomad + Raft/Consul. Many authors therefore call Kubernetes "the datacenter OS" (Mesosphere's "Datacenter Operating System", DC/OS, used exactly that framing).
Consequence: an edge OS needs lightweight orchestration (small control plane footprint), autonomy (edge keeps working when disconnected from the cloud), eventual or causal state sync with the cloud and other edges, latency- and locality-aware placement, secure isolation (microVMs, WASM, containers w/ hardening, TEEs).
Examples: G-Counter, PN-Counter, G-Set, 2P-Set, OR-Set (observed-remove), LWW-Register, MV-Register, RGA/Logoot/LSEQ/Treedoc for sequences.Collaborative editing: Automerge (Kleppmann & Beresford, 2017: "A Conflict-Free Replicated JSON Datatype"), Yjs, Diamond Types.Systems: Riak DT, Redis Enterprise CRDB, Akka Distributed Data, AntidoteDB, Azure Cosmos DB multi-master (conflict policies), Anna KVS (Wu et al., ICDE 2018/VLDB 2019: lattice-composed, coordination-free, autoscaling).Limits: not every invariant is expressible (e.g., bank balance ≥ 0 needs coordination). See "Coordination avoidance" (Bailis et al., VLDB 2014) and I-confluence. Metadata growth (tombstones, vector clocks) needs garbage collection — important on constrained edge nodes.
Key idea: reconciliation loops (desired state vs observed state), the Kubernetes controller pattern, are naturally idempotent and tolerant of lost messages.
The ideas behind today’s orchestrators are old; these systems explored them first.
Bibliographic details are given as recorded in the source notes; check them against the original publications before citing.