{ "archive_path": "archive/1792836413.044168", "base_url": "philipotoole.com/twelve-years-of-distributed-sqlite", "basename": "", "bookmarked_date": "2026-10-24 10:06", "canonical": { "archive_org_path": "https://web.archive.org/web/philipotoole.com/twelve-years-of-distributed-sqlite", "dom_path": "output.html", "favicon_path": "favicon.ico", "git_path": "git/", "google_favicon_path": "https://www.google.com/s2/favicons?domain=philipotoole.com", "headers_path": "headers.json", "htmltotext_path": "htmltotext.txt", "index_path": "index.html", "media_path": "media/", "mercury_path": "mercury/content.html", "pdf_path": "output.pdf", "readability_path": "readability/content.html", "screenshot_path": "screenshot.png", "singlefile_path": "singlefile.html", "warc_path": "warc/", "wget_path": "philipotoole.com/twelve-years-of-distributed-sqlite/index.html" }, "domain": "philipotoole.com", "extension": "", "hash": "RB4ESXGZYFBTN52P7DDZ", "history": { "archive_org": [], "dom": [], "favicon": [ { "cmd": [ "curl", "--silent", "--location", "--compressed", "--max-time", "120", "--output", "favicon.ico", "--user-agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36 ArchiveBox/0.7.4 (+https://github.com/ArchiveBox/ArchiveBox/) curl/curl 8.14.1 (x86_64-pc-linux-gnu)", "https://www.google.com/s2/favicons?domain=philipotoole.com" ], "cmd_version": "curl 8.14.1 (x86_64-pc-linux-gnu)", "end_ts": "2026-10-24T10:07:32.272328+00:00", "index_texts": null, "output": "favicon.ico", "pwd": "/data/archive/1792836413.044168", "schema": "ArchiveResult", "start_ts": "2026-10-24T10:07:31.791023+00:00", "status": "succeeded" } ], "git": [], "headers": [ { "cmd": [ "curl", "--silent", "--location", "--compressed", "--head", "--max-time", "120", "--user-agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36 ArchiveBox/0.7.4 (+https://github.com/ArchiveBox/ArchiveBox/) curl/curl 8.14.1 (x86_64-pc-linux-gnu)", "https://philipotoole.com/twelve-years-of-distributed-sqlite/" ], "cmd_version": "curl 8.14.1 (x86_64-pc-linux-gnu)", "end_ts": "2026-10-24T10:07:32.568172+00:00", "index_texts": null, "output": "headers.json", "pwd": "/data/archive/1792836413.044168", "schema": "ArchiveResult", "start_ts": "2026-10-24T10:07:32.282719+00:00", "status": "succeeded" } ], "htmltotext": [], "media": [], "mercury": [ { "cmd": [ "/app/node_modules/@postlight/parser/cli.js", "https://philipotoole.com/twelve-years-of-distributed-sqlite/" ], "cmd_version": "1.0.0", "end_ts": "2026-10-24T10:07:39.548526+00:00", "index_texts": null, "output": "mercury", "pwd": "/data/archive/1792836413.044168", "schema": "ArchiveResult", "start_ts": "2026-10-24T10:07:36.571060+00:00", "status": "succeeded" } ], "pdf": [], "readability": [ { "cmd": [ "/app/node_modules/readability-extractor/readability-extractor", "/tmp/tmpu4f0hw7i", "https://philipotoole.com/twelve-years-of-distributed-sqlite/" ], "cmd_version": "0.0.11", "end_ts": "2026-10-24T10:07:36.559188+00:00", "index_texts": [ "In September 2014 I published a short post announcing rqlite. The idea was simple: take SQLite, put a Raft log in front of it, and replicate every write to a cluster so that the loss of a node did not mean the loss of the data. The source went on GitHub that same weekend. As far as I can tell, that was the first open-source system to combine Raft and SQLite. Nobody else was doing it, and for several years nobody else tried.\n\nToday \u201cdistributed SQLite\u201d is a category with a dozen projects in it, a few of them venture-funded, two of them acquired. Since the people now writing about the category tend to start the story around 2021, I thought I should write down the version that starts in 2014. I was there for all of it.\n2014: rqlite\nThe need was concrete. A clustered system I worked on at jut.io needed a small amount of relational storage, and the question of which node should hold it had no good answer. Replicating SQLite with a consensus protocol solved it. We went another way at work, but I kept building rqlite because the problem was general and the solution was small. The first version used goraft, the same library InfluxDB\u00a0and etcd used at the time (I subsequently joined InfluxDB as a core developer around the same time). In early 2016 I replaced it with Hashicorp\u2019s Raft implementation which rqlite still uses.\nThe design decision that mattered most was made then and never changed. rqlite is a CP system. Writes must go through the leader, a majority must acknowledge, and a partitioned minority refuses writes. Everything else, the HTTP API, the read-consistency levels, the snapshotting, followed from that.\nBefore 2014, and alongside it\nTwo things deserve mention for honesty\u2019s sake. Expensify had been replicating SQLite inside its own infrastructure for years with a Paxos-style protocol; they released it as Bedrock in 2016. And ActorDB, an Erlang system that gave each \u201cactor\u201d its own SQLite database replicated by Raft, appeared around the same time as rqlite. Neither was widely adopted outside its origin, but both show the idea was in the air. What rqlite did was make it a standalone, general-purpose, open-source thing anyone could run.\n2017: dqlite\nCanonical built dqlite to get LXD and later MicroK8s an embedded replicated store without running etcd. Mark Shuttleworth actually called me once, and we talked, hoping that rqlite could be made to work, but it never quite fit. dqlite began in Go and was rewritten in C with its own Raft library. It is the project most similar to rqlite in intent, and the main difference is of one of packaging: dqlite is a library you embed, rqlite is a server you run. Canonical\u2019s need was a database inside their products: mine was a database beside anyone\u2019s product.\n2021: Litestream, and a change of approach\nBen Johnson (whom I worked with at InfluxDB) released Litestream in early 2021. Litestream does not use consensus at all. It watches the SQLite WAL and streams it to S3, giving you disaster recovery for a single-node SQLite database at almost no cost. Ben\u2019s argument was that most applications do not need a cluster; they use one machine and need a way to recover if it dies. He was right about a great many applications, and the \u201cSQLite is enough\u201d movement of the next few years, with blog posts and conference talks, owes a lot to that argument.\nLitestream and rqlite answer different questions. Litestream: what if the machine dies? rqlite: what if the machine dies and I cannot wait to restore? The first question is more common, the second is the one people have when they come to rqlite.\nFly.io hired Ben in 2022 and shipped LiteFS, which replicates SQLite at filesystem level with leases from Consul. It was the first attempt to make replicated SQLite a platform feature rather than a thing you deploy. And in a final twist, Litestream inspired rqlite\u2019s v8.0 approach to handling the SQLite WAL.\n2022: forks and extensions\nTwo projects in 2022 took approaches that neither rqlite nor Litestream had. cr-sqlite added CRDTs to SQLite as an extension, so that multiple writers could diverge and merge without a leader. That is the AP corner of the design space, and it suits local-first applications where rqlite\u2019s CP guarantees would be a hindrance.\nlibSQL forked SQLite itself. ChiselStrike, later Turso, wanted to change things inside the engine that the SQLite project, which does not accept outside contributions, and would not change. On top of the fork they built a hosted service with replicas at the edge. Turso raised venture money on that, and in 2026 Supabase acquired them.\nAround the same time, SQLite Cloud appeared, offering hosted SQLite replicated with Raft. It is the project closest to rqlite\u2019s original design, wrapped in a cloud service.\n2022 onward: the platforms\nCloudflare announced D1 in 2022, SQLite as a serverless database inside their edge network, and later put SQLite storage inside Durable Objects. This is replicated SQLite as a managed product from a company that does not need to tell you how it works. Turso\u2019s service is the same shape. The category had gone from \u201ca thing you run\u201d to \u201ca thing you buy.\u201d\nWhat the twelve years show\nThree things, I think.\nThe core was right. SQLite as a storage engine, with replication added on the outside, has been reinvented by at least eight teams with different needs and different protocols. Nobody has found a reason to stop doing it.\nThe design space has three corners, and they appeared in order. Consensus first (rqlite, dqlite): one leader, strong consistency, a cluster. Streaming second (Litestream): one writer, cheap recovery, no cluster. Merge third (cr-sqlite, and parts of the local-first movement): many writers, eventual consistency. Each is correct for a different question, and most of the arguments are arguments between their users about which question is being asked.\nAnd the money went to the platforms. rqlite, dqlite, and Litestream are open source and free. The projects that raised the capital or got acquired are the ones that turned replicated SQLite into a service with a bill. That is the usual fate of infrastructure ideas, and I do not mind it. But it is worth recording that the idea was open source for eight years before anyone sold it.\nrqlite is still here. It does the same thing it did in 2014, a good deal better, and it is still the thing you reach for when you want a fault-tolerant relational database you can run yourself, understand fully, and not pay anyone for. Twelve years in, I am content with that position in the story." ], "output": "readability", "pwd": "/data/archive/1792836413.044168", "schema": "ArchiveResult", "start_ts": "2026-10-24T10:07:34.766858+00:00", "status": "succeeded" } ], "screenshot": [], "singlefile": [], "title": [ { "cmd": [ "curl", "--silent", "--location", "--compressed", "--max-time", "120", "--user-agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36 ArchiveBox/0.7.4 (+https://github.com/ArchiveBox/ArchiveBox/) curl/curl 8.14.1 (x86_64-pc-linux-gnu)", "https://philipotoole.com/twelve-years-of-distributed-sqlite/" ], "cmd_version": "curl 8.14.1 (x86_64-pc-linux-gnu)", "end_ts": "2026-10-24T10:07:34.758026+00:00", "index_texts": null, "output": "Twelve years of distributed SQLite \u2013 Philip O'Toole", "pwd": "/data/archive/1792836413.044168", "schema": "ArchiveResult", "start_ts": "2026-10-24T10:07:34.726787+00:00", "status": "succeeded" } ], "wget": [ { "cmd": [ "wget", "--no-verbose", "--adjust-extension", "--convert-links", "--force-directories", "--span-hosts", "--no-parent", "-e", "robots=off", "--reject=js,mjs,woff,woff2,ttf,otf,eot,mp4,m4v,mov,webm,ogv,mkv,avi,flv", "--timeout=120", "--restrict-file-names=windows", "--page-requisites", "--user-agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36 ArchiveBox/0.7.4 (+https://github.com/ArchiveBox/ArchiveBox/) wget/GNU Wget 1.21.3", "--compression=auto", "--timestamping", "https://philipotoole.com/twelve-years-of-distributed-sqlite/" ], "cmd_version": "GNU Wget 1.21.3", "end_ts": "2026-10-24T10:07:34.718328+00:00", "index_texts": null, "output": "philipotoole.com/twelve-years-of-distributed-sqlite/index.html", "pwd": "/data/archive/1792836413.044168", "schema": "ArchiveResult", "start_ts": "2026-10-24T10:07:32.577994+00:00", "status": "succeeded" } ] }, "icons": null, "is_archived": true, "is_static": false, "latest": { "archive_org": null, "dom": null, "favicon": "favicon.ico", "git": null, "media": null, "pdf": null, "screenshot": null, "singlefile": null, "title": "Twelve years of distributed SQLite \u2013 Philip O'Toole", "warc": null, "wget": "philipotoole.com/twelve-years-of-distributed-sqlite/index.html" }, "link_dir": "/data/archive/1792836413.044168", "newest_archive_date": "2026-10-24T10:07:36.571060+00:00", "num_failures": 0, "num_outputs": 6, "oldest_archive_date": "2026-10-24T10:07:31.791023+00:00", "path": "/twelve-years-of-distributed-sqlite/", "schema": "Link", "scheme": "https", "snapshot_id": "706ed32a-b9f5-4e74-ac35-5575b15012a0", "sources": [ "/data/sources/1792836413-import.txt" ], "tags": null, "tags_str": "", "timestamp": "1792836413.044168", "title": "Twelve years of distributed SQLite \u2013 Philip O'Toole", "updated": "2026-10-24T10:07:31.789099+00:00", "updated_date": "2026-10-24 10:07", "url": "https://philipotoole.com/twelve-years-of-distributed-sqlite/" }