A rare bug in SQLite caused months of disruptions at Tailscale. The network provider experienced nineteen instances of database corruption over the course of half a year. After months of investigation , the cause was found to be a bug that had been present in SQLite since 2010.
The first issues surfaced in August, when a data pipeline detected a corrupted database backup. Database corruption is extremely rare in SQLite under normal circumstances, but the problem kept recurring.
Tailscale has been using SQLite since 2022 to power the databases that run its control plane. These databases contain configuration data about networks and devices, but no private keys or network traffic. When corruption occurred, the affected part of the control plane had to be temporarily shut down. Existing connections continued to function, but new devices, for example, were unable to retrieve network configurations.
Written data disappears
Finding the cause proved difficult. The incidents were not linked to specific servers, customers, load levels, or times, and could not be reproduced. Tailscale therefore enlisted the help of the SQLite developers.
The breakthrough came from a transaction logging system that Tailscale had built to restore databases more quickly. This revealed that data that had been successfully written and committed sometimes suddenly disappeared during subsequent transactions.
Attention shifted to SQLite’s Write-Ahead Log (WAL). In this system, changes are first written to a separate log file and then copied to the actual database during a checkpoint. Tailscale does not let SQLite handle this process automatically; instead, it triggers checkpoints itself to create backups quickly and predictably.
Using new debugging software from the SQLite developers, a race condition was eventually discovered between a write transaction and a checkpoint. When the two coincided in exactly the wrong way, SQLite could mistakenly believe that certain WAL pages had already been copied to the database when they had not. The data was lost, and the database could become corrupted.
The developers refer to this as the WAL-Reset bug. According to them, the bug dates back to SQLite 3.7.0, released in July 2010. The fact that it went unnoticed for sixteen years is due to the exceptional circumstances required to trigger it: WAL mode must be active, multiple database connections must have the same file open, and read and write operations must coincide in exactly the wrong way.
For normal SQLite use, the likelihood of this happening is very low. Tailscale was at greater risk because it handles checkpointing itself and does so relatively aggressively. SQLite, however, advises users to upgrade to a patched version.
Initial fix triggers new alert
The fix was initially included in SQLite 3.52.0. However, after the rollout, Tailscale reported corruption in thirteen databases. This turned out to be a false alarm: another change in SQLite had altered the rounding of certain floating-point conversions, causing existing indexes to be flagged as corrupt. SQLite withdrew version 3.52.0 and released version 3.51.3, which contained only the fix for the WAL reset bug.
Tailscale then added logging to verify whether the race condition actually occurred in production. Two months later, that exact situation was detected, without the database becoming corrupted. After that, the platform ran for another four months without any new database incidents.
According to Tailscale, the issue demonstrates that even mature technology can pose risks when deployed in an unconventional manner. The configuration used was supported by SQLite, but manually and frequent checkpoint execution deviated from standard usage. Tailscale also funded the VFS debugging tool that ultimately helped uncover the race condition.