Solana outage was caused by infinite loop bug previously seen on devnet

Solana Labs said that the recent outage that halted blockchain transactions for five hours on Feb. 6 was caused by a bug developers caught a week earlier.

Solana Labs published a detailed report on Feb. 9 outlining its investigation of the outage, which revealed that the known bug had multiple triggers, and developers had only deployed a fix for one of them when the outage occurred.

Root cause

Though earlier reports partially identified the issue, the latest report provides a technical reason for the outage and states that a “deploy-evict-request cycle of a legacy loader program triggered an infinite recompile loop in the JIT cache.”

Simply put, Solana’s transaction process entered an infinite loop due to a bug.

The relevant bug only affected version 1.17 of Solana’s validator client and was not present in version 1.16 of the client. However, almost all validators stalled because 95% of the cluster stake was running 1.17 at the time of the affected block.

The latest report also reveals that developers had identified the bug as the cause of an outage on Solana’s devnet one week earlier. Developers had already patched one possible trigger for the bug when the outage occurred and subsequently sped the release of a patch for the second trigger.

The fix is in version 1.17.20 of the client; developers intend to release a more complete fix in the future, according to the report.

Solana has experienced numerous outages

Solana has regularly attracted controversy for its outages, which occurred frequently in its earlier days.

According to data from CryptoManiaks, Solana has experienced nine outages since September 2021, with over 150 hours of downtime — excluding the latest outage this week.

Until the latest incident, the network did not have an outage for almost one year. Its previous outage occurred in February 2023 and lasted 18 hours.

Solana (SOL) prices have more than recovered since the most recent Feb. 6 incident, which caused SOL to hit a six-day low of $93.75. The token was trading at $105.46 as of press time, up 2.4% over the past 24 hours.

Editor’s Note: Although Solana Labs confirmed the details of the outage on its own website, the report was originally authored and published by Anza co-founder Jeff Washington — a former engineer at Solana Labs.