Solana to Prevent Future Network Shutdown If Runtime Bug Arises Again: Details

The Solana network experienced its latest outage on June 1, when the Solana Mainnet Beta cluster stopped creating blocks due to stalled consensus. This was created by a run-time bug in the durable nonce transactions feature, which governs how the blockchain handles a specific form of the transaction intended for offline use.

06-01-22 Solana Mainnet Beta Outage Report: The durable nonce transaction feature was disabled in releases v1.9.28/v1.10.23 to prevent the network from halting if the same situation were to arise again. https://t.co/2HwkSWkLaR

— Wu Blockchain (@WuBlockchain) June 5, 2022

After disabling these “durable nonce transactions,” validators began restarting the network four and a half hours later. The following day, at 9:00 p.m. UTC, block production restarted, and network operators continued to restore client services over the next several hours.

The recent outage suggested a failure in Solana’s ability to handle durable nonces. The network’s validators double counted these niche inbounds as a single transaction at two separate block heights, rather than considering them as a single transaction. Solana’s consensus mechanism was effectively broken by this impossible predicament.

Solana takes proactive step

To increase throughput, Solana uses parallel processing of nonoverlapping transactions. An incrementing nonce can be used by networks that execute transactions serially; Solana employs a different mechanism to ensure transactions are not handled twice. Because durable nonce transactions are not meant to expire, they require a different approach to avoid double processing and are handled sequentially.

The processing of a durable nonce transaction revealed a bug in the runtime that stopped the network from progressing throughout the outage. A successful transaction would not have caused this problem because the durable nonce transaction would have failed.

The durable nonce transaction functionality was deactivated in releases v1.9.28/v1.10.23 to prevent the network from stalling if the same circumstance arose again. Durable nonce transactions will not execute until the mitigation has been deployed and the feature reactivated in a future version.