Solana Went Down Again—It Likely Won’t Be the Last Time

Solana just had another major outage—and that’s OK, the developers behind the network say. It’s likely going to happen again.

That’s what happens when you’re building on the bleeding edge of tech, and at least one prominent builder on the network—Helius Labs co-founder and CEO Mert Mumtaz—says he’s prepared to take the heat that comes with it.

“What we can’t do is be afraid of progress for the sake of not being harassed on Twitter,” Mumtaz told Decrypt.

Solana nearly made it a full year without a major outage—a milestone that has proven evasive amid the network’s rise, and one that will remain out of reach for at least another 12 months following Tuesday’s morning’s stretch of downtime.

The network went offline for nearly five hours Tuesday, according to data from the official Solana Status tracker. The previous outage, on February 25, 2023, lasted for nearly 19 hours and was later blamed on “custom block-forwarded software [that] inadvertently transmitted a huge amount of data,” according to the Solana Foundation.

Tuesday’s outage doesn’t have an official cause yet, but a tweet from Matthew Sigel—VanEck’s Head of Digital Assets Research—that’s been circulated by Solana leaders and developers blames a bug that had been fixed on Solana’s testnet, but not yet deployed to the mainnet.

“This bug was noticed on testnet and a fix was created, but not yet implemented due to testing,” Sigel wrote.

It’s not just a piece of a working explanation behind Solana’s latest network outage, but also a hint at a development ethos that prioritizes network safety and security over perfect uptime. It’s a comment that has been shared in the past by Solana co-founder Anatoly Yakovenko, Mumtaz, and other notable Solana builders.

While rare and relatively short-lived in the grand scheme of things, Solana’s outages have become a key part of the narrative around the blockchain. It’s also an easy target for critics who claim that the chain can’t compete with Ethereum or other notable rivals, despite Solana’s growing use by traders and developers alike thanks to cheap, fast transactions.

Outages are disappointing, no doubt, and Yakovenko has spoken in the past about finding ways to overcome them for good. Even so, the Solana co-founder and others in the space have been candid about saying that outages may still happen even amid the renewed focus on stability, thanks to that ethos of focusing on long-term security.

“If we are truly in the early innings of crypto (and I believe we are), we need to improve these systems as fast as possible so we are ready when the time comes,” said Mumtaz, who’s built up a reputation for being quick to defend Solana from critics and “FUD.”

“Short-term it’s risky and painful, but long-term things like this make the systems more resilient and anti-fragile. Same as any evolutionary system,” he said.

One prominent solution that Solana backers keep floating is the upcoming launch of Firedancer, a second validator client for the network that would exist separately from the current one that Solana Labs built. Announced in August 2022, Firedancer is expected to improve Solana performance and boost decentralization, all while providing a fallback for the network if the Labs validator hits an issue. 

It won’t be a matter of flipping a switch on Firedancer and expecting Solana to be failure-free going forward, however. Yakovenko tweeted in December that users running Solana validators are likely to operate both clients, and would act to stop the network if inconsistencies emerge.

“I actually think that Solana validator culture is to pick safety over liveness, and people are much more likely to run a setup where if Firedancer and Labs disagree, halt both,” he wrote in reply to a thread by Dragonfly Capital Managing Partner Haseeb Qureshi. “Outages suck, but state corruption is catastrophic.”

Decrypt reached out to Solana to seek comment from Yakovenko about Tuesday’s outage and the impending Firedancer launch, but did not immediately receive a response.

Firedancer could have its own hitches during rollout, too, and it may take time to work through them. But Mumtaz said he believes that the second validator client will eventually help boost the reliability of the Solana network as it continues to expand.

“Once Firedancer goes live, there will likely be some unforeseen issues in mainnet. That’s just how software goes; humans make mistakes,” he told Decrypt. “But after a few iterations, we will likely set up automatic failovers, and the probability of failures will go down drastically over time.”

Edited by Guillermo Jimenez