50% of Cardano nodes hit with disconnection bug, Input Output investigates

Input Output (IO) are investigating the cause of an apparent bug that disconnected Cardano nodes — triggering a brief outage.

50% of the network nodes disconnected, then restarted for unknown reasons on Jan. 22, IO reported. However, the issue did not affect edge nodes, which act as gateways for end users to reach “worker nodes.”

“This impacted relay nodes and block producing nodes – edge nodes appear to have been unaffected.”

Cardano node disruption was “a transient anomaly”

IO played down the seriousness of the disruption, stating that the incident was “a transient anomaly” affecting some nodes. Additionally, the developers said such eventualities “were considered in the design,” meaning no surprises or severe disruption resulted.

“This appears to have been triggered by a transient anomaly causing one of two reactions in the node; some disconnected from a peer, others threw an exception and restarted.”

Block production was briefly impacted, temporarily causing the network to fall out of sync, but such instances can happen during normal operations — such as when transitioning between epochs — said IO.

Rick McCracken, lead SPO at the Digital Fortress stake pool, said the anomaly did not knock out the network. Still, there was a brief window “of degradation,” with the network resuming normal operations after the affected nodes restarted.

“There was a brief period of degradation. Most nodes impacted had gracefully recovered. No network restart was required.“

Input Output to further investigate

Comments posted by SPOs in a GitHub post pinpointed an issue with relay and producer nodes, but not all posters experienced that issue — the issue passed within five minutes without any action required on the part of SPOs.

Despite the anomaly’s low impact, IO said it would further investigate the incident by looking for the root cause. In the meantime, it will continue to track network performance.

“We’re now investigating the root cause for this anomalous behavior and implementing further logging measures alongside our regular monitoring procedures.”