November 24, 2016 marked a critical day for the Ethereum network as developers discovered a serious consensus bug in the Geth client that caused a network fork. The Ethereum Foundation quickly responded by releasing an emergency update, highlighting the challenges faced by emerging blockchain networks in maintaining consensus across different client implementations.
TL;DR
- Ethereum network experienced consensus bug causing blockchain fork at block #2686351
>Geth client failed to properly revert empty account deletions during out-of-gas exceptions
>165 blocks were mined on the abandoned chain before the issue was resolved
>Geth 1.5.3 released to fix the journaling mechanism and repair the network fork
The Consensus Crisis
On November 24, 2016, at approximately 14:12:07 UTC, the Ethereum network encountered a critical consensus issue. An issue has been identified with Geth’s journaling mechanism, which caused a network fork at block #2686351. This resulted in two separate chains being created – one maintained by the updated Geth client and Parity, and another running on the older, buggy version of Geth.
The core of the problem was technical in nature: Geth was failing to revert empty account deletions when the transaction causing the deletions of empty accounts ended with an out-of-gas exception. This created inconsistent state between different client implementations, breaking the fundamental principle of blockchain consensus.
Technical Impact and Response
The fork created significant operational challenges for the Ethereum ecosystem. Approximately 165 blocks were mined on the abandoned chain before validators recognized the issue and switched to the correct chain. Transactions were broadcast across the network, meaning most transactions likely appeared on both chains, but mining rewards and transaction fees on the abandoned chain were permanently lost.
The Ethereum Foundation acted swiftly, releasing Geth 1.5.3 which fixed the journaling issue and repaired the network fork. The updated client automatically synchronized the blockchain from the point of the fork, even if nodes had already synced past the problematic block. Additionally, an issue was found in Parity client where it incorrectly failed to revert empty account deletions in limited contexts involving out-of-gas calls to precompiled contracts.
Industry-wide Implications
This incident underscored the critical importance of client diversity and robust consensus mechanisms in blockchain networks. The Ethereum Foundation emphasized that exchanges and other high-value users should run multiple clients and automatically halt operations if they go out of sync by more than approximately 10 blocks. This recommendation highlighted the security risks associated with relying on a single client implementation.
Third-party providers like MetaMask, Jaxx, and MyEtherWallet largely handled the updates transparently for their users, but the incident served as a reminder of the technical risks inherent in blockchain participation. Users were advised that participating in emerging blockchain platforms involved significant risks, including unexpected bugs and other technical complications that could result in loss of ether and other consequences.
Why This Matters
The Ethereum consensus bug of November 24, 2016, represents a pivotal moment in blockchain infrastructure development. At a time when Bitcoin was trading at $740.29 and the total cryptocurrency market cap stood at $12.6 billion, this incident demonstrated that even relatively mature blockchain networks faced significant technical challenges that could impact user funds and network integrity.
The rapid response from the Ethereum Foundation demonstrated the maturation of blockchain governance structures, with clear communication channels and established protocols for handling critical vulnerabilities. This incident led to strengthened recommendations for network security, including the implementation of better monitoring systems and more robust consensus mechanisms across different client implementations.
For the broader cryptocurrency ecosystem, this event served as a learning opportunity about the importance of client diversity, the risks of centralized control over network infrastructure, and the need for continuous security auditing in decentralized systems. As Ethereum traded at $9.23 on this day, the network’s ability to quickly resolve such issues helped maintain confidence in the platform’s long-term viability.
Disclaimer: This article is for informational purposes only and does not constitute financial advice. The cryptocurrency market is highly volatile and past performance is not indicative of future results. Always conduct your own research and consult with a qualified financial advisor before making any investment decisions.
165 blocks on the wrong chain before anyone noticed. early eth was held together with duct tape and prayers
165 blocks on the wrong chain. thats roughly 8 minutes of invalid transactions that got reverted. early eth was genuinely scary
165 blocks mined on the wrong chain. thats 165 blocks of transactions that just ceased to exist. anyone relying on confirmations during that window got reorged
reorg_truther_ 165 blocks of transactions ceasing to exist is why chain reorgs scare me more than smart contract bugs. at least with a contract bug you know where the money went
8 minutes of invalid blocks and nobody’s monitoring tools caught it faster. devops was non-existent in 2016 eth
the empty account deletion bug was subtle. out-of-gas exceptions reverting state changes incorrectly is the kind of thing that keeps protocol devs up at night
empty account deletion on out-of-gas is the kind of edge case that only shows up in production. testing cant catch everything and this proved it
state reverts on out-of-gas exceptions is one of those edge cases that seems obvious in hindsight but is nearly impossible to catch in testing
this exact class of bug still shows up in EVM clients. foundry helps catch more now but state transition edge cases are brutal
petra makes a great point. state transition edge cases still cause consensus failures. the besu bug in 2023 was the same class of issue
Henrik S. the empty account deletion edge case on out-of-gas is genuinely terrifying protocol design. one subtle state transition bug and the whole chain forks