Ethereum Recovers from Sharp Decline in Validator Participation due to Prysm Consensus Client Bug

After undergoing the Fusaka network upgrade, the Ethereum network was hit by a sharp decline in validator participation. This happened due to a glitch in the Prysm consensus client that caused some votes to go offline. As per the announcement made on Thursday by Prysm, the client’s version v7.0.0 unnecessarily generated old states while processing outdated attestations. This, according to Terence Tsao, a core developer of Prysm, rendered the nodes non-operational. A temporary workaround suggested by the developers was to launch the client with the –disable-last-epoch-targets flag. Deeper into the chain reaction spurred by the bug, we look at the network data provided by Beaconcha.in. They reported that at epoch 411,448, the network managed to secure only 75% sync participation. This was taken from the percentage of 512 randomly selected nodes signing chain heads. Furthermore, the voting participation was recorded at 74.7%. A 25% decrease in voting participation brings the network precariously close to losing the two-thirds supermajority. This kind of a majority is essential for maintaining finality and regular operation of the network. At the time this problem was investigated, Ethereum network’s current epoch was 411,712. Now, the network showed a voting participation of almost 99% and had regained 97% sync participation. This indicated that it had recovered from the incident. Before this incident occurred, epochs commonly saw above 99% of voting participation. The decrease observed in vote participation corresponds to the approximate percentage of validators that were using the Prysm consensus client. This was estimated to be 22.71% on the day before the first signs of the problem emerged. But after the incident happened, this figure fell to 18%. These numbers suggest that the attestation failure was mainly among Prysm validators. The Ethereum Foundation and Prysm developer organization Offchain Labs had not responded to requests for comment at the time of publication. But it’s important to note here, if voting participation fell beneath two-thirds of the total staked Ether, the Ethereum network would lose its finality. Despite this, under Ethereum’s design, blocks could still be produced, however, the chain will not be considered as finalized. Likely consequences of such an occurrence would be freezing of layer-2 bridges, rollups pausing withdrawals, and exchanges increasing their block confirmation requirements due to an expansion of the risk of chain reorganization. Theoretical considerations aside, the possibility of an incident that could end in Ethereum losing finality was experienced in tangible terms earlier. In May 2023, Ethereum’s mainnet lost finality twice in a span of 24 hours due to bugs in the handling of old-target attestations in the Prysm and Teku consensus clients. A point to highlight here is that these incidents could have led to far worse outcomes. Back in September 2021, Prysm was estimated to run on more than two-thirds of the consensus nodes. This majority situation, however, was reported to have changed by January 2022, when new data showed that Prysm was only running on 68.1% of nodes. The diversity among Ethereum consensus clients has made some progress since 2022, but it is still a considerable stretch from achieving a client count under 33%. This is the limit that should ensure that a bug in a single client will not be enough to prevent network finality. As per the latest MigaLabs information, Lighthouse alone handles 52.55% of consensus nodes, with Prysm following at 18%. Certainly, this is a deterioration from the situation prior to the incident, when Lighthouse held less than 48.5% and Prysm had about 22.71%. It is indicative of how such bugs can influence the network’s stability. An Ethereum educator called attention to the fact that had Lighthouse been the one to harbor the bug, then the network would have indeed lost finalization. In conclusion, incidents like these highlight the ongoing challenges faced by the Ethereum network and the efforts to solve them. The glitches in these core clients show that the network’s ability to maintain functionality relies heavily on the performance of these consensus clients. There is a crucial need for further diversification to ensure the network’s resilience and capacity to overcome such problems in the future.

Ethereum Recovers from Sharp Decline in Validator Participation due to Prysm Consensus Client Bug

Latest posts by James Carter

Latest posts from the category News