All You Need to Know About the Recent Network Upgrade
December 29, 2021
Earlier this month, Polygon’s core development team with help from bug bounty platform Immunefi successfully fixed a critical network vulnerability. Considering the nature of this upgrade, it had to be executed without attracting too much attention. We are now ready to give the full account of what happened.
A group of whitehat hackers notified Immunefi, which hosts our bug bounty, of a vulnerability in the Polygon PoS genesis contract on Dec. 3. The Polygon core team engaged with the group and Immunefi’s expert team and immediately introduced a fix. The validator and full node communities were notified, and they rallied behind the core devs to upgrade 80% of the network within 24 hours without stoppage.
The upgrade was executed on Dec. 5 at block #22156660 without impacting liveness and performance of the network in any major way. The vulnerability was fixed and damage was mitigated, with there being no material harm to the protocol and its end-users. All Polygon contracts and node implementations remain fully open source.
“All projects that achieve any measure of success sooner or later find themselves in this situation,” said Polygon’s co-founder Jaynti Kanani. “What’s important is that this was a test of our network’s resilience as well as our ability to act decisively under pressure. Considering how much was at stake, I believe our team has made the best decisions possible given the circumstances.”
There is a natural tension between security and transparency, both of which are cornerstone values at Polygon. Our initial disclosure was minimal because we follow the “silent patches” policy introduced and used by the Geth team. All in all, the core development team struck the best possible balance between openness and doing what is best for the community, partners and the broader ecosystem in handling this extremely urgent and sensitive issue. But you can be the judge of that.
Here is the timeline of the events as they unfolded:
Dec. 3, 10:11 (UTC) The first white hat hacker submits a report of a possible exploit to Immunefi, which hosts Polygon’s $2 million bounty program.
Dec. 3, 16:18 (UTC)Polygon confirms the vulnerability. Within one hour, various options are considered. The decision is made to upgrade the mainnet as soon as possible.
Dec. 3, 20:18 (UTC)The Polygon team provides release Bor v0.2.12-beta1 to validators on Mumbai testnet at Block #22244000.
Dec. 4, 04:26 (UTC)Mumbai update is complete. The Polygon team, white hat and Immunefi validate the fix and prepare for the update of the mainnet.
Dec. 4, 13:46 (UTC)The vulnerability is used to steal MATIC tokens, the first in a series of transfers that removes 801,601 MATIC in total.
Dec. 4, 18:53 (UTC)The second white hat submits a report to Immunefi.
Dec. 4, 21:08 (UTC)The Polygon team informs Validators of an “Emergency Bor Upgrade for Mainnet.”
Dec. 5, 07:27 (UTC)Mainnet update is complete for +90% validators at Block #22156660.
Polygon paid a total of about $3.46 million as bounty to two white hats who helped discover the bug. Despite our best efforts, a malicious hacker was able to use the exploit to steal 801,601 MATIC before the network upgrade took effect. The foundation will bear the cost of the theft.
“The Polygon team’s response to this disclosure was swift and effective,” said Immunefi's Chief Technology Officer Duncan Townsend. “That this incident had a happy ending is a testament to their expertise. Tight coordination with the Polygon validators helped avert what could’ve been a major disaster.”
In the days after the upgrade, Polygon’s core team has carried out an extensive post mortem and identified a number of existing processes that can be improved as well as actions that will make the network and our community more resilient in the future. These measures include the following:
Updating our critical response processes;
Consolidating partner contact info and communications channels;
Identifying and formalizing backups for key internal resources to eliminate single points of failure during time sensitive situations
This experience highlighted the importance of investing into an ecosystem of security expert partners. We are very grateful to Immunefi for all their help. At the end of the day, this brought Polygon a step closer to becoming the most battle-tested scaling solution for Ethereum.
You can read Immunefi’s technical report on the fixhere. A security audit certificate completed by Quantstamp can be viewed here. To keep up with the latest developments, check out our blog.
Polygon Zero’s mission is simple: to use zero-knowledge proofs to scale Ethereum to a billion users, without compromising decentralization or security. Achieving this requires fast and efficient proof systems. Today, we’re excited to share Plonky2, a major milestone for zero-knowledge cryptography. Plonky2 is a recursive SNARK that is 100x faster than existing alternatives and natively […]
TL;DR: We are excited to announce that Mir, a startup building groundbreaking ZK tech, is joining the Polygon family in a $400M* deal. Mir gathered a team of talented cryptographers and engineers, and after almost two years of work developed the world’s fastest ZK scaling technology. The Mir team is now joining Polygon in order […]
TL;DR: We are excited to announce the newest member of the Polygon solution suite - Polygon Miden, an upcoming STARK-based, EVM-compatible rollup. The project is led by Bobbin Threadbare, former Facebook’s core ZK researcher who led the development of Winterfell. Today we are releasing Polygon Miden’s core component - Miden VM, the first open-source STARK-based […]
When it comes to deploying on Ethereum, there is only one place that lets developers easily migrate their projects, offers a broad range of scaling options and has some of the industry’s lowest transaction rates. These are just some of the reasons why the number of teams using Polygon jumped 100-fold in the past year. […]
Welcome to 2022! This is shaping up to be another amazing year for the Polygon ecosystem. But after an absolutely breakthrough 2021, it's got big shoes to fill. Over the past year, Polygon made big strides toward accomplishing its mission of bringing the next billion users to Ethereum. The network’s low-fee infrastructure attracted some of […]
Earlier this month, Polygon’s core development team with help from bug bounty platform Immunefi successfully fixed a critical network vulnerability. Considering the nature of this upgrade, it had to be executed without attracting too much attention. We are now ready to give the full account of what happened. A group of whitehat hackers notified Immunefi, […]