All You Need to Know About the Recent Network Upgrade
December 29, 2021
Earlier this month, Polygon’s core development team with help from bug bounty platform Immunefi successfully fixed a critical network vulnerability. Considering the nature of this upgrade, it had to be executed without attracting too much attention. We are now ready to give the full account of what happened.
A group of whitehat hackers notified Immunefi, which hosts our bug bounty, of a vulnerability in the Polygon PoS genesis contract on Dec. 3. The Polygon core team engaged with the group and Immunefi’s expert team and immediately introduced a fix. The validator and full node communities were notified, and they rallied behind the core devs to upgrade 80% of the network within 24 hours without stoppage.
The upgrade was executed on Dec. 5 at block #22156660 without impacting liveness and performance of the network in any major way. The vulnerability was fixed and damage was mitigated, with there being no material harm to the protocol and its end-users. All Polygon contracts and node implementations remain fully open source.
“All projects that achieve any measure of success sooner or later find themselves in this situation,” said Polygon’s co-founder Jaynti Kanani. “What’s important is that this was a test of our network’s resilience as well as our ability to act decisively under pressure. Considering how much was at stake, I believe our team has made the best decisions possible given the circumstances.”
There is a natural tension between security and transparency, both of which are cornerstone values at Polygon. Our initial disclosure was minimal because we follow the “silent patches” policy introduced and used by the Geth team. All in all, the core development team struck the best possible balance between openness and doing what is best for the community, partners and the broader ecosystem in handling this extremely urgent and sensitive issue. But you can be the judge of that.
Here is the timeline of the events as they unfolded:
Dec. 3, 10:11 (UTC) The first white hat hacker submits a report of a possible exploit to Immunefi, which hosts Polygon’s $2 million bounty program.
Dec. 3, 16:18 (UTC)Polygon confirms the vulnerability. Within one hour, various options are considered. The decision is made to upgrade the mainnet as soon as possible.
Dec. 3, 20:18 (UTC)The Polygon team provides release Bor v0.2.12-beta1 to validators on Mumbai testnet at Block #22244000.
Dec. 4, 04:26 (UTC)Mumbai update is complete. The Polygon team, white hat and Immunefi validate the fix and prepare for the update of the mainnet.
Dec. 4, 13:46 (UTC)The vulnerability is used to steal MATIC tokens, the first in a series of transfers that removes 801,601 MATIC in total.
Dec. 4, 18:53 (UTC)The second white hat submits a report to Immunefi.
Dec. 4, 21:08 (UTC)The Polygon team informs Validators of an “Emergency Bor Upgrade for Mainnet.”
Dec. 5, 07:27 (UTC)Mainnet update is complete for +90% validators at Block #22156660.
Polygon paid a total of about $3.46 million as bounty to two white hats who helped discover the bug. Despite our best efforts, a malicious hacker was able to use the exploit to steal 801,601 MATIC before the network upgrade took effect. The foundation will bear the cost of the theft.
“The Polygon team’s response to this disclosure was swift and effective,” said Immunefi's Chief Technology Officer Duncan Townsend. “That this incident had a happy ending is a testament to their expertise. Tight coordination with the Polygon validators helped avert what could’ve been a major disaster.”
In the days after the upgrade, Polygon’s core team has carried out an extensive post mortem and identified a number of existing processes that can be improved as well as actions that will make the network and our community more resilient in the future. These measures include the following:
Updating our critical response processes;
Consolidating partner contact info and communications channels;
Identifying and formalizing backups for key internal resources to eliminate single points of failure during time sensitive situations
This experience highlighted the importance of investing into an ecosystem of security expert partners. We are very grateful to Immunefi for all their help. At the end of the day, this brought Polygon a step closer to becoming the most battle-tested scaling solution for Ethereum.
You can read Immunefi’s technical report on the fixhere. A security audit certificate completed by Quantstamp can be viewed here. To keep up with the latest developments, check out our blog.
Nothing says “you have arrived” like people caring about your carbon footprint. Polygon has recently marked that milestone with a commitment to becoming carbon negative by the end of the year. Below is the case for why other projects should follow our lead and how to go about doing it. Ethereum’s smart contract blockchain has […]
What’s the future of DeFi? How can blockchain projects shake the reputation for being bad for the environment? What in the world are zero knowledge proofs? How can Web3 attract and accommodate the next billion users? When Flippening? You want to know the answers and so do we. That’s why Polygon is launching Polygon Pod […]
We’re in the early innings when it comes to enterprise needs for blockchains. Whether it’s supply chain management or marketplaces, the potential for secure business transactions on a public ledger has enormous implications. That’s why we’re pleased to announce Polygon Nightfall, a blockchain solution that leverages cutting edge Optimistic-Zero Knowledge technology to give businesses an […]
At Polygon, we believe that putting people firmly in control of their digital identities is at the core of Web3’s promise to empower users over networks. We have been working behind the scenes to fulfill this promise and today are proud to introduce Polygon ID, the self-sovereign, decentralized and private identity for the next iteration […]
Today marks the start of a new, sustainable era for the Polygon ecosystem -- the core team has committed to eliminating all of the network’s carbon debt going back to inception and making the chain climate positive into the future. Polygon’s Green Manifesto announced today is part of a broader vision for sustainable development that […]
MetaMask is the most popular cryptocurrency wallet app and browser extension used to access the Web3 universe. In just a few simple steps, you can unlock this powerful tool and begin exploring the Polygon ecosystem. There are more than 19,000 decentralized applications (dApps) running on Polygon. That list includes DeFi protocols like Aave and Uniswap […]