All You Need to Know About the Recent Network Upgrade
December 29, 2021
Earlier this month, Polygon’s core development team with help from bug bounty platform Immunefi successfully fixed a critical network vulnerability. Considering the nature of this upgrade, it had to be executed without attracting too much attention. We are now ready to give the full account of what happened.
A group of whitehat hackers notified Immunefi, which hosts our bug bounty, of a vulnerability in the Polygon PoS genesis contract on Dec. 3. The Polygon core team engaged with the group and Immunefi’s expert team and immediately introduced a fix. The validator and full node communities were notified, and they rallied behind the core devs to upgrade 80% of the network within 24 hours without stoppage.
The upgrade was executed on Dec. 5 at block #22156660 without impacting liveness and performance of the network in any major way. The vulnerability was fixed and damage was mitigated, with there being no material harm to the protocol and its end-users. All Polygon contracts and node implementations remain fully open source.
“All projects that achieve any measure of success sooner or later find themselves in this situation,” said Polygon’s co-founder Jaynti Kanani. “What’s important is that this was a test of our network’s resilience as well as our ability to act decisively under pressure. Considering how much was at stake, I believe our team has made the best decisions possible given the circumstances.”
There is a natural tension between security and transparency, both of which are cornerstone values at Polygon. Our initial disclosure was minimal because we follow the “silent patches” policy introduced and used by the Geth team. All in all, the core development team struck the best possible balance between openness and doing what is best for the community, partners and the broader ecosystem in handling this extremely urgent and sensitive issue. But you can be the judge of that.
Here is the timeline of the events as they unfolded:
Dec. 3, 10:11 (UTC) The first white hat hacker submits a report of a possible exploit to Immunefi, which hosts Polygon’s $2 million bounty program.
Dec. 3, 16:18 (UTC)Polygon confirms the vulnerability. Within one hour, various options are considered. The decision is made to upgrade the mainnet as soon as possible.
Dec. 3, 20:18 (UTC)The Polygon team provides release Bor v0.2.12-beta1 to validators on Mumbai testnet at Block #22244000.
Dec. 4, 04:26 (UTC)Mumbai update is complete. The Polygon team, white hat and Immunefi validate the fix and prepare for the update of the mainnet.
Dec. 4, 13:46 (UTC)The vulnerability is used to steal MATIC tokens, the first in a series of transfers that removes 801,601 MATIC in total.
Dec. 4, 18:53 (UTC)The second white hat submits a report to Immunefi.
Dec. 4, 21:08 (UTC)The Polygon team informs Validators of an “Emergency Bor Upgrade for Mainnet.”
Dec. 5, 07:27 (UTC)Mainnet update is complete for +90% validators at Block #22156660.
Polygon paid a total of about $3.46 million as bounty to two white hats who helped discover the bug. Despite our best efforts, a malicious hacker was able to use the exploit to steal 801,601 MATIC before the network upgrade took effect. The foundation will bear the cost of the theft.
“The Polygon team’s response to this disclosure was swift and effective,” said Immunefi's Chief Technology Officer Duncan Townsend. “That this incident had a happy ending is a testament to their expertise. Tight coordination with the Polygon validators helped avert what could’ve been a major disaster.”
In the days after the upgrade, Polygon’s core team has carried out an extensive post mortem and identified a number of existing processes that can be improved as well as actions that will make the network and our community more resilient in the future. These measures include the following:
Updating our critical response processes;
Consolidating partner contact info and communications channels;
Identifying and formalizing backups for key internal resources to eliminate single points of failure during time sensitive situations
This experience highlighted the importance of investing into an ecosystem of security expert partners. We are very grateful to Immunefi for all their help. At the end of the day, this brought Polygon a step closer to becoming the most battle-tested scaling solution for Ethereum.
You can read Immunefi’s technical report on the fixhere. A security audit certificate completed by Quantstamp can be viewed here. To keep up with the latest developments, check out our blog.
Tokens are the élan vital of Web3 networks, and projects are often perceived through the price at which they trade. But token price, subject to market sentiment and speculation, is not synonymous with token value. To understand the fundamentals, we must first unpack the different ways that value accrues to tokens in the new Internet […]
With Ethereum’s Merge finally upon us, we’ve been thinking about how this historic milestone will affect our sustainability efforts. Polygon’s initial commitment to become carbon neutral and eventually carbon negative was largely addressing emissions generated by our relationship with Ethereum. The Merge dramatically changes that equation. Ethereum’s transition from Proof-of-Work to Proof-of-Stake consensus, known as […]
Today, we are thrilled to announce that Starbucks Coffee Company is working with Polygon to provide the blockchain technology to build its recently announced Web3 experience, Starbucks Odyssey. As a result of the partnership, this new Web3-powered experience will allow Starbucks Rewards loyalty program members and Starbucks partners (employees) in the United States to earn […]
The Polygon Avail Testnet is now live. As users begin incorporating Avail into their chain designs, a question that often comes up is, “How many transactions can Avail process?” This is Part Two in a three-part series of articles that will address Polygon Avail’s current performance, as well as its ability to scale in the […]
Ethereum’s much-anticipated transition to the proof of stake (PoS) consensus, known as the Merge, is almost upon us. It’s a momentous event with broad-ranging implications for the greater Ethereum ecosystem. So we wanted to provide a definitive list of frequently asked questions (and answers) as well as address some of the common misconceptions, both of […]
Polygon has teamed up with Alpha Venture DAO and Icetea Labs to launch the inaugural Icetea Labs Accelerator Program for blockchain startups. Working in a tight three-way partnership, Polygon is co-creating an accelerator to funnel resources, mentorship, education, and business acumen to promising Web3 companies from around the world. The startups selected for the accelerator […]
Applications are open for Polygon Bootcamp Africa, launched in partnership with Xend Finance! The eight-week intensive educational course and hackathon combo will put developers in Africa on the Web3 map. This is Polygon’s biggest step in providing resources to developers, builders, and creators in Africa, and it comes during an important time. An internet and […]
As part of our ongoing efforts to inform the Ethereum community about the efforts of Polygon’s zero-knowledge (ZK) teams, we will be posting a series of technical papers by our engineers and researchers. We hope that everyone who’s interested in the inner workings of Polygon’s ZK projects, Ethereum itself, and cryptographic engineering in general will […]