Preventing Crypto Armageddon: A retrospective on Immunefi, 3 years later

Reviewing 3 years of efforts to make the onchain economy a safer place; did Immunefi succeed in what it set out to do?

Before we begin, do me a favor and hit the Subscribe button. Subscripting shows me you value this work and motivates me write more. Thanks for your help!

---

Preamble

Over the last few years, I put aside writing to protect the onchain economy by creating Immunefi. This post is an honest review of that mission to date, and the results we’ve achieved–$25 billion in hack damage averted at minimum. In drafting this post, I asked myself what next contribution I should make. My conclusion: my insights and experiences at the cutting edge of onchain security have proven valuable in helping my colleagues and founder peers avoid the same mistakes.

So, I’ve decided to share my learnings with the broader community, starting with this review of Immunefi and the Scaling Bug Bounty standard. I hope it proves helpful to you.

Post TLDR:

1. Immunefi was founded to prevent an onchain financial armageddon, which hack records showed was incoming circa 2021.

2. Immunefi succeeded in this effort, preventing over 500 critical security incidents and preventing at least $25 billion USD from being stolen.

3. Driving adoption of bug bounties in crypto was very challenging, and is not yet complete to this day. See end of post for lessons on how to drive future market standards.

---

In 2020, I watched the initial wave of smart contract hacks during DeFi summer. 

Having spent that year analyzing the then state-of-the-art defenses to match against my threat forecasts, a menacing future appeared. If the blockchain security stack did not quickly mature, then the blockchain economy would face a financial armageddon of hacking crime that risked de-legitimatizing the entire industry. 

After reflecting on how to prevent this impending catastrophe and exploring the space, I concluded that the most effective tool would be the mass adoption of bug bounty programs. 

In 2020, bug bounty programs were rarely adopted and deeply unappreciated by the blocksec community. I, however, believed they would allow and incentivize the entire global blockchain security community to shore up the defenses of key industry infrastructure through decentralized, permissionless code review. Even better, such programs would become increasingly effective as the blocksec community expanded, while also acting as a powerful reflexive steroid for security community growth in the form of financial and career incentives. 

At the time, I estimated there were less than a thousand serious security professionals responsible for all key industry security duties diluted across projects, from propping up the industry’s most important infrastructure to auditing, to hunting for bugs in mission-critical code. 

To say blocksec was understaffed would have been a dramatic understatement.

Without dramatic growth in the security talent pool, we wouldn’t be able to do all the internal security work, audits, responsible disclosures, security tool and product creation required just to survive. 

Yes, it had become clear: only mass adoption of bug bounty programs could save crypto from a dark forest armageddon. 

That’s why I founded Immunefi. 

Solving the hack problem

There were two huge problems Immunefi had to solve to achieve this mission and make the industry safe: 

1. Drive bug bounty adoption so that incentives and power dynamics would disfavor criminal hacking activity

2. Grow the aggregate blocksec community in the face of countless alternative temptations

In the web2 space, tech companies would customarily pay out $25-50k for a big bug bounty. Even great Web2 hackers might get just six figures for a truly critical zero-day sold through a zero-day broker (often with intelligence agencies as the end purchaser), with only the most exceptional zero-days going for seven figures. The world’s largest tech companies (Apple, Google, Microsoft) have just begun to offer 7 figure bug bounties, though I have no evidence that they’ve made any such payments yet.

But in the web3 space, the unique attributes of the onchain economy make monetizing onchain vulnerabilities so much easier. A blackhat could use their own zero-day vulnerability to make many lifetimes' worth of riches in a single heist. The only whitehat alternative at the time was to contact the project, explain the problem, and usually get ridiculed, ignored, or paid a laughable pittance. No sir, this dynamic would not be viable.

These were big problems, and barring a powerful new catalyst, we were going to be very, very screwed.

To break through this depressing status quo, we needed a conceptual battering ram. We needed an utterly novel, yet easy-to-adopt solution that would make protecting the onchain world both meaningful and financially worthwhile for security professionals worldwide.

After much reflection, I created a simple thesis that: 1) bug bounties are the best way to protect onchain assets, and 2) the size of bounties should grow together with the amount of capital at risk (you can read the original thesis here), subject to practical financial limits. For example, if you have $10 million at risk from a particular bug, the bug bounty payout should be up to $1 million (a reward of 10% of the impacted funds), caveated by program-specific rules and a cap corresponding to cash and tokens earmarked for the bounty. 

A hacker could still earn more by executing the exploit, but then they would be breaking the law and have to spend the rest of their lives looking over their shoulder (see Ilya Lichtenstein and Razzlekhan as a good example). I reasoned that if a hacker could get life-changing money through either path, the benefit of not having to forever watch their back would be preferable to most hackers, even if the financial return was not as high.

I called this new thesis the scaling bug bounty standard.

Since then, scaling bug bounties (most especially on Immunefi) have become an industry standard and prevented immense harm. But did the scaling bug bounty standard really make the blockchain economy a safer place? And was the immense harm prevented worth the dramatic 10-100x increase of bug bounty sizes that the scaling bug bounty standard drove?

In this retrospective, we’ll examine the numbers and evaluate the real impact of the thesis, and whether or not it prevented the ‘financial armageddon’ I foresaw.

—

Reviewing the damage

We can assess the impact of Immunefi and the scaling bug bounty standard by looking at the frequency and impact of hacks over the years. If hack incidence or aggregate impact is going down over time, then we can be confident that overall security in the industry is improving, which Immunefi and the scaling bug bounty may be responsible for.

Reduction in hack incidence or hack impact is not sufficient for us to conclude that Immunefi itself is responsible for that outcome. However, increasing ecosystem security (which reduced hack incidence and impact both indicate) is required to prove Immunefi a possible cause.

Fortunately, Immunefi has been monitoring hack incidence and impact for the lifetime of its operation, so we have hard data (though it is probably not inclusive of all incidents). 

The data tell us there were 107 hacks in 2021, 134 hacks in 2022, and 247 hacks in 2023, for a total of 488 publicly known hacks from 2021-2023.

These hacks impacted $2,334,863,067 USD in 2021, $3,773,906,837 USD in 2022, and $1,699,632,321 USD in 2023, for a total of $7,808,402,225 in funds impacted from 2021-2023. For clarity, funds impacted means funds hacked, stolen, or otherwise lost, but doesn’t include funds returned or reclaimed by whitehats and investigators.

Conclusion: Hacks are up in volume (and increasing), but aggregate impact appears to be trending downwards in magnitude, in a very material way. In other words, more hacks are happening, but they’re becoming much less severe on a per-hack basis, and the aggregate hack impact appears to be decreasing as of 2023. So, onchain security is objectively improving according to the metric that matters most (funds stolen). 

Another fact becomes apparent: the future I was so concerned about (an onslaught of onchain hacks) did occur as forecasted, but the impact of these increasing hacks appears to have been limited. The crime wave came, but the industry had blunted its force.

Clearly, an early onchain financial armageddon had been successfully prevented!

But this doesn’t show us what prevented it. While Immunefi and the scaling bug bounty standard are logical candidates, there could be other causes. To understand that, we need to look toward funds saved and exploits patched as evidence of security success. 

Attributing impact from theft prevention

The most effective way to understand what drives security impact is to quantify and categorize interventions that successfully prevent security incidents. Here’s a quick review of security measures and their quantifiable impact:

• Security incidents averted via audits and external code reviews

  • We can look at the quantity of vulnerability findings in public audit reports. Unfortunately, it will be an incomplete dataset, since countless audit reports are private, and many audit findings are unlikely to result in security incidents (as most findings aren’t critical severity findings). Finally, nearly every firm uses a different measure of severity, when even most such ‘high’ severity vulnerabilities may not lead to loss of funds or material security events. There are some rare exceptions to this rule, like ThreeSigma, which uses Immunefi’s Vulnerability Scoring System to hold itself to a high performance bar.

  • For external code reviews, most of the code reviews aren’t happening on mainnet. Additionally, the focus is not on the most critical types of bugs. Finally, the data on any funds that would have been saved from such disclosures is unavailable. 

  • This isn’t to say audits and external code reviews aren’t worthwhile. We believe that audits are highly effective tools for preventing hacks, and that every onchain founder should use them as standard tools in their security toolbox. It’s just that their real-world impact is challenging to quantify in dollar terms. As a reflection of our conviction, Immunefi itself offers two audit contest products, Boosts and Invite-Only Programs, connecting founders with Immunefi’s top hackers to provide unparalleled code security outcomes prior to deployment.

  • Still, we can measure the degree to which they failed to prevent failures by looking at the number of hacks that occurred on audited protocols. Unfortunately, the data shows that nearly all major hacks occurred on widely and regularly audited protocols. It seems unlikely that audits are predicting hacks per se. It seems much more likely that audits are so widely adopted and predictive of future success that all major targets of value for hacking necessarily get audits, explaining the correlation. Still, the quantified impact of audits on ecosystem security remains unclear. I may do a deeper dive into the historical efficacy of onchain audits in the future to settle this question for my own understanding.

• Security incidents averted via monitoring and interception solutions

  • Onchain monitoring systems have a similar advantage to bug bounties. Their specific impact can be quantified, and a security event can be clearly classified as resolved successfully (or not). 

  • Unfortunately, I’m aware of no public and verifiable store of onchain interception events that can be trusted to come to such estimates (though if you find one, please let me know!). Adoption only really began in earnest in 2023, so it seems safe to conclude that monitoring solutions were not responsible for major security successes over the past three years, but this is hard to know for certain.

  • With luck, monitoring solutions will get more adoption and more transparency over 2024, so that we can all understand more about their efficacy.

• Security incidents averted via bug bounties and responsible disclosures

  • Bug bounties have the advantage of being quantifiable in impact, so we should be able to derive a specific, high-conviction estimate on the efficacy of bug bounties and responsible disclosures in driving threat prevention by looking at real $Value saved from exploitation.

  • This number will necessarily be very conservative, because the dataset is incomplete (bug bounty platforms do not currently share data with one another), and the difficulty of assessing quantified exploit impact (which forces us to use the minimum impact estimate to stand on the firmest ground, and that from only a small sample of reports).

Can we tell how much of this security impact is driven by Immunefi? Sort of. Since we can’t effectively measure security impact from most of the practices, we can’t establish weightings between the measures, but we can weigh the security impact of bug bounties themselves.

And by weighing this impact according to funds saved, we can then make our own evaluation as to whether the potential impact could or would have resulted in an onchain financial cataclysm, and this would be sufficient for us to understand whether Immunefi achieved its mission.

To weigh the security impact of onchain bug bounties, there are two meaningful indicators: 

1. The aggregate funds saved from theft and hacks by critical bug reports. Out of the two indicators, this is the most important. 

   1. For context, critical onchain vulnerabilitiess are defined in Immunefi’s Vulnerability Severity System as those that directly cause loss of funds (via theft or freezing), cause total network shutdown or unintended chain splitting necessitating a contentious hard fork to fix, or serious cryptographic flaws that undermine the integrity of the entire system. Such vulnerabilities truly deserve the critical levelling.

2. The number of critical bug reports. Immunefi’s severity definition of ‘critical’ almost always refers to direct funds at risk.

   1. We can consider this a useful proxy of bug bounty efficacy in aggregate due to the vast majority of onchain funds-at-risk vulnerabilities flowing through Immunefi for several years, though it does mean that this estimate is necessarily conservative.

   2. We can then compare this number directly against hack numbers to understand the impact bug bounties are having on preventing hack events.

In reviewing 1), there is no comprehensive dataset tallying funds saved by bug bounties. Such a dataset would require a massive amount of manual work in quantifying impact report by report. Fortunately, however, Immunefi keeps track of a small subset of these previously mentioned critical reports, covering roughly 33 of the total 573 critical reports (2021-2023), for the purpose of making public educational bugfix reviews for the community. We can call this our ‘bugfix review dataset’. While it’s just a small percentage of all critical reports, it does provide us with a set of publicly verifiable reports.

This bugfix review dataset alone prevented over $25 billion USD in direct harm, based on conservative criteria. By conservative, I mean: 1) an analysis of only a small fraction of Immunefi’s paid report database (again, just 33 of 573 total critical reports), 2) measurement of only stablecoins and liquid assets with material market depth, 3) no counting of dependent equity, most tokens (including all but directly affected and highly liquid governance tokens), derivatives, and market impact on token price depending on the underlying vulnerability affected, and finally 4) no counting of financial impact to products and assets built on top vulnerability-affected platforms. This $25b figure can be re-derived from all of the bugfix reviews posted on our Medium. 

If we compare this number to the aggregate hacks that did occur ($7,808,402,225 USD in hacks between 2021-2023), we can see that the Immunefi community alone saved a 3.2 multiple of that figure. By this measure, too, bug bounties have proven themselves to be a phenomenally successful security tool. If we remove some of these assumptions and include impacted value across ecosystem platform products, collateral damage to tokens and equity, or derivatives of various kinds, the probable impact would be multiples of the above number. Loosening the conservatism of $25 billion quickly brings us into the $30 billion, $50 billion, and $70 billion in funds saved range.

For 2), we can review valid critical smart contract and blockchain reports on Immunefi, year by year, and stack them up against hacks for comparison.

The data is stark: Immunefi prevented approximately 594 security events (where each security event is a potential hack), against 488 realized hacks and exploits. Clearly, bug bounties are carrying an enormous security load by preventing hundreds of would-be hacks and security events.

It’s now clear that bug bounties and responsible disclosures have played a massive role in protecting the industry over the last three years, and one vastly in excess of bounties paid (sitting comfortably at $95 million total over the last three years).

It also becomes clear that, if Immunefi and bug bounties had not been widely adopted, the damage that would have been caused ($25 billion USD in the most conservative estimates) would be multiples of the real hack damage that did occur ($7.8 billion between 2021 and 2023 according to our statistics). 

Do you think an extra $25 billion USD in hacks would have constituted crypto-financial armageddon? The negative impact of damage on that scale would probably have been so severe that DeFi itself would likely have been delegitimized and made illegal in many countries. 

Clearly, Immunefi and bug bounty adoption have indeed played a crucial role in preventing a blockchain hacking apocalypse.

Impact from talent pool growth

The scaling bug bounty standard has also driven security impact by incentivizing and nurturing the growth of the blocksec community. 

While it’s impossible to put a direct dollar sign on this factor, it’s safe to say that it will be a major driver of long-term security impact in the blockchain ecosystem, and one that compounds yearly as more professionals find ways to keep the onchain economy safe. 

But we should attempt to quantify the impact somehow. Any assessment will be incomplete, but the growth of the blockchain security community has exploded, so we can still extract some useful insights:

Immunefi alone now has >40,000 registered hackers and auditors, many times more than my previous estimate of the entire industry in 2020 (which was less than the estimated 1,000 security professionals in total). 

In 2020, there were less than 30 notable auditing firms. Today, there are multiples more auditing firms, plus hundreds of solo auditors acting as independent contractors and thousands of security researchers participating in Immunefi Boosts and audit contests.

In late 2020, when Immunefi started, there were only four major DeFi security startups of note (Immunefi, Hats Finance, Code4rena, and Forta). Today, there are dozens, and at least 50 by my count (and probably double that in reality). That is an explosion of blocksec ventures.

In 2020, security hires for blockchain startups were far and few between. In 2024, I frequently see a security hire among the first 10 hires, which demonstrates a huge increase in the number of security engineers working in blockchain. 

More anecdotally, I have had hundreds of conversations with security researchers, engineers, and at least a dozen active blocksec founders that share that they got into blockchain security thanks to Immunefi and scaling bug bounties. 

These proxy measurements indicate a massive increase in the size of blockchain’s security talent pool, and that this dramatic blocksec community growth was (and is) nurtured by Immunefi and scaling bug bounties.

That’s nice, but this is a retrospective, so what didn’t work?

It turns out that driving the adoption of Immunefi and the scaling bug bounty standard was not an instant success but rather a roller coaster of problem-solving and miscalculations. Here are a few.

I thought scaling bug bounty adoption would be straightforward; it wasn’t. 

I really thought when we were getting started that we would figure out the adoption question pretty quickly. I used to tell Immunefites that there were just fifty key problems that we needed to solve, and it’d be all unicorns from there. I was wrong, for with every big success came a whole new set of problems, multiplying with every victory.

Driving bug bounty adoption proved to be a Herculean task. It took hundreds–even thousands–of meetings and dozens of product iterations to really make adoption happen.

It’s hard to persuade people to adopt a new standard. When we started in 2020, the vast majority of onchain projects had no bug bounty program, and many of them weren’t even familiar with the concept. They had no budget for anything security-related outside audits, and showing them why they should spend even more money on something so unproven was a challenge.

The key insight ended up being framing: we needed to show people that if they put money here, they were much more likely to avoid catastrophic security events than not, and furthermore, there were no effective alternatives at the time. When we figured that out, we had a path to victory, but we still needed to hike that path.

Adoption happened through countless pushes to roll the boulder to the top of the hill. We knew we needed to hit a critical mass for the standard to take hold, but we did not understand that this critical mass was in the 100+ project range, including many giant industry names and multiple million-dollar bounties by early adopter partners.

Without any one of those ingredients, industry adoption would have failed. 

Even more worrying, adoption has been limited mainly to the most security-proficient projects, which understand the standard's effectiveness and value. The majority of DeFi projects do not run a bug bounty program to this day! Insanity! 

I am still pushing the adoption boulder up the hill to this day.

I thought bounties might scale in bounty size indefinitely; they haven’t.

The more the scaling bug bounty got adopted, the more an asymptote began to appear. First came the invention of the bounty cap, wherein scaling clauses would be bounded by a preset hard figure (ie. 10% of funds at risk up to $10m USD). This turned out to be a very good idea, as many projects simply did not have the funds to scale their bounties (for reasons we may cover in another post), and to pretend otherwise would have led to a bad place.

As more and more projects adopted the thesis, a ‘market pricing’ phenomenon began to occur, as projects anchored their bounty sizing to what other similar projects were doing. A sort of ‘market rate’ for bug bounty caps emerged. Total bounties available and average critical bounty size continue to trend upwards, but now do so gradually as projects grow larger and their bug bounty becomes more important for ongoing survival, as opposed to the dramatic upward surges that we saw in the 2021-2022 period.

It has been fascinating to observe market standards within market standard (the scaling bug bounty itself) emerge, but fascinating does not mean expected, and it wasn’t a helpful surprise at the time.

In early 2022, I predicted that by 2025, there would be $1 billion in bug bounties available. Sitting comfortably now at the beginning of 2024, I’m not so sure that prediction will come true, but I do think a gradual rise to this level by 2030 remains viable. We could hit those numbers on the onchain DeFi-market growth alone, leaving current bounties-per-project flat. But I no longer think that bounties will scale as aggressively as they once did.

I thought the incentives alone would be compelling enough for almost the entire security community to participate in bug bounty hunting; they didn’t.

I knew that bug bounty hunting required a certain type of person. You need to be highly self-motivated, to love cracking puzzles that nobody has solved, risk-tolerant of the ups and downs of bug-hunting, and you need a strong sense of personal honor and integrity (or you might succumb to blackhat temptations). While I’ve been blessed to work with many such people over the past few years, this life isn’t for everyone. 

Almost everyone in security has one or two of these traits, but a much smaller subset has all four.

But that doesn’t mean the rest of the security community didn’t help! Many of these amazing professionals did fantastic work championing the standard and driving the adoption of bug bounty programs in the organizations they touched. For that, they have my enduring respect and appreciation. We couldn’t have done it without them.

But still, I definitely missed the mark here.

I thought effective alternative standards would emerge. One hasn’t.

When I put out the scaling bug bounty standard, I thought that this would be the beginning of an evolutionary process for bug bounty standards. This didn’t happen. Immunefi and the scaling bug bounty standard itself evolved (as this retrospective shows) thanks to our friends in the wider blocksec community (and most especially Immunefi’s customers), but no credible challenger standards have emerged.

Immunefi and the scaling bug bounty standard stand alone as the best-in-class bug bounty standard to this day.

Finishing up; what’s next?

Three years ago, I founded Immunefi with the aim of preventing an onchain financial armageddon through the mass adoption of bug bounties. It seems to me that we at Immunefi, in fellowship with the wider onchain security community, have succeeded in preventing just such a disaster. It also seems that Immunefi has had a special role in safeguarding the onchain economy over the last few years. Immunefi really did become a crucial part of the onchain immune system, as we had hoped.

But in achieving this, I’ve been inspired to think in bigger terms. So, Immunefi now has a vastly bigger mission, toward making the onchain world safe for a world of open applications to thrive. Preventing the hacking apocalypse? Now that’s our every day, but there’s so much more that is needed.

So what's next? While we've raised the bar on blockchain security worldwide, the value onchain today remains a sliver of what's to come; we must prepare for the next $10 trillion dollars onchain to arrive. That means we need to raise the quality bar of onchain security by yet another order of magnitude. And there’s no other way to achieve that than to invest deeper into onchain security across the stack, from audits to bug bounties, from automated monitoring and prevention to open standards. Consider this an open call to leading projects and top investors to triple down on their onchain security investments; it will be the most effective industry growth spend ever spent.

And while the onchain economy at large needs to make big investments into security to succeed, the security industry needs to deliver technologies and solutions that are even more effective than they are today, until frequent hacks become a thing of the past.

That’s why Immunefi is building the future of onchain security to secure the next billion users and the next $10 trillion dollars. Our bug bounty platform was just the beginning. With the advent of our Boosts/Invite-Only Programs audit contests, and our leading security partnerships across the industry, Immunefi will provide everything protocols need to stay safe and secure.

We will do our part to raise the bar for onchain security. As always, I invite you to be a part of it.

As for me, consider this post the beginning of a new journey, where I share my hard-won discoveries from the security frontier for the benefit of our onchain world. 

---

If you like my blog, please subscribe & share it with your friends. I write in my free time, so seeing more people read these posts motivates me to write more. I don’t send anything except my writing.