Five years ago today, our co-founders launched Gremlin with a simple but bold mission: Build a more reliable internet. Over the past five years, the practice of Chaos Engineering is increasingly employed as a means for proactively testing systems to make them more resilient and reliable. Chaos Engineering has reached mainstream media (with articles in Politico and Bloomberg), community conferences on the subject have grown from a few hundred attendees to 3,500+ registrants, and Gremlin’s own user base has been actively testing systems, having executed nearly half a million Chaos Engineering attacks. 

All of this interest got us curious: who is actually practicing Chaos Engineering, what techniques are they using, what problems are they solving, and what are they seeing as a result? To find out, we surveyed the community and examined Gremlin platform data, resulting in the first State of Chaos Engineering Report. While the report confirms the growing interest in Chaos Engineering, we also observed that the nature of software failure modes has shifted. To better support teams of all sizes in their efforts to prepare their systems for a wide variety of failure scenarios, we are making the entire Gremlin attack library available to users of Gremlin Free.

State of Chaos Engineering Report

With more than 500 responses, primarily from software and site reliability engineers, we identified the ways in which these roles use Chaos Engineering to improve the reliability and resilience of their systems. 

The top benefits to Chaos Engineering? Increased availability and decreased mean time to resolution (MTTR). In fact, teams who frequently run Chaos Engineering experiments were more likely to have >99.9% availability—an absolutely impressive feat. 23% of teams have an MTTR of under 1 hour, and over 60% of teams have an MTTR of under 12 hours. Not only is Chaos Engineering keeping services up and running, it's creating more informed dev and ops teams who are better equipped to respond to incidents when they do happen.

However, the fear of testing in production is real; only 34% of respondents run Chaos Engineering experiments in production. Dev and staging are much more common environments for running attacks. Proactive testing in lower environments can provide better confidence in the stability of services without directly impacting customer experiences. While Gremlin has continued to advocate for testing across all environments, and we expect the percent of respondents testing in production to increase over time, we also recognize that not all critical services can be experimented on in production—emergency response systems and autonomous vehicles come to mind. It’s important to consider a system’s function and impact on customer experiences.

The report also uncovers trends in technology environments and looks to the future to explore where Chaos Engineering is heading.

Get your copy of the State of Chaos Engineering report.

Unleash all the Gremlins, for Free

As interest in Chaos Engineering continues to grow, we want to provide teams of all sizes with a safe, secure, and scalable platform to run experiments. When we launched Gremlin Free, teams were able to run CPU and shutdown attacks, and shortly after we added blackhole attacks. We believed that offering access to these three attack types helped teams get started with the foundations of running Chaos Engineering experiments, but it also prohibited teams from identifying vulnerabilities across their entire stack. 

So much of our world has shifted online over the past 12 months and as a result, significant increases in network traffic are a new reality for businesses in just about every industry. Running latency attacks helps engineers ensure customer experience continuity, and improves the overall reliability of services. Latency attacks are the 2nd most popular attack type for Gremlin paid customers, following closely behind blackhole attacks. A latency attack allows users to intentionally slow down network requests and observe how this affects response time, page load time, application stability, and ultimately the customer experience. 

Starting today, all Gremlin Free users will have access to the full library of attack types. They can now test for memory leaks, disks filling up, latency, process killers, blocked DNS access, and more. In doing so, we’ve also opened up all of our Recommended Scenarios, allowing users to easily run Chaos Engineering experiments based on common real-world outages.

Community and Chaos Champions

A few years ago, we launched a Slack community to connect Chaos Engineering practitioners, learn best practices, find mentorship, and build reliable systems, together. Now, that community has nearly 7,000 members with more joining each day. And it’s not just the Slack community that’s growing. Last year, the world’s largest Chaos Engineering conference, Chaos Conf, saw a 440% YoY increase in registrations, and Gremlin’s Chaos Engineering Bootcamps had close to 2,000 registrants for these free, hands-on workshops.  

In October 2020, we announced the Gremlin Chaos Champions program to recognize the work practitioners were doing for their teams, community, and the Chaos Engineering field at large. The freshman class consisted of four Chaos Champions, and today we’re excited to expand that group. Please meet the newest members of the Gremlin Chaos Champion program!

Chaos Engineering has allowed us to finally test long-held assumptions about our services and enables us to continuously build more reliable infrastructure.

Doug Campbell

SRE, Grubhub

Chaos can lurk just behind the facade of order. For me, chaos engineering has been an amazing journey for discovering that reliability is based on thousands of tiny failures, and that success of resilience is based on how many times we have failed at something.

Yury Niño

SRE, Aval Digital Labs

Chaos Engineering has helped me a lot in getting a better understanding of our systems and working towards making them resilient. Sometimes chaos is the only way to achieve stability.

Saurabh Shendye

Senior SRE, Shipt

If you know someone who deserves to be recognized for their efforts leading Chaos Engineering in their organization with Gremlin, nominate them today for the Gremlin Chaos Champion Program.

We’re excited to see how the practice of Chaos Engineering and the larger reliability space evolves in 2021. We expect to see the global theme of ‘resilience’ continue to lead our thinking, as teams across all sectors and industries look for ways to make their organizations, teams, and systems more resilient and better equipped to handle the unexpected.