Don’t Let a Flawed Release Take Your Company Down

These days, software is at the core of almost everything we do as human beings. Sometimes we take that for granted. Recently, we collectively experienced one tool impacting lives across the world. Global banking systems were brought down. Payment systems were impacted and people couldn’t get paid. Airplanes were grounded. Airlines were unable to run simple applications to even tell people why. Critically, healthcare providers were taken offline and doctors couldn’t work.

The most painful part of all of this is how preventable it was. CrowdStrike released a flawed software configuration that took down millions and millions of Windows PCs. It’s wild to think that in 2024, a single software update can cause a global outage.

Thanks to their release processes, a large but little known company is now famous for all the wrong reasons. Outside of the PR nightmare and instant memes, the company also saw billions of dollars wiped off its market cap in the time it took for millions of PCs to blue screen.

How could feature flags have helped?

The concept of a feature flag is simple—but the capabilities they unlock for companies are powerful. In the simplest form, our customers use flags to mitigate risk when something goes wrong; rolling back code with the click of a button. 

In the case of larger organisations releasing critical software, you can also set flags up to enable canary releases. For those who aren’t familiar with the term, a canary release is a progressive rollout that allows you to release code initially to just a small percentage of your users. If you’re a company like CrowdStrike, you can roll out your software update to 10% of your users and see how it performs. If all goes well, you can increase that figure over time, all through the click of a button. If it doesn’t, you can pivot or roll it back without needing to redeploy.

What would this look like in Flagsmith?

There are two ways to do this in Flagsmith. The simplest way is to put the feature behind a feature flag that allows your product team to be able to shut off a feature when you notice an issue. That process would look like this:

‍

‍

The second, the canary release, would have been a recommendation we would’ve made to a client like CrowdStrike. Instead of rolling out the feature to all Windows’ machines, do a progressive delivery that allows you to target a percentage of your users. You can then connect this with observability tools (like Dynatrace) to see how it’s performing in production before deciding if you increase the percentage or not. It would look like this:

‍

‍

Let’s try to prevent cases like the CrowdStrike scenario from happening to others. Talk to us about how you can implement feature flagging to evolve how you release code. With Flagsmith, you can do this with our cloud solution or integrate it fully with your stack on premise.