How to Roll out Pricing Changes With Zero Customer Complaints

We use feature flags across all components of the Flagsmith system. Though this in itself will be unsurprising, it means we get some interesting real-life applications. I wanted to share a recent one that you might find helpful. 

This autumn, we rolled out a pricing update to our users (not necessarily newsworthy), but it resulted in zero customer complaints. People don’t generally like pricing changes, however justified, so let’s talk about how we did it.

The challenge

Flagsmith has historically not automatically monitored API overages in our SaaS environment. As we grew, we needed to change this to protect the platform from misuse. This meant that we had to introduce automated alerting, blocking, and charging for these overages. This was a daunting challenge, and one that would not have been possible without the use of feature flags. 

‍

The process

Before setting out to develop the required functionality, we planned which flags we would need and what the rollout plan would be. First, though, we need some context about the requirements for the functionality. This can also be seen in our documentation here, but I’ve also summarised it in this article for convenience. 

1. All customers should be notified when they hit certain thresholds based on their API request limits
2. After a defined grace period, customers on the free plan should have blocks placed upon their usage if they exceed their usage
3. After a defined grace period, paid customers should be charged for any overages at the end of each billing period

Based on this information, and before writing any code, we created the flags that we wanted to use. As you can probably guess, we created a flag for each of the requirements defined here.

Once we had the flags in place, we wrote each piece of functionality wrapped in its flag. Since this functionality was entirely localised to the backend code of the platform, and we still have a staging environment for our backend, we deployed it there (by virtue of a merge to our main branch) and verified that, with the flags disabled, none of the new functionality was executed. As soon as we confirmed that, we were able to continue to deploy code into production as frequently as we liked, while we worked on testing the functionality. 

‍

Testing

Initially, we tested the basics of the functionality in our staging environment since it allowed us to easily test the logic against our staging payments provider using fake bank details. However, using segments, we were also able to test the functionality in production, particularly for free organisations—which are easy to create and test in production. 

Once we completed the testing, we began to roll out the changes to our customers. Initially, we began to enable just the alerting, and just for our free customers. We started with 5% of our free customers, increasing the percentage over a few days to 100%. We were able to do this using the specific granular feature flag and using segments to target only customers on our free plan.

The results

Once we were happy with the alerting, we were then able to enable the flag that controlled the blocks that were applied to free plan accounts. For these, since the code only applied them to free plan customers, we didn’t need to use segments and were able to just enable the flag for all of our customers at once. We still had the safety of being able to disable these at any point should we need to. 

Now, onto our paid customers, this came with a lot more risks as we didn’t want to end up in a situation where we overcharged one of our customers (or undercharged them – just a little bit of foreshadowing for you!). As such, we began with a very small percentage of our paid customers, again using segments to target specific plans, and to restrict the rollout to a small percentage of those customers. 

This proved to be invaluable. We quickly realised our calculations were off by a factor of 100. Instead of charging the first customer we rolled this out to $150, we charged them $1.50. We were monitoring the rollout and caught the issue through an alert, immediately correcting it, but there were some tense moments! Thankfully, having the flags in place allowed us to disable the functionality while we worked on resolving the issue.

Once we fixed our maths, we were able to continue monitoring the functionality and verify that the charges were being correctly calculated and added to the accounts. In case of any issues, we would be able to quickly disable the flag for all customers and investigate the issue before continuing the rollout.

This was a high-stakes initiative for us, and feature flags made it possible while also providing peace of mind – not to mention helping us avoid an awkward mass email explaining why we’d charged customers for a lemonade instead of an overage. 

If you’re curious, you can check out more on segmentation and staged rollouts, as well as dig into how the fintech platform Rain uses segmentation to enable features for specific users.

‍

About the author

Matthew helped start Flagsmith and is the CTO. He brings a background in project management to his software engineering work.