6 Metrics to Monitor When Rolling Out a New Feature Flag

Feature flags are a powerful tool to ensure your application updates work as expected before exposing your entire user base. Today, it’s standard practice to roll new updates out to a subset of users first, where you can monitor the results and track all necessary metrics.

You’ve got the update roll-out process down, but now what about the monitoring side of things? What exactly should you be looking at after rolling out your latest update? When monitoring feature releases, you can track results compared to a historical baseline or compare to your control — users who haven’t received the update yet. 

As you roll your feature out, it’s important to track relevant metrics. If you are like most companies we work with, it’s easiest to view and analyse this data in your existing analytics platform. For that reason Flagsmith has native integrations with all the major metrics monitoring tools, and has Webhooks so that you can export flag analytics to your own infrastructure if that’s where you need it. 

There are two types of integrations we offer: analytics platforms (like Heap & Mixpanel), and data platforms (like Datadog & AppDynamics). Most of the tactics in this post are intended for analytics platforms. You can check out all of our integrations in our docs here. 

There 

This article digs into the metrics you should track each time you roll out a new application update using a feature flag. Whether you are doing a sophisticated A/B test or a simple phased rollouts, these are helpful metrics to watch.

Time Spent in the Application

This is a valuable metric to track, but what you’re looking for will vary depending on your update. Did you roll-out a fix to reduce friction on your application? Then you may be looking for reduced time spent on the platform. If users are spending the same amount of time or are spending longer on your application than before, you may consider rolling back your update.

Meanwhile, if you simply added a new feature, you may be hoping to increase time spent on your platform. If your users are suddenly struggling to use your application or aren’t a fan of your latest update, they’ll likely spend far less time on your application. This signals an ineffective new feature and the need to roll back your update. 

Application Open Frequency

Do your users return to your application often? If a new feature is confusing or off-putting, your users will be less likely to re-open the application. Monitor how often users return to your application. If the frequency dips significantly, consider rolling back your update. Conversely, if users begin opening your application with increased frequency, your update may be successful.

Goal Completion & Time to Goal Completion

The logistics of this metric will depend on your application and the update you rolled out. No matter the specifics of your application, there’s a goal users have when using it. On Facebook Messenger, for example, the goal would be sending a message.

Consider the goal — or goals — of your application. It should be relatively easy and fast for your users to complete their goal on your application. Especially if you rolled out an update designed to reduce friction, you should look for a reduced time to goal completion and increased goal completion rate for users who have received the update.

>>Sing up for free and use feature flags!

Application Load Time

Increased application load time is incredibly frustrating for your users and is likely to drive application exits, reduce open frequency, and increase time to goal completion. It’s important to monitor your application load time and ensure it stays the same — if not lower — with a new update.

Downtime & Crashes

Similar to increased application load time, increased downtime and crashes create negative user experiences. Aim to reduce downtime and crashes with each update. If you see a significant jump in either, consider rolling back with the feature flag.

Support Requests Made

This metric is a little more nuanced than the last few. While you may assume any increase in support requests is bad, it’s important to consider that with any new update, your users are likely to need a little extra help understanding how to use the new feature.

Instead of just looking at the number of support requests only, look at the requests themself as well. Oftentimes, this information can help you ensure that you’re providing the right support when you roll out the update to your wider userbase.

>>Sing up for free and use feature flags!

Improve Your New Feature Roll-outs with Flagsmith

Flagsmith helps you streamline your feature roll-outs with feature flags. Flagsmith integrates with your existing tech stack, making it easy to analyze the performance of new features you roll out. Revamp your new feature rollout strategy with Flagsmith — get started for free.