Achieving agility and high performing teams with feature flags
Agile software development has been around for 23 years now, or at least that was when the manifesto was signed. Not all companies are fully following all of the principles, and not all methodologies truly reflect the concepts of agile, but an incredible amount of processes and organizational structures changed over these years with the end goal to become more agile.
The idea of using feature flags, however simple it may look at first, contributes to achieving at least 4 out of 12 principles of the agile manifesto. The software development process, regardless how optimized, cannot be fully agile without them. Feature flags are also a prerequisite to achieving high performance teams. Here we will look at the benefits feature flags bring from the perspective of several different roles in a typical team/company, as they are an enabler for iterations, gathering feedback, and controlling changes.
Software Architect
Architects can manage the changes in architecture of an application much easier with feature flags. For example, they can control which version of a service is used by which users, giving them the ability to release new versions much sooner, validate them and gradually move the users between versions. This means that a new service can start off on a very small instance and scale, so architects can see the relationship between user needs and performance and adapt by making much fewer assumptions. Feature flags and versions also make the architecture loosely coupled in a multi service environment, as new versions of one service don't impact the work of the others, and they tend to be compatible with the existing services.
Software Developer
Developers usually are bothered by two big problems (at least): specification and quality.
Specification means common understanding by everyone on what needs to be built - and usually when something is not working as expected it's the first thing to blame. Typical sentences being "it was not specified like that", "there was no such scenario", "they said this but wanted that'', "we didn't understand it like that". So how can feature flags help here? Well, like it or not, real user acceptance testing mostly happens on production. You can specify on thousands of pages, design files, prototypes, deploy on test/stage environments - but the real deal happens on production. That is where the problems become obvious as that is where people truly care. Welcome feature flags - let's release a version as soon as it covers the main scenarios and the happy flows, and only to a tiny amount of people. Amazingly enough, these people specify what they want and need much more accurately now! We get their feedback in the form of true user stories, not made up ones.
And then quality: releasing features is always going to produce bugs, unexpected problems and god forbid incidents. Developers are among the first on the line, and their explanation is "it wasn't tested", "we didn't have enough coverage", "test data was not accurate enough"... So the same things happen with feature flags but now these problems become part of the process. They are expected, can be controlled and have way smaller impact. For example, imagine we are releasing a change in pricing calculations. With feature flags we release to some selected user groups that provide variety and we notice the big problem: rounding accuracy. Ok, we switch back, fix it properly in a couple of days and release it again. Users continued working, damage was controlled, we had time to think of a fix and find a proper solution. Imagine the same scenario without the possibility to switch the feature to its previous version. Point is: control increases quality.
Quality Assurance
We all have had a lot of risky deployments with big impact and we have seen the eyes of a QA on the day of the release. They are supposed to be a guarantee that things will work as expected. Let's go several steps back: what was expected? How many times did those expectancies change? How many times deadlines change and how much time did we have to test the application at peace without someone 'fixing' stuff midway? Did everyone else on the team test things? Did we have time to write automated tests for all important scenarios? Did we have enough time for regression testing? It's easy to see the impossible situation QA faces and why their eyes don't portray confidence on that day.
So how do feature flags help? Well, QAs are confident when they can do automated regression testing which buys them the time to think about the future version and how to properly test it. By having the possibility to deploy a version which makes no changes on production regression testing is much easier. Also, QAs can now compare test results between the two versions, find the differences and analyze if that is intended behavior. So now we moved to a situation where everything already running on production is considered as a baseline and we can only test the changes. Now we have much more time to cope with changing requirements and guarantee quality. And it's ok if the new version has bugs, it's even expected.
Product owner
As a product owner you are at times (hopefully only at times) caught between users and the development team. Changing requirements coming from users are always met with a sad face, and you need to explain to the team why we have to change things and still meet the deadline, forcing them to cut corners knowing that it will deteriorate the quality of the product in the end.
Feature flags are an important tool if used right; a product owner can now form a group of 'friends' of the product and this group gets a part of the feature first. Change requirements are now met as improvements for the next version and planning shifts to releasing what we can as soon as possible. If you know SQL or you have UI to select which tenants\accounts\users get the feature (lucky you), you are now in control of releasing while the team is in control of deploying changes. You are no longer surprised by wake-up calls early in the morning with strange questions about rounding issues. Instead, you prepare the release email, closely monitor the activity, contact the small group of friends to get structured feedback and translate it to the team. Plus, you are now meeting the deadline most of the time! Without feature flags it's 1 or 0 game, where 1 gets usually delayed. With feature flags it's a value between, and people are mostly satisfied if it's more than 0.7 on the deadline, with the promise to get closer to 1 over time.
DevOps
Decoupling release from deployment is on top of most DevOps engineers Christmas present list. CI\CD can finally function as it should without so much risk, as branches are much shorter, so integration happens more frequently. Software is in a deployable state at all times. Scaling the environment can happen while releasing gradually to users. Proper monitoring of key indicators becomes something of value to everyone, not only to the DevOps and support teams, since now everyone is asking how is the new version performing? It's everyone's goal to get the new version up to speed.
Marketing & Sales
Yes, feature flags bring agility to marketing and sales. Marketing campaigns can now turn from selling a finished feature in which we have invested tons of time to selling an early version to a segment of the market and checking the pulse. Then with this information decisions for pricing and how to approach potential customers are objective, not hunches. Pricing models are now much more flexible, as each feature and version can be priced differently. Also, the benefit of using the early birds group as promoters is very valuable, as their opinion matters to the potential customers.
UI\UX Design
As designers we strive for perfection, a genius product that is built around the users and responds seamlessly to all their needs. However, achieving that goal is like running a super marathon while preparing lunch and cleaning at home. You need to have the final goal laser focused, but you have to make daily decisions for small things that matter. If you get involved in one of these activities too much and lose balance you will either end up with a perfect design that nobody is implementing or with a bunch of screens for smaller features that don't fit in the puzzle.
Feature flags and versions help designers find this balance. Designers can use them to their advantage by first designing a visionary design and then evolutionary versions depicting how to get there. Each version gives designers feedback, so that in the same way as evolution they can adapt the design to the environment (but way faster). Versions also help designers doing AB testing on focus groups on production, which is of great value to product development.
Use case of feature flags and versioning standard
Having an overview of what is important to every role - and why, gives us a clear understanding of how to implement feature flags and versions. Feature flags should support granular levels of activation (accounts, roles, specific users, maybe geographical areas or even users that apply for beta versions), multiple versions of a feature (can be semver or alphabetic versions), proper monitoring and analytics in order to see usage and statistics, a UI for product owners to configure them, and they should be accompanied by feedback tools like surveys or small feedback forms as part of the UI to systematically gather the users feedback.
One use case we can share as an example is an implementation that is a module of an authorisation service. The feature flags are closely tied to the permissions issued in the JWT, which means that they are transparent to other services and can be used with basically no implementation changes. Monitoring is done with elastic and Kibana and a specific dashboard is created for every version targeting important metrics relevant to the success of the respective feature. We still lack the UI for the product owner, meaning that control for activating features and overview of who is using what is somewhat cumbersome, but that is the next step.
Conclusion
Feature flags are valuable to every member of the team, when done right. They enable agility by improving delivery speed and continuity, gathering feedback, communicating with users, responding to change and by closing the gap between the business and engineering slides, making them work together as a unit with shared responsibilities. They promote high performance teams by eliminating stressful releases, shortening iterations by working in smaller batches and contributing to CI, providing time and focus for QA automation, providing ground for experimenting and limiting work in progress.
