Why You Need an Experimentation Template

I have spent almost half of my data career working at Block, starting as an IC, managing people for some time and now as a DS tech lead. I’ve also been on multiple teams here: Developer Empowerment, e-commerce, sales analytics and most recently - at Trusted Identity within Payment Platform. Most of my time here has been focused on product data science.

We run a lot of A/B tests at Block since it’s one of the most data-driven ways of showing the impact of a new product feature. Many data science teams here use an internally developed A/B testing template and I’ve found it to be incredibly useful to ensure that stakeholders have thought about the right business questions prior to asking for something to be launched. The template also creates consistency across teams, which makes it easy to see and understand the tests that other teams have run. I’ll go into other reasons why we love using the template but it also provides the right guidance for less experienced data scientists. It poses important questions for both stakeholders and data scientists to ensure that key details are thought through prior to setting up a test and launching it.

I’m sharing a genericized version of our template created by Doug Logue and you’re welcome to make a copy and modify it as necessary for your own use cases! There’s no need to reinvent the wheel here. Some parts of the template are self-explanatory. I’ll describe two of the most useful sections of the template below so you don’t have to learn the hard way.

Overview section

Translate your hypothesis into a business problem: One thing worth calling out here is the problem statement. I find that we often focus on the hypothesis, e.g. that removing a field would help users go through the signup process faster. However, we don’t often think about what the actual problem is that we are solving. Make sure that you translate the hypothesis into a business problem. For example, a hypothesis may be that a conversion rate will increase if you remove fields to be completed during the signup process. The business problem that you are solving here is removing friction for new users.

Agree on the launch criteria ahead of time: We often say ‘we will launch if there is a 1pp lift in a key metric’. However, an A/B test that produces neutral results, some may have a bias to launch and this can result in a conflict between data scientists and stakeholders. Often, in our roles, we don’t want to roll something out as we don’t see a positive impact. Even though the results of our A/B test haven’t resulted in a positive lift to any metrics, a PM may still want to ramp up the experience to 100%. This could be because they have seen the results of user experience research showing positive feedback. Agreeing on launch criteria ahead of time between all stakeholders involved would also mean that you can avoid confusion and a potentially long-winded discussion or conflict at the end of the test, trying to decide if it should be rolled out or not.

Learnings section - stay curious

Make note of any interesting follow up analyses or A/B tests to run after your experiment has finished

Running an A/B test isn’t just about rolling out an experience and showing a positive lift to our primary metric. Once we set out primary and secondary metrics, we not only want to test if they move but also why. It’s worth calling out any analyses or learnings we may want to take from a test, for example while testing an onboarding flow. We want to learn where users drop off or what points of friction we have removed for them. This helps to get to the _why _of user behavior, which can inspire future improvements or inform decision making.

As an example, imagine that you see an increase in the % of users that complete the onboarding and take a payment. What else can you learn from it? One thing to indicate is to see which steps users drop off at the most, or inversely - to highlight any friction that may have been unintentionally introduced. Keep it to 1-2 things you want to look at and as always, think about what can drive the most impact for your stakeholders.

Link to the template - feel free to make a copy!