The text discusses the challenges and limitations of A/B testing for smaller companies, as well as the need to carefully allocate resources and set realistic expectations for experimentation. It emphasizes the importance of test sensitivity, resource-first design, and categorizing changes into “natural” and “experimental” to manage resources effectively. The author recommends a gradual approach to introducing A/B testing and emphasizes the need to understand and work within one’s limitations when conducting experiments.
“`html
Not A/B Testing Everything is Fine
Leading voices in experimentation suggest that you test everything. Some inconvenient truths about A/B testing suggest it’s better not to.
The Resources
If you know absolutely nothing about A/B testing, the idea may seem quite simple — just take two versions of something and compare them against each other. The one that shows a higher number of conversions (revenue per user, clicks, registrations, etc) is deemed better.
The Users and the Sensitivity
The majority of successful experiments at companies like Microsoft and AirBnb had an uplift of less than 3%. Those of you who are familiar with the concept of statistical power, know that the more randomization units we have in each group (for the sake of simplicity lets refer to them as “users”), the higher the chance you will be able to detect the difference between the variants (all else being equal), and that’s another crucial difference between huge companies like Google and your average online business —yours may not have nearly as many users and traffic for detecting small differences of up to 3%, even detecting something like 5% uplift with an adequate statistical power (the industry standard is 0.80) may be a challenge.
The Problem
When working on your A/B testing strategy, you have to look at a bigger picture: available resources, amount of traffic you get and how much time you have on your hands.
The Solution
In a more traditional setting, the flow for launching an A/B test goes something like this: someone comes up with an idea of a certain change, you estimate the required resources for implementing the change, those involved make the change come true (designers, developers, product managers), you set up MDE (minimum detectable effect) and the other parameters (alpha, beta, type of test — two-tailed, one-tailed), you calculate the required sample size and find out how long the test have to run given the parameters, you launch the test.
The Source of Change
I should also introduce the term “the source of change” to expand on my idea and methodology further. At SendPulse, like any other company, things get pushed to production all the time, including those that deal with the user interface, usability and other cosmetics. They‘d been released long before we introduced experimentation because, you know, a business can’t stand still. At the same time, there are those changes that we specifically would like to test, for example someone comes up with an interesting but a risky idea, and that we wouldn’t release otherwise.
The Results (Hopefully Positive)
How exactly does this strategy help a growing business to adapt to experimentation mindset? I feel that the reader have figured it out by this time, but it never hurts to recap.
If you want to evolve your company with AI, stay competitive, use for your advantage Not A/B Testing Everything is Fine.
Discover how AI can redefine your way of work.
Identify Automation Opportunities: Locate key customer interaction points that can benefit from AI.
Define KPIs: Ensure your AI endeavors have measurable impacts on business outcomes.
Select an AI Solution: Choose tools that align with your needs and provide customization.
Implement Gradually: Start with a pilot, gather data, and expand AI usage judiciously.
For AI KPI management advice, connect with us at hello@itinai.com. And for continuous insights into leveraging AI, stay tuned on our Telegram t.me/itinainews or Twitter @itinaicom.
Spotlight on a Practical AI Solution:
Consider the AI Sales Bot from itinai.com/aisalesbot designed to automate customer engagement 24/7 and manage interactions across all customer journey stages.
Discover how AI can redefine your sales processes and customer engagement. Explore solutions at itinai.com.
“`
