Don’t Wait For Statistical Significance

You don’t need statistical significance to make data informed decisions. In fact, it may be slowing down your growth.

While I do think you should get statistical significance when you can, for a lot of startups this could mean a long wait because there’s not enough data available. This is especially bad when you consider the growing importance of high tempo testing.

But this doesn’t mean you can just make decisions on your gut, a magic 8 ball or darts. You should still try and gather as much data and info as you can.

Before I get too far ahead of myself, here’s something to keep in mind…

We’ve been using P-values wrong

The American Statistical Association (ASA), recently came out with a definitive statement that we’re all relying too heavily on p-values and statistical significance to validate results.

In their statement, the ASA gave six guidelines on using statistical significance:

1. P-values can indicate how incompatible the data are with a specified statistical model.
2. P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.
3. Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold.
4. Proper inference requires full reporting and transparency.
5. A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.
6. By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis.

This is pretty significant considering how big a factor optimization is to drive growth. But when you’re early stage in a startup and don’t have tons of data to run tests with “significance,” you need other options.

Go for big swings over small tweaks

Andrew Chen gave a great interview to Mixpanel’s blog about the state of growth hacking, and in it the point was made that “Growth isn’t about 2x. It’s 10x or 100x.”

To reach that level of growth requires big swings.

“I’ll talk with people about their road maps and I’ll ask them, ‘You have this great thing that’s working, but how do you 10x it?’ Maybe they’ll have some ideas, but then I’ll say, ‘Okay, now what would you do to 100X it?’ It gets to a point where you need to take really big swings to make that happen.”

That type of thinking can help you overcome a lack of data points available (like website traffic), by avoiding small tweaks when setting up tests. Rather than changing a button color or copy on a landing page, test against a new layout, value prop or offer.

As an example, we gave our sales team three very different offers to use as closing tools at the point of sale. One offer closed 98 sales with the next closest being single digits. By setting up the test with big differences in variables, we were able to “force” significance from relatively small data size to inform our next marketing campaign.

Dave Gerhardt of Drift summed it up nicely:

“Every time we take a big swing, something good tends to happen.”

Use qualitative data to make decisions

You don’t have to just rely on low volume quantitative data to make decisions. Qualitative data is a every bit as valuable and can round out your decision making. Qualitative data is great by itself too.

Here’s a great article by ConversionXL that goes into detail about getting good qualitative data to work with. In it they go through all the nuts and bolts from choosing the type of survey through analyzing the results.

But the main takeaway is that “soft data” can bring great insight into prospects, customers and former customers that can unlock areas of growth.

It’s also not limited to customer happiness surveys. The team at Price Intelligently shows that getting qualitative data to measure price sensitivity is key to understand how much people are willing to pay for a service. This insight can be especially critical when coming up with a pricing strategy that emphasizes value over low price.

Use other people’s data

Hopefully, you are already doing research on competitors and have a decent swipe file of what other people are doing. There are a lot of smart people out there, it would be inefficient to not build on what they are doing.

Reading case studies, interviews, and whitepapers are all minimum things you should be doing. If not, you can start with these resources I put on my Growth List.

But being able to see what competitors are doing in market can help inform your decision as well.

For example, the McRib is McDonald’s go-to menu item when they want to boost sales. They constantly go back to it because it’s a successful promotion.

What’s the “McRib” in your industry?

Look at competitor landing pages, Adwords ads, and offers to see what’s worked or not. If a competitor consistently brings back a promotion or offer, then that’s worth noting. What is it about the offer that’s working? Price? Value statement? Call to action?

This is especially true for the market leaders. Chances are they have already spent a lot of money on figuring out what works, so it only makes sense to get a headstart by learning from them.

All that being said…

It’s best to use as much data as you can to inform your decisions, with enough data points to have confidence in the outcomes. But if you don’t, that doesn’t mean you can’t use what you have to make as informed a decision as possible. Sometimes directional indicators is enough to set you on a path for growth.