How to Answer Big Questions with Simple Experiments

Millions of years ago, humans were weak and vulnerable creatures.

Compared to other mammals, we weren't strong...

We weren't fast...

We didn't have sharp teeth...

We couldn't fly...

So how did we not only survive but make it out of the food chain?

In simple terms…science and politics.

The Scientist and the Politician

Lurking beneath the surface of each of us is a scientist.

This part of us succeeds by doing good. It is curious, creative, and creates tools to help our community thrive.

There is also a politician.

This part of us succeeds by looking good. It is composed, cooperative, and tells the community what they want to hear.

The scientist wants to ask questions, the politician wants to pretend it has all the answers.

To see which one is more useful to both us as individuals and as a society, just look at where science and politics are today.

The former is getting us to Mars; the latter is making Mars look appealing.

To train my inner scientist, achieve better results, and defeat my inner politician, I designed simple experiments by answering 6 simple questions from "Get Things Automated."

1. What Question Are You Trying to Answer?

The first step in designing an experiment is to state the question you are trying to answer.

A question on my mind recently has been, “What is the best way to spend my time?”

There are many processes in place to run Educo: marketing, sales, product development, operations, etc.

With so much on my plate, it's difficult to know how to allocate my time to produce the best results for customers, for experts, and for members of my team.

Your inner scientist has significant questions about the world – embrace them. Don’t take the easy path by pretending you have the answers. Instead, write down “What precisely do I want to know?“

2. What Is Your Hypothesis?

To answer this question, you must come up with a clear and testable hypothesis.

I wanted to learn the most valuable use of my time, so I needed an educated, testable prediction of what that activity might be.

My hypothesis was that automation was the best use of my time.

Every hour invested in automation leads to more free time in the future. So it seemed logical that time spent automating our marketing process would be more valuable than time spent producing new marketing content.

In designing your experiments, be clear and specific about your hypotheses. Make sure that you can test them.

3. That Hypothesis Will Fail If...

Here's where many of us make mistakes when it comes to designing experiments.

You shouldn't design an experiment to prove yourself right – rather, you should design an experiment to try to prove yourself wrong.

To see if automation was a more valuable use of my time, I needed to set a clear benchmark for failure.

The key performance indicator for our marketing content is Click-thru Rates.

So, I needed to test to see if 1 hour invested in marketing would yield a better Click-thru Rate than 1 hour spent on automation.

When designing your experiment, be sure that there is a clear way for you to fail. And try to make it happen as objectively as possible.

If your hypothesis survives the attempt at failure, you have more reason to believe in it.

4. What Specific Steps Do You Need to Take to Test That Hypothesis?

Now that you have a clear benchmark for failure, it's time to create the experiment.

To test whether marketing would lead to a better Click-thru Rate than automation, I split my time between the two of those both within the marketing realm.

For example, I spent half my time creating content like this which can't be automated.

I spent the other half of my time focusing on automated marketing activities like recommending already published content based on subscriber interests.

By isolating those variables, I could get a better understanding of which activity was more valuable.

When designing your experiment, isolate the variables as much as you can. And do the little things to get the details right.

5. When Will You Assess the Results?

To get a clear understanding, you need to have enough data to make a meaningful interpretation.

To do this, you could check the statistical significance through a service like A/B calculator.

But always remember that you don't need to be perfect in answering your questions through experiments.

I set my deadline for a month to test these two things against each other because that would give me enough time to get meaningful results.

When testing the length of your experiment, remember that you don't need to be perfect. As long as you can be 90-100% sure, it may be worth it to shorten the length of your experiment to move onto the next one.

Conclusion

Here are the results of my experiment…

As I suspected, automation yielded much higher results than manual actions. This resulted in a higher-than-average Click-thru Rate.

That is not to say that doing things like creating new content is not necessary. You have to keep things fresh. However, it should take a lower priority than automation.

I have created a lot of content over the years. And it is a better use of my time to get the content already published to people who could use it to improve their lives than it is to keep sending them a new post that may or may not be relevant to them.

However, this was just one experiment. I still need to test automation against sales, product development, and many other areas to feel that it is indeed the top priority in the business.

But at least I'm one step closer. And that's what science is all about.