Statistical significance in Email Marketing, an interview with Marcin Luks, our Marketing Director

[vc_row][vc_column][vc_column_text][/vc_column_text][/vc_column][/vc_row][vc_row][vc_column][boc_heading html_element=”h4″ color=”#333333″ margin_top=”-45px” margin_bottom=”30px”]Expert Expressions[/boc_heading][vc_single_image alignment=”center” image=”1186″][vc_column_text]Today, Adam Ambrożewicz took the time to speak to Marcin Luks, ExpertSender’s Marketing Director about statistical significance and the future of mathematics in the Email Marketing industry.

AA: Marcin, could you tell us a little bit more about split testing in the Email Marketing industry?

ML: Split tests are one of the most important tools in the hands of email marketers and one of the key things that email marketers do is test, test and test some more. Split testing is a key functionality set of every Email Service Provider, because it allows you to check different versions of your email’s content, the headers, such as the subject line or the date when the message will be sent.

A very important aspect of split testing is the ability to automatically choose the winning version and have it sent to the rest of your list when the split tests have finished running.

AA: Many ESP’s don’t actually have statistical significance analysis of split tests and ExpertSender decided to implement this functionality. What were the reasons why you decided to introduce this feature to the platform?

ML: There were a couple of reasons, one of the things that sparked the idea was a questionnaire from the Email Marketing Software Buyer’s Guide conducted by Email Vendor Selection where one of the questions they asked was about split test statistical significance. Although we knew what split test statistical significance was, we knew how to calculate it, at the time our platform didn’t have that particular functionality or algorithms and that obviously sparked an idea, why don’t we add it sometime soon?

We conducted some internal tests and it turned out that a large percentage of the split tests that had been run on our platform throughout the year were statistically insignificant. In other words, the results were not certain. This is one of the weaknesses of split testing with marketing platforms in general and now we’ve significantly improved that.

AA: And you engaged the services of mathematicians to help you create an algorithm to find out whether your tests were statistically significant? Could you tell us a little more about how you collaborated with them?

ML: We started working with our mathematicians and we met every two weeks where we talked about the potential features that we would like to implement and one of the things that they were working on with us from the very beginning was the split test statistical significance algorithm.

First we conducted different tests, to try and find the right algorithm because there’s a lot of algorithms and they vary a lot, so we needed to find a sweet spot and that’s what we did. One of the most challenging things was not only showing a text based result of the split test, but also trying to visualise it and graphically represent why a certain split test was statistically significant or certain.

The solution to that problem was provided to us by one of our mathematicians when he drew an axis with confidence intervals explaining the mathematics behind it and we thought you know what, that’s a great idea! Maybe that’s how we should present it to our clients.

AA: A lot of mathematics is difficult to understand for most people, maybe they studied it a school or university but later they forgot about it. Did you have to go back to school to remind yourself a few things?

ML: No (laughing), I did not need to go back to school. To be honest, I am all for simplifying what’s complex and advanced and using advanced technology or advanced mathematics and showing it in a simple way.

AA: So every marketer can actually understand it without understanding the mathematics behind it?

ML: Yes, you don’t need to know the exact algorithm behind it, but every email marketer using this feature needs to quickly understand what advantage they’ll gain from it. From the very beginning, my focus was on trying to translate something mathematically complex into something that would be easy to understand, something tangible and something that could be widely used by our clients.

AA: How do you expect this feature to change future marketing decisions?

ML: Most importantly the level of uncertainty in their decision making will be much lower. I expect our clients to start thinking more about how they perform their split tests, because conducting a split test and selecting the winning version based on the aggregate number of opens in the split test versions is just not enough. If we educate our clients, we believe that this will have a great impact on their campaign performance because they’ll be optimising their campaigns more consciously and on the basis of suggestions backed up by mathematics.

AA: Indeed, one of the strengths of the ExpertSender platform is that it automatically gives you different suggestions on what you could improve in future split tests. Could you elaborate a little bit more on some of the suggestions that a marketer might receive?

ML: Well, if a split test finishes with a certain result then that’s great.

AA: That means that your result makes sense and you should go with it?

ML: Yes, but sometimes, the winning version performs so much better than the other versions that we thought, hey, apart from just telling our marketer that their test result is certain, why not give them some advice so that they can optimise their future emails?

Even if the first result is certain, don’t just give them a binary answer whether the result is statistically significant or not, in 90% of the cases we also provide suggestions on what should be done next to improve future tests. I mean, if it was insignificant what should you do? Should you increase the size of the test group? Or remove one of the versions? There’s always some actionable advice at the end of each split test result.

AA: We hear a lot about big data in just about every industry and particularly in the marketing industry where a lot of decisions are based on data. What’s more interesting in my opinion however, is how we make use of that data and leverage it with mathematics. We’ve been talking about various algorithms and the power of mathematics, do you see this trend continuing and how do you see mathematics shaping the marketing industry?

ML: To be honest, a lot of new features and new technologies are based on advanced mathematics which can be used to the advantage of the marketer. I’m talking about algorithms that for example, match personalised content on the basis of the end user’s behaviour, whether that’s email, display or any other channel. I see this trend continuing to grow and I can tell you that we’ll continue our collaboration with our mathematicians because there’s many projects that they’re already involved in.

One thing I can share is that we’re not done with improving split tests yet. I don’t want to delve into the specifics but it is truly going to be a new chapter in intelligent split testing.

AA: Thanks for your thoughts Marcin, it was a pleasure.

Read more about the statistical significance feature here:

http://www.emailvendorselection.com/featurewatch/expertsender-statistical-significance-testing/

[/vc_column_text][/vc_column][/vc_row]