How Product Managers Test Experiments Part II
In my last article, I introduced the theoretical background needed to understand how experiments are tested statistically. If you are yet to read Part 1 and don’t have basic Hypothesis Testing knowledge, kindly check it out here.
As stated in my previous article, we shall be testing experiments with a few use cases. In this Part II, we shall be performing A/B testing for Social Media Ad Campaign. Shall we?
Outline
- Introduction
- Objectives
- Statement of Hypothesis
- Analysis
- Conclusion
Disclaimer: All data points used in this article are simulated and do not represent a real-world scenario. Hence, results drawn from this analysis are not useful for any studies.
Objectives
The broad objective of this analysis is to unveil the Ad creative with higher performance and thereby recommend how the company should run its Ads. This can be broken down into smaller objectives as below:
- Explore the parameters (Clicks, Impressions, Click Through Rates(CTR)) across the Ad creatives
- Compare the performance of Ad Creative A and Ad Creative B
Definitions of Terms
- Clicks: This is the number of clicks done by users per day for respective Ads
- Impressions: This is the total number of users that got to see the respective Ads
- Click Through Rates (CTR): This is the number of clicks done by users divided by the number of impressions per day.
Notes: This experiment was conducted (simulation) for a period of 100 days and each day represented one observation.
Statement of Hypothesis
In the objectives stated above, the first objective is basic exploratory data analysis of the data and the second one aims to compare the Ads with inferential statistics. Therefore, we are going to set a hypothesis for the second objective only.
Note: We do not set a hypothesis for descriptive analysis, but rather for inferential analysis.
- Ho: Ad A = Ad B; there is no significant difference between Ad A and Ad B
- Hi: Ad A != Ad B; there is a significant difference between Ad A and Ad B
Analysis
Descriptive Analysis
This type of analysis involves the use of charts, graphs, and some summarization functions (mean, median, mode, quartiles, percentiles, variance, standard deviations, etc) to have an understanding of how the data points are distributed. The results of descriptive analysis guide us to ask the right questions and most times lead to the type of experiments to be tested with inferential statistics.
For the purpose of this analysis, we shall be limiting ourselves to the use of basic summary statistics and line plots.
Inferential Analysis
For the purpose of this experiment, we shall use an independent T-test to access if there is a significant difference in the Clicks, Impressions, and CTR between Ad A and Ad B.
As shown in the table below, the number of Clicks, Impressions, and CTR are significantly different with t-stat (p-value) of 3.244(0.001), -2.318(0.022), and -6.033(0.000) at a 5% level of significance.
| | T-Stat (P-Value)|
|--------------|-----------------|
| Clicks | 3.244(0.001) |
| Impressions |-2.318(0.022) |
| CTR |-6.033(0.000) |Conclusion
A closer look at the Clicks and Impressions line plots is saying that Ad B is better than Ad B. However, this is not true and can be seen when CTR (Clicks/Impressions) is used. Though, Ad B has more Clicks and Impressions, Ad A is relatively better when Clicks per Impression(CTR) is used to measure performance.
Therefore, we can conclude that Ad A performs better than Ad B and this is drawn from the CTR line plots above.
In this article, we explored A/B testing for social media ad campaigns, comparing Ad A and Ad B to uncover the significance of Click Through Rates (CTR) in determining true performance differences. In the next article, we’ll dive into the power of Analysis of Variance (ANOVA) and how it enables data-driven decisions by comparing multiple experiments at once. Don’t miss the opportunity to elevate your strategies and products through innovative product analytical methodologies.
Click “Follow” now for updates and be part of this exciting journey!
