Pricing Optimization using A/B Testing Part I: Data Validation

Dataset used:

Project II Blog Data Set Used.xlsx

Download XLSX • 15.93MB

GOAL: I will be evaluating whether a price change from $49 to $69 for a popular tax-preparation software, running on a website, has been successful. The focus will be on user segmentation and providing insights on segments of customers who behave differently.

1. Data Validation with Users in Test and Control Groups

FYI: In the experiment 66% of the users have seen the old price ($49), while 33% of the users will see a higher price ($69).

Knowing the information above, we want to make sure that our users saw the correct price. By doing so we can insure that our data is correctly captured and can move on to the analysis stage.

From our Pivot Table we can see that 76 of users were in the wrong group. This will skew our results if the information is not fixed, so we'll need to clean this data in order to move forward.

After cleaning our data, we can see that all of our users are now correctly captured in either the control or test group.

2. Analyze if a Bias Exists in the Data

Now that we have our data cleaned in our control and test groups, we want to make sure that the groups are evenly distributed (proportion-wise) so we can make sure equal emphasis was put on both groups. A good way to do this would be to subdivide our control and test group data into device type (web or mobile devices).

From this pivot table (above) we can see that the proportions of web and mobile users in each group are relatively the same so no bias can be concluded from this table. In this case, I'd dig a little deeper and subcategorize our groups into user source to see if a bias exists there.

CONCLUSION: After further subcategorization, we can conclude that our groups are evenly split in relation to each other, and that no bias is evident. Now that our data has been cleaned and validated, we can move onto the next step in this project: Running the A/B test to see which price will result in more revenue for the company.

Coming soon...User conversion rates and price change testing using A/B Testing.