Control groups work by creating a small group of shoppers that are not shown advanced personalization content, so that you can compare the outcomes for these visitors with those who did see the content.
The system areas related to the control group feature are:
Control Groups Settings
Expand the User menu and go to Settings > System Configuration > Control Groups.
Control Groups Report
Go to Reports > All Reports > System Data > Control Groups.
How control groups work
When enabled, a small percentage of shoppers are assigned to a control group. Once in a control group, the shopper stays in that control group forever, unless the control group size is changed or they are manually moved out using the Person report. A shopper can only be in one control group at a time.
The size of each control group can be controlled on the Control group settings page. When the size of the group is changed some shoppers are moved either in or out of that group in order to maintain the correct size. After changing the size of the group, it may take 7 days for the change to take full effect on the Control group report.
Triggers control group
When enabled, customers who are in the Triggers control group don’t receive any triggered messages. This includes Cart Abandon, Browse Abandon, Purchase Complete, Price Drop, and Back in Stock emails.
Content control groups
Customers who are in either the Web Content or Email Content control group don’t see any Slots of that type, and instead are shown an empty SmartBlock.
This is based on the type of Slot being shown, not how it is used.
You can disable this behavior for individual Slots by selecting the Show Content to People in the Control Group option on the Slot's Edit page. Doing this means every customer can see the Slot, even if they are in the control group.
Reporting
Attribution
Unlike the standard attribution system, control group results are based on what a shopper has viewed, but not necessarily clicked. This is required because customers in the control group can never click a Slot, as they are not shown, meaning this group always has zero attributions for that content.
The Control group report displays the improvement in RPM (Revenue Per Mille) between shoppers who saw the content or received a triggered email and made a purchase vs those who didn't because they were in the control group. The values on the control group summary report are based on the last 30 days of data and are updated in real-time.
Significance
On the Control group report each value is accompanied by a Significance indicator.
These significance indicators show if the two numbers are considered statistically different, beyond what is considered normal variation or fluctuations. If the Significance achieved shows a tick, then it suggests that the difference between the two numbers can be attributed to the impact of being in the control vs the main group.
If the Significance achieved shows a cross, then you need to either wait longer, or check through the Troubleshooting points below.
Significance tests are impacted by the volume of data, the consistency of the data and the extremities and fluctuations within the data. High data volumes, with a consistent difference between the groups is more likely to be significant compared to lower volumes, with high variability between the groups
Hover over the icon to see the result of the test as a percentage value. 5% or below is considered significant, so you can see if a test was close, or not, to being considered significant.
How the Significance is calculated
The Significance is based on the calculated p-value. This value is the probability that when the null hypothesis is true — meaning there is no difference in outcome between the control group and normal group — the result would be greater than or equal to the observed results.
A p-value less than 5% is considered high confidence and a value less than 10% is reported as a medium confidence. To see the raw p-value and sample size, hover over the Significance indicator on the Control Group report. The p-value for the RPM and average order value is calculated using a 2-tailed t-test. The Conversion rate's p-value is calculated using a z-test.
How the results are calculated
Result | Description |
RPM | Revenue generated per 1000 sends/sessions/impressions. |
Click Through Rate | The percentage of people making a purchase in each group. |
Average Order Value | The average value of orders made by people in each group. |
Extra Revenue | The extra revenue from each type of content. This is calculated by comparing the performance of the control group vs everyone who saw the content. |
Extra Orders | The number of extra orders from each type of content. This is calculated by comparing the performance of the control group vs everyone who saw the content. |
Detailed calculation breakdown
Detailed calculation breakdown
Below is a detailed explanation of how each value is calculated. All the data used is provided on the Control group report so you can do the calculations yourself to check the results if you want to.
Trigger Average Order Value
Take the total revenue made by each group and divide it by the number of orders.AOV = Total Revenue / Number of Orders
Trigger Conversion rate
Take the number of orders made by each group and divide it by the number of emails sent. Multiply by 100 to convert to a percentage.Conversion Rate = (Number of Orders / Number of Emails sent) * 100
Trigger RPM
Take the total value of all purchases made by each group and divide it by the number of emails sent. Then multiply by 1000 to get the value for every thousand emails sent.RPM = (Total Revenue / Number of Emails sent) * 1000
Trigger Extra Revenue
Takes the control group RPM value and multiply this by the total number of emails sent in the main group. This is the total revenue that the main group would have produced if they saw no content. The difference between this and the actual revenue is the amount of extra revenue generated by the Triggers.Expected Revenue if in Control Group = (Control Group RPM / 1000) * Number of Emails sent from Main GroupExtra Revenue = Revenue from Main Group - Expected Revenue if in Control Group
Trigger Extra Orders
Take the control group conversion rate and multiply this by the total number of emails sent in the main group. This is the total revenue that the main group would have produced if they saw no content. The difference between this and the actual number of orders is the number of extra orders generated by the Triggers.Expected Number of Orders if in Control Group = (Control Group Conversion Rate / 100) * Total Number of Emails sent from Main GroupExtra Orders = Number of Orders from Main Group - Expected Number of Orders if in Control Group
Web Content Average Order Value
Take the total revenue made by each group and divide it by the number of orders.AOV = Total Revenue / Number of Orders
Web Content Conversion rate
Take the number of orders made by each group and divide it by the number of Sessions. Multiply by 100 to convert to a percentage.Conversion Rate = (Number of Orders / Number of Sessions) * 100
Web Content RPM
Take the total value of all purchases made by each group and divide it by the number of sessions. Then multiply by 1000 to get the value for every thousand sessions.RPM = (Total Revenue / Number of Sessions) * 1000
Web Content Extra Revenue
Takes the control group RPM value and multiply this by the total number of sessions in the main group. This is the total revenue that the main group would have produced if they saw no content. The difference between this and the actual revenue is the amount of extra revenue generated by the web content.Expected Revenue if in Control Group = (Control Group RPM / 1000) * Number of Sessions from Main GroupExtra Revenue = Revenue from Main Group - Expected Revenue if in Control Group
Web Content Extra Orders
Takes the control group conversion rate and multiply this by the total number of sessions in the main group. This is the total revenue that the main group would have produced if they saw no content. The difference between this and the actual number of orders is the number of extra orders generated by the web content.Expected Number of Orders if in Control Group = (Control Group Conversion Rate / 100) * Total Number of Sessions from Main GroupExtra Orders = Number of Orders from Main Group - Expected Number of Orders if in Control Group
Email Content Average Order Value
Take the total revenue made by each group and divide it by the number of orders.AOV = Total Revenue / Number of Orders
Email Content Conversion rate
Take the number of orders made by each group and divide it by the number of impressions. Multiply by 100 to convert to a percentage.Conversion Rate = (Number of Orders / Number of Impressions) * 100
Email Content RPM
Take the total value of all purchases made by each group and divide it by the number of impressions. Then multiply by 1000 to get the value for every thousand impressions.RPM = (Total Revenue / Number of Impressions) * 1000
Email Content Extra Revenue
Take the control group RPM value and multiply this by the total number of impressions in the main group. This is the total revenue that the main group would have produced if they saw no content. The difference between this and the actual revenue is the amount of extra revenue generated by the email content.Expected Revenue if in Control Group = (Control Group RPM / 1000) * Number of Impressions from Main GroupExtra Revenue = Revenue from Main Group - Expected Revenue if in Control Group
Email Content Extra Orders
Take the control group conversion rate and multiply this by the total number of impressions in the main group. This is the total revenue that the main group would have produced if they saw no content. The difference between this and the actual number of orders is the number of extra orders generated by the email content.Expected Number of Orders if in Control Group = (Control Group Conversion Rate / 100) * Total Number of Impressions from Main GroupExtra Orders = Number of Orders from Main Group - Expected Number of Orders if in Control Group
Testing
You can see which control group a shopper is in on the Visitors report. To do this, go to Reports > All Reports > Site Activity & Insight > Visitors. Use the search field to locate the visitor and open their record, then expand the Control Groups section in the bottom left.
You can move individual shoppers in or out of a control group here. This allows you to test what customers see when in or out of each control group by moving your own shopper record into the different control groups.
Troubleshoot control groups
If the results from control groups don't look right, or aren't achieving significance, you should check your Slot configurations.
Each Slot has a setting which can force it to be displayed to everyone, even if they are in the control group. This is intended for pieces of content which are essential, and should be shown for everyone. If you want to use control groups, then the Show to Everyone option shouldn't be set for pieces of marketing content which aren't essential for the performance of the site.
You can see the setting for every Slot on the Content > Slots page.
You need to switch to the List view to see this.