Control groups let you compare visitors who see personalized content with visitors who don’t. By holding back content from a small group of visitors, you can measure the true impact of your triggers, web content, and email content, and make confident, data‑driven decisions.
The system areas related to the control group are:
Control Groups Settings
Expand the User menu and go to Settings > System Configuration > Control Groups.
Control Groups Report
Go to Reports > All Reports > System Data > Control Groups.
Overview
When the feature is enabled, a small percentage of visitors are assigned to a control group. Once a visitor is assigned to a control group, they remain in that group unless the group size changes or you manually move them in Reports > Visitors section. A visitor can only be in one control group at a time.
You can adjust the size of control groups in Control Group Settings. When the size of the group changes, some visitors are moved either in or out of that group to maintain the correct size. After you change the group size, it may take up to seven days for the Control group report to reflect the change.
Triggers control group
When enabled, customers in the Triggers control group don’t receive triggered messages. This includes Cart Abandon, Browse Abandon, Purchase Complete, Price Drop, and Back in Stock emails.
Content control groups
Customers who are in either the Web Content or Email Content control group don’t see any Slots of that type, and instead are shown an empty SmartBlock.
This is determined by the Slot type, not by how the Slot is used.
You can disable this behavior for individual Slots by selecting the Show Content to People in the Control Group option on the Slot's Edit page. This allows all customers to see the Slot, even if they’re in a control group.
Reporting
This section explains how control group results are measured and reported.
Attribution
Unlike the standard attribution system, control group results are based on what a visitor has viewed, not what they clicked. This is necessary because customers in a control group never see Slots, so they can’t click them and always have zero attributions for that content.
The Control group report shows the improvement in RPM (revenue per mille) by comparing visitors who saw content or received triggered emails and made a purchase with visitors in the control group. The values on the control group summary report are based on the last 30 days of data and are updated in real-time.
Significance
On the Control group report each value is accompanied by a Significance indicator.
The significance indicators show if the two numbers are considered statistically different, beyond what is considered normal variation or fluctuations. A tick indicates that the difference between the control group and main group is statistically significant.
If the Significance achieved shows a cross, then you need to either wait longer, or check through the Troubleshooting points below.
Significance tests are impacted by the volume of data, the consistency of the data, and the extremities and fluctuations within the data. High data volumes, with a consistent difference between the groups is more likely to be significant compared to lower volumes, with high variability between the groups.
Hover over the icon to see the result of the test as a percentage value. 5% or below is considered significant, so you can see if a test was close, or not, to being considered significant.
Significance calculation
The Significance is based on the calculated p-value. This value is the probability that when the null hypothesis is true - meaning there is no difference in outcome between the control group and normal group - the result would be greater than or equal to the observed results.
A p-value less than 5% is considered high confidence and a value less than 10% is reported as a medium confidence. To see the raw p-value and sample size, hover over the Significance indicator on the Control Group report. The p-value for the RPM and average order value is calculated using a 2-tailed t-test. The Conversion rate's p-value is calculated using a z-test.
Result calculation
Result | Description |
RPM | Revenue generated per 1000 sends/sessions/impressions. |
Click Through Rate | The percentage of people making a purchase in each group. |
Average Order Value | The average value of orders made by people in each group. |
Extra Revenue | The extra revenue from each type of content. This is calculated by comparing the performance of the control group vs everyone who saw the content. |
Extra Orders | The number of extra orders from each type of content. This is calculated by comparing the performance of the control group vs everyone who saw the content. |
Detailed calculation breakdown
Detailed calculation breakdown
Below is a detailed explanation of how each value is calculated. All the data used is provided on the Control group report so you can do the calculations yourself to check the results if you want to.
Trigger Average Order Value
Divide the total revenue made by each group by the number of orders.AOV = Total Revenue / Number of Orders
Trigger Conversion rate
Divide the number of orders made by each group by the number of emails sent. Multiply by 100 to convert to a percentage.Conversion Rate = (Number of Orders / Number of Emails sent) * 100
Trigger RPM
Divide the total value of all purchases made by each group by the number of emails sent. Then multiply by 1000 to get the value for every thousand emails sent.RPM = (Total Revenue / Number of Emails sent) * 1000
Trigger Extra Revenue
Multiply the control group RPM value by the total number of emails sent in the main group. This is the total revenue that the main group would have produced if they saw no content. The difference between this and the actual revenue is the amount of extra revenue generated by the Triggers.Expected Revenue if in Control Group = (Control Group RPM / 1000) * Number of Emails sent from Main GroupExtra Revenue = Revenue from Main Group - Expected Revenue if in Control Group
Trigger Extra Orders
Multiply the control group conversion rate by the total number of emails sent in the main group. This is the total revenue that the main group would have produced if they saw no content. The difference between this and the actual number of orders is the number of extra orders generated by the Triggers.Expected Number of Orders if in Control Group = (Control Group Conversion Rate / 100) * Total Number of Emails sent from Main GroupExtra Orders = Number of Orders from Main Group - Expected Number of Orders if in Control Group
Web Content Average Order Value
Divide the total revenue made by each group by the number of orders.AOV = Total Revenue / Number of Orders
Web Content Conversion rate
Divide the number of orders made by each group by the number of Sessions. Multiply by 100 to convert to a percentage.Conversion Rate = (Number of Orders / Number of Sessions) * 100
Web Content RPM
Divide the total value of all purchases made by each group by the number of sessions. Then multiply by 1000 to get the value for every thousand sessions.RPM = (Total Revenue / Number of Sessions) * 1000
Web Content Extra Revenue
Multiply the control group RPM value by the total number of sessions in the main group. This is the total revenue that the main group would have produced if they saw no content. The difference between this and the actual revenue is the amount of extra revenue generated by the web content.Expected Revenue if in Control Group = (Control Group RPM / 1000) * Number of Sessions from Main GroupExtra Revenue = Revenue from Main Group - Expected Revenue if in Control Group
Web Content Extra Orders
Multiply the control group conversion rate by the total number of sessions in the main group. This is the total revenue that the main group would have produced if they saw no content. The difference between this and the actual number of orders is the number of extra orders generated by the web content.Expected Number of Orders if in Control Group = (Control Group Conversion Rate / 100) * Total Number of Sessions from Main GroupExtra Orders = Number of Orders from Main Group - Expected Number of Orders if in Control Group
Email Content Average Order Value
Divide the total revenue made by each group by the number of orders.AOV = Total Revenue / Number of Orders
Email Content Conversion rate
Divide the number of orders made by each group by the number of impressions. Multiply by 100 to convert to a percentage.Conversion Rate = (Number of Orders / Number of Impressions) * 100
Email Content RPM
Divide the total value of all purchases made by each group by the number of impressions. Multiply by 1000 to get the value for every thousand impressions.RPM = (Total Revenue / Number of Impressions) * 1000
Email Content Extra Revenue
Multiply the control group RPM value by the total number of impressions in the main group. This is the total revenue that the main group would have produced if they saw no content. The difference between this and the actual revenue is the amount of extra revenue generated by the email content.Expected Revenue if in Control Group = (Control Group RPM / 1000) * Number of Impressions from Main GroupExtra Revenue = Revenue from Main Group - Expected Revenue if in Control Group
Email Content Extra Orders
Multiply the control group conversion rate by the total number of impressions in the main group. This is the total revenue that the main group would have produced if they saw no content. The difference between this and the actual number of orders is the number of extra orders generated by the email content.Expected Number of Orders if in Control Group = (Control Group Conversion Rate / 100) * Total Number of Impressions from Main GroupExtra Orders = Number of Orders from Main Group - Expected Number of Orders if in Control Group
Test control groups
You can see which control group a visitor is in on the Visitors report. Go to Reports > All Reports > Site Activity & Insight > Visitors. Use the search field to locate the visitor and open their record, then expand the Control Groups section in the bottom left.
You can move individual visitors in or out of a control group here. This allows you to test what customers see when in or out of each control group by moving your own visitors record into the different control groups.
Troubleshoot control groups
If the results from control groups don't look right, or aren't achieving significance, check your Slot configurations.
Each Slot has a setting which can force it to be displayed to everyone, even if they are in the control group. This is intended for pieces of content which are essential, and should be shown for everyone. If you want to use control groups, then the Show to Everyone option shouldn't be set for pieces of marketing content which aren't essential for the performance of the site.
You can see the setting for every Slot on the Content > Slots page.
You need to switch to the List view to see this.