Incrementality Testing

Incrementality Testing

Incrementality testing is a set of statistical methods that help measure the impact of a marketing campaign or activity. RevSure supports the following Incrementality Testing methods and widgets.

1. Conversion Lift Analysis

2. Pre-Post Analysis

3. Multi-variate Causal Impact Analysis

4. Campaign Lift Analysis - Across Campaigns

5. Campaign Lift Analysis - Individual Campaigns

These widgets are available under the Customize side panel of the Marketing Performance Module.

Conversion Lift Analysis Widget

This widget helps measure the statistical validity of conversion differences between two groups:

• Control (where a particular marketing / GTM activity has not been performed)

• Test (for which a particular marketing/GTM activity has been performed)

In RevSure, you can identify Control vs. Test based on Boolean (true/false) or Binary Categorical (Y/N) dimension, such as (Y/N). Examples of such dimensions include Lead Google Touched, Lead LinkedIn Touched, etc.

The statistical test is done using a Chi-squared T test for proportions.

Setting up the Conversion Lift Analysis Widget

• Select Filters and the (Created) Period as appropriate

• Select Progressed Period for the Conversion Metric

• Select Split By Dimension that specifies the Test and Control Groups

• Select View by Stage for the Start Funnel Stage of Conversion

Once the above selections are done

• Select the To funnel stage to compute the conversion, and

• Select the value from the Split by Dimension

Based on the above selections, the Conversion Lift Analysis test runs dynamically. The widget also gives you the value of the conversions for each value, as well as an estimate of the Lift (difference in the conversion between the test and the control group) and whether the Conversion Lift is statistically significant or not.

The statistical significance is indicated by the p-value of the test.

• If the p-value < 0.05, then the Lift is statistically significant. This means, the inference can be made that the marketing/GTM activity made a statistically significant impact on the conversions.

• Else if the p-value > 0.05 the the Lift is statistically insignificant. This means that no inference can be made on whether the marketing/GTM activity made an impact on the conversion. Any lift or difference observed could just be a random observation.

Pre & Post Analysis Widget

In RevSure, the ‘Pre & Post Analysis’ widget in the Marketing Performance module is designed to help with testing the impact of a marketing/GTM Treatment on a Response Variable.

• The Treatment could be running a new campaign, an Event, a competitor action, increasing digital spend, etc.

• The Response could be an outcome such as pipeline generation, lead generation, or booking value generation. etc.

This analysis is designed to statistically compare the difference in the Response variable before the treatment (pre) and the Response variable (post) the treatment.

The statistical method used in this test is the 2-sample Welch t-test.

Setting up the Pre-Post Analysis Widget

• Select Filters, Date Filter Period for the Period Filtering, and whether to Consider only First Movements

• Select Pre-Time Period (this is the time period before the treatment)

• Select Post-Time Period (this is the time period before the treatment)

• Select Treatment Metric/Variable from the drop-down

• Select Response Metric/Variable from the drop-down

• Select Test Type

  • Response (Uses the Response Variable as is for the pre-post comparison)

  • Ratio (Uses the ratio of Response/Treatment for the pre-post comparison)

• Select Sample Granularity (Weekly/Monthly/Quarterly) for analysis of the time trend and averages of the Response and Treatment Metric

Output

• Chart and Table of the Averages of the Response and Treatment Metric for the Pre and Post time period, respectively

• The average of the respective differences of the Response and Treatment Metric between the Post and the Pre time periods

• The total respective differences of the Response and Treatment Metric between the Post and Pre time periods

• Summary of the Outcome: Difference in Response Metric/Difference in Treatment Metric

• Statistical Significance of the Pre-Post Test:

  • The statistical significance is indicated by the p-value of the test.

    • If the p-value < 0.05, then the Difference and Outcome are statistically significant. This means the inference can be made that the Treatment made a statistically significant impact on the Response.

    • Else if the p-value > 0.05, the measured Difference and Outcome is statistically insignificant. This means that no inference can be made on whether the Treatment made an impact on the Response. Any difference observed/measured could just be a random observation.

Multi-variate Causal Impact Analysis

This version is an advanced version of the Pre-Post Analysis, which includes a more rigorous causal impact analysis along with the ability to analyze the impact of marketing and GTM interventions that might involve multiple levers/tactics/campaigns.

The methodology is inspired by Google’s Causal Impact approach and data science methodology: https://google.github.io/CausalImpact/CausalImpact.html

Setting up the Multi-variate Causal Impact Analysis widget

• Select Filters, Date Filter Period for the Period Filtering, and whether to Consider only First Movements

• Edit Configuration

  

  • This widget provides the ability to run multiple configurations of the Causal Impact Analysis

  • One can use configurations to test different interventions, different response variables, and different periods

  • Each configuration has a combination of selections

    • Select Pre-Time Period (this is the time period before the treatment)

    • Select Intervention Time Period (this is the time period of the intervention)

    • Select Post-Time Period (this is the time period after the intervention)

    • Select Treatment Metrics/Variables from the drop-down (these are marketing / GTM interventions that have been done in the hope of driving a particular outcome in the Response)

    • Select Response Metric/Variable from the drop-down

    • Select Covariate Metrics/Variables from the drop-down (these are metrics/variables that are not part of the marketing/GTM intervention but impact the response. These could be variables like macroeconomics, weather conditions, etc.)

    • Select Sample Granularity (Weekly/Monthly/Quarterly) for analysis of the time trend and averages of the Response and Treatment Metric

• The configuration can take a few minutes to run

Output

• Once the configuration runs, the test results are updated on the widget

• The Widget shows the outputs across three tabs

  • Response Variable Tab (image above)

    • Trend Chart

      • Shows the trend of the Response Variable across the Pre, Intervention, and Post Period

      • In the Post Period region, the dotted line shows the expected Response (Without Treatment Forecast) if no intervention had been done

        • Key Test Metrics are summarized

          • • Cumulative Effect: This is the sum of the differences between the Actual Response (With Treatment) and the Expected Response (Without Treatment Forecast) during the Post time period.

            • % Relative Effect: This is the % of the Cumulative Effect divided by the sum of the Expected Response (Without Treatment Forecast) during the Post Time Period

            • Together, the Cumulative Effect and Relative Effect % summarize the magnitude of the outcome

        • Further, the test summarizes the Statistical Significance of the Impact/Outcome

    • Relative Difference Chart

      • The Relative Difference chart shows the relative difference (difference between the Actual Response (With Treatment) and the Expected Response (Without Treatment Forecast)) for each time bucket during the Post time period. This relative effect is indicated by the red and green bars

      • Further, the chart shows the rolling cumulative effect for each bucket during the Post time period. This is indicated by the dotted line chart

  • Treatment Tab

    

    • The Treatment tab shows the trend charts of the different treatment variables across the Pre, Intervention, and Post Period, respectively

  • Covariates Variable Tab

    • The Covariates Variable tab shows the trend charts of the different covariate variables across the Pre, Intervention, and Post Period, respectively

Campaign Lift Analysis - Across Campaigns

The Campaign Lift Analysis widget helps you measure the impact of specific campaigns, campaign types, or channels on conversion rates across your funnel. It uses statistical significance to determine whether a campaign meaningfully contributes to conversion from one stage to another.

This analysis compares the conversion lift from a selected campaign against others using a Chi-Squared test. It helps answer:

• Does this campaign/channel/type touch improve conversion likelihood?

• Is the improvement statistically significant?

• Where in the funnel is the lift most visible?

To use the campaign lift analysis,

Set Your Filters:

• Created & Progressed Periods: Select the time range (e.g., previous 365 days).

• Stages: Choose the stage pair (e.g., Lead → MQL or MQL → Pipeline).

• Touch Type: Select from First Touch, Last Touch, or Any Touch.

• Dimension: Pick the level of comparison – Campaign Type, Campaign Channel, Campaign Name etc.,

• Contacts Considered: For lead to opportunity stage conversion, which contacts to consider

  • All Contacts in Account

  • All Contacts in Opportunity

  • Primary Contact in Opportunity

• Base Value: Choose whether lift is measured against:

  • Average of all campaigns

  • Average of selected campaigns

  • A specific campaign

Understand the Output:

• Conversion %: Percentage of leads/accounts with a touch of that campaign/channel/type in the journey from the start stage to the end stage that converted to the end stage.

• Lift %: How much higher (or lower) the conversion % is compared to the base.

• Statistical Significance:

  • If a campaign/channel shows a Significant Lift, it’s backed by a p-value < 0.05, indicating high confidence.

  • Example: Marketing Form-Fill shows +1903.57% Lift with a strong p-value – it’s first touch significantly boosts conversion (72.73%) from Lead to MQL against all other campaigns (3.63%)

Campaign Lift Analysis - Individual Campaigns

The Campaign Lift Analysis - Individual Campaigns widget uses the same methodology as the Campaign Lift Analysis - Across Campaigns widget. The difference lies in how the lift is calculated

• Campaign Lift Analysis - Individual Campaigns compares lift with the same campaign. It analyses whether having the touch of the campaign is more beneficial than not having the touch of the campaign. For example, In case of first touch, it analyzes whether an MQL with the first touch of Marketing Form-Fill has a higher chance of conversion than not having the first touch of Marketing Form-Fill

• Campaign Lift Analysis - Across Campaigns compares lift with other campaigns. It analyses whether having a touch of the campaign is more beneficial than any other campaign. For example, in the case of first touch, it analyzes whether leads with the first touch of Marketing Form-Fill have a higher chance than any other campaign.

  Note: You will notice that the conversion% in Across Campaigns is the same as the True value in Individual Campaigns

  

  Control Value - Refers to which value you want to compute the lift for. Since we would like to compute the lift of true over false in most cases, the control value remains False, and the test value remains True.

Calculation Logics of Campaign Lift Analysis

Individual Campaigns

1. Calculating the conversion % of a campaign touch

   1. Calculate the Number of leads with a campaign touch that converted - X

   2. Calculate the Number of leads with a campaign touch that did not convert - Y

   3. Calculate the Number of leads without a campaign touch that converted - X'

   4. Calculate the Number of leads without a campaign touch that did not convert - Y'

   5. Conversion % of True = X/(X+Y)

   6. Conversion % of False = X'/(X'+Y')

2. Calculating the lift

   1. If Control Value is False, (Conversion of True - Conversion of False)/Conversion of False

   2. If Control Value is True, (Conversion of False - Conversion of True)/Conversion of True

Across Campaigns

1. Calculating the conversion % of a campaign

   1. Calculate the Number of leads with a campaign touch that converted - X

   2. Calculate the Number of leads with a campaign touch that did not convert - Y

   3. Conversion % of a campaign = X/(X+Y)

2. Calculating the conversion % of all campaigns

   1. Calculate the Number of leads with any of the campaigns touched that converted - X

   2. Calculate the Number of leads with any of the campaigns touched that did not convert - Y

   3. Conversion % = X / (X+Y)

3. Calculating the conversion % of selected campaigns

   1. Calculate the Number of leads with any of the selected campaigns touched that converted - X

   2. Calculate the Number of leads with any of the selected campaigns touched that did not convert - Y

   3. Conversion % = X/(X+Y)

4. Lift is calculated,

   1. When comparing against a base value of the average of all campaigns, (Conversion of a campaign- Conversion of all other campaigns)/(Conversion of all other campaigns)

   2. When comparing against a base value of the average of selected campaigns, (Conversion of a campaign- Conversion of selected other campaigns)/(Conversion of selected other campaigns)

   3. When comparing against a base value of a campaign, (Conversion of a campaign- Conversion of base campaign)/(Conversion of base campaign)