Incrementality Testing

Incrementality testing is a set of statistical methods that help measure the impact of a marketing campaign or activity. RevSure supports the following Incrementality Testing methods and widgets.

1. Conversion Lift Analysis

2. Pre-Post Analysis

3. Multi-variate Causal Impact Analysis

4. Campaign Lift Analysis - Across Campaigns

5. Campaign Lift Analysis - Individual Campaigns

These widgets are available under the Customize side panel of the Marketing Performance Module.

Conversion Lift Analysis Widget

This widget helps measure the statistical validity of conversion differences between two groups:

• Control (where a particular marketing / GTM activity has not been performed)

• Test (for which a particular marketing/GTM activity has been performed)

In RevSure you can identify Control vs. Test based on Boolean (true/false) or Binary Categorical (Y/N) dimension such as (Y/N). Examples of such dimensions include Lead Google Touched, Lead LinkedIn Touched, etc.

The statistical test is done using a Chi-squared T test for proportions.

Minimum Sample Size Check

Before running the test, we calculate the minimum required sample size per group using a Chi-Square.

Inputs: expected control/test conversion rates, significance level, and desired statistical power.

This ensures that the groups are large enough for the Chi-Square approximation to be valid and that the test has sufficient statistical power.

If the observed sample size is below this threshold, results may be flagged as unreliable.

Setting up the Conversion Lift Analysis Widget

• Select Filters and the (Created) Period as appropriate

• Select Progressed Period for the Conversion Metric

• Select Split By Dimension that specifies the Test and Control Groups

• Select View by Stage for the Start Funnel Stage of Conversion

Once the above selections are done

• Select the To funnel stage to compute the conversion, and

• Select the value from the Split by Dimension

Based on the above selections the Conversion Lift Analysis test runs dynamically. The widget also gives you the value of the conversions for each value as well as an estimate of the Lift (difference in the conversion between the test and the control group) and whether the Conversion Lift is statistically significant or not.

The statistical significance is indicated by the p value of the test.

• If the p-value < 0.05 then the Lift is statistically significant. This means, the inference can be made that the marketing/GTM activity made a statistically significant impact on the conversions.

• Else If the p-value > 0.05 the the Lift is statistically insignificant. This means, that no inference can be made on whether the marketing/GTM activity made an impact on the conversion. Any lift or difference observed could just be a random observation.

Pre & Post Analysis Widget

In RevSure, the ‘Pre & Post Analysis’ widget in the Marketing Performance module is designed to help with testing the impact of a marketing/GTM Treatment on a Response Variable.

• The Treatment could be running a new campaign, an Event, a competitor action, increasing digital spend, etc.

• The Response could be an outcome such as pipeline generation, lead generation, booking value generation. etc.

This analysis is designed to statistically compare the difference in the Response variable before the treatment (pre) and the Response variable (post) the treatment.

The statistical method used in this test is the 2-sample Welch t-test.

Minimum Sample Size Method used: Two-sample one-tailed t-test–based sample size calculation with finite population correction (FPC) method is used. This method also includes calculations of the cohen’s distance, which helps to identify how far two samples lie based on their mean and variance.

One-tailed: because it assumes directionality (e.g., post ratio > pre ratio).

Two-sample: because it compares variance from pre-period vs post-period ratios.

Setting up the Pre-Post Analysis Widget

• Select Filters, Date Filter Period for the Period Filtering and whether to Consider only First Movements

• Select Pre-Time Period (this is the time period before the treatment)

• Select Post-Time Period (this is the time period before the treatment)

• Select Treatment Metric/Variable from the drop down

• Select Response Metric/Variable from the drop down

• Select Test Type

  • Response (Uses the Response Variable as is for the pre-post comparison)

  • Ratio (Uses the ratio of Response/Treatment for the pre-post comparison)

• Select Sample Granularity (Weekly/Monthly/Quarterly) for analysis of the time trend and averages of the Response and Treatment Metric

Output

• Chart and Table of the Averages of the Response and Treatment Metric for the Pre and Post time period respectively

• The average of the respective differences of the Response and Treatment Metric between the Post and the Pre time periods

• The total respective differences of the Response and Treatment Metric between the Post and Pre time periods

• Summary of the Outcome: Difference in Response Metric/Difference in Treatment Metric

• Statistical Significance of the Pre-Post Test:

  • The statistical significance is indicated by the p value of the test.

    • If the p-value < 0.05 then the Difference and Outcome is statistically significant. This means, the inference can be made that the Treatment made a statistically significant impact on the Response.

    • Else If the p-value > 0.05 the measured Difference and Outcome is statistically insignificant. This means, that no inference can be made on whether the Treatment made an impact on the Response. Any difference observed/measured could just be a random observation.

Multi-variate Causal Impact Analysis

This version is an advanced version of the Pre-Post Analysis the includes a more rigorous causal impact analysis along with ability to analyze the impact of marketing and GTM interventions that might involved multiple levers/tactics/campaigns.

The methodology is inspired by Google’s Causal Impact approach and data science methodology: https://google.github.io/CausalImpact/CausalImpact.html
It takes two datasets:

Pre-period: before the treatment/campaign/change.

Post-period: after the treatment/campaign/change.

We pass one response/KPI (target), a list of covariates (variables not affected by the treatment/campaign), and optional treatment fields (kept for reporting).

We fit a time‑series model that learns how the KPI normally moves with several covariate variables. Training on the pre-period prevents the treatment’s or campaign’s effect from leaking into the model.

In the after window, we plug in the variables, and the model forecasts what the KPI (Counterfactual forecast) would have been without the initiative. The forecast includes a confidence band (uncertainty).

Post that the impact is computed:

Compute impact, Impact per day/week = Actual - Counterfactual; we also show cumulative lift and % lift.

We report a Bayesian equivalent of the statistical significance p-value of whether the lift > 0 to convey confidence.

If the p-value < 0.05, then the Impact/Lift is considered significant..

Else if the p-value > 0.05, the Lift/Impact is considered statistically insignificant. This means that no inference can be made on whether the marketing/GTM activity made an impact on the lift.  Any lift or difference observed could just be a random observation.

Setting up the Multi-variate Causal Impact Analysis widget

• Select Filters, Date Filter Period for the Period Filtering and whether to Consider only First Movements

• Edit Configuration

  

  • This widget provides the ability to rub multiple configurations of the Causal Impact Analysis

  • One can use configurations to test different interventions, different response variables and different periods

  • Each configuration has a combination of selections

    • Select Pre-Time Period (this is the time period before the treatment)

    • Select Intervention Time Period (this is the time period of the intervention)

    • Select Post-Time Period (this is the time period after the intervention)

    • Select Treatment Metrics/Variables from the drop down (these are marketing / GTM interventions that have been done in the hope of driving a particular outcome in the Response)

    • Select Response Metric/Variable from the drop down

    • Select Covariate Metrics/Variables from the drop down (these are metrics/variables that are not part of the marketing/GTM intervention but impact the response. These could be variables like macroeconomics, weather conditions, etc.)

    • Select Sample Granularity (Weekly/Monthly/Quarterly) for analysis of the time trend and averages of the Response and Treatment Metric

• The configuration can take a few minutes to run

Output

• Once the configuration runs, the test results are updated on the widget

• The Widget shows the outputs across three tabs

  • Response Variable Tab (image above)

    • Trend Chart

      • Shows the trend of the Response Variable across the Pre, Intervention and Post Period

      • In the Post Period region the dotted line shows the expected Response (Without Treatment Forecast) if no intervention had been done

        • Key Test Metrics are summarized

          • • Cumulative Effect: This is the sum of the differences between the Actual Response (With Treatment) and the Expected Response (Without Treatment Forecast) during the Post time period.

            • % Relative Effect: This is the % of the Cumulative Effect divided by the sum of the Expected Response (Without Treatment Forecast) during the Post Time Period

            • Together the Cumulative Effect and Relative Effect % summarize the magnitude of the outcome

        • Further, the test summarizes the Statistical Significance of the Impact/Outcome

    • Relative Difference Chart

      • The Relative Difference chart shows the relative difference (difference between the Actual Response (With Treatment) and the Expected Response (Without Treatment Forecast)) for each time bucket during the Post time period. This relative effect is indicated by the red and green bars

      • Further the chart shows the rolling cumulative effect for each bucket during the Post time period. This is indicated by the dotted line chart

  • Treatment Tab

    

    • The Treatment tab shows the trend charts of the different treatment variables across the Pre, Intervention and Post Period Respectively

  • Covariates Variable Tab

    • The Covariates Variable tab shows the trend charts of the different covariate variables across the Pre, Intervention and Post Period Respectively

Campaign Lift Analysis - Across Campaigns

The Campaign Lift Analysis widget helps you measure the impact of specific campaigns, campaign types, or channels on conversion rates across your funnel. It uses statistical significance to determine whether a campaign meaningfully contributes to conversion from a given stage to another.

This analysis compares the conversion lift from a selected campaign against others using a Chi-Squared test. It helps answer:

• Does this campaign/channel/type touch improve conversion likelihood?

• Is the improvement statistically significant?

• Where in the funnel is the lift most visible?

To use the campaign lift analysis,

Minimum Sample Size Method used: Two-proportion z-test

Set Your Filters:

• Created & Progressed Periods: Select the time range (e.g., previous 365 days).

• Stages: Choose the stage pair (e.g., Lead → MQL or MQL → Pipeline).

• Touch Type: Select from First Touch, Last Touch, or Any Touch.

• Dimension: Pick the level of comparison – Campaign Type, Campaign Channel, Campaign Name etc.,

• Top 5: Based on selected dimension - Campaign Type, Campaign Channel, Campaign Name - ranked by highest conversion percentage.

• Contacts Considered: For lead to opportunity stage conversion, which contacts to consider

  • All Contacts in Account

  • All Contacts in Opportunity

  • Primary Contact in Opportunity

• Base Value: Choose whether lift is measured against:

  • Average of all campaigns

  • Average of selected campaigns

  • A specific campaign

    

Understand the Output:

• Conversion %: Percentage of leads/accounts with a touch of that campaign/channel/type in the journey from the start stage to the end stage that converted to the end stage.

• Lift %: How much higher (or lower) the conversion % is compared to the base.

• Statistical Significance:

  • If a campaign/channel shows Significant Lift, it’s backed by a p-value < 0.05, indicating high confidence.

  • Example: Organic Social shows +22.65% Lift with a strong p-value – it’s first touch significantly boosts conversion (100%) from Lead to MEL against all other campaigns (81.53%)

Campaign Lift Analysis - Individual Campaigns

The Campaign Lift Analysis - Individual Campaigns widget uses the same methodology as the Campaign Lift Analysis - Across Campaigns widget. The difference lies in how the lift is calculated

• Campaign Lift Analysis - Individual Campaigns compares lift with the same campaign. It analyses whether having the touch of the campaign is more beneficial than not having the touch of the campaign. For example., In case of first touch, it analyzes whether a MQL with first touch of Marketing Form-Fill has a higher chance of conversion over not having the first touch of Marketing Form-Fill

• Campaign Lift Analysis - Across Campaigns compares lift with other campaigns. It analyses whether having a touch of the campaign is more beneficial than any other campaign. For example., In case of first touch it analyzes whether leads with first touch of Marketing Form-Fill has higher chance than any other campaign.

  Note :You will notice that the conversion% in Across Campaigns is the same as True value in Individual Campaigns

  

  Control Value - Refers which value do you want to compute the lift. Since we would like to compute the lift of true over false in most cases, the control value remains False and test value remains True.
  
  Minimum Sample Size Method used: Two-proportion z-test

Calculation Logics of Campaign Lift Analysis

Individual Campaigns

1. Calculating the conversion % of a campaign touch

   1. Calculate the Number of leads with a campaign touch that converted - X

   2. Calculate the Number of leads with a campaign touch that did not convert - Y

   3. Calculate the Number of leads without a campaign touch that converted - X'

   4. Calculate the Number of leads without a campaign touch that did not convert - Y'

   5. Conversion % of True = X/(X+Y)

   6. Conversion % of False = X'/(X'+Y')

2. Calculating the lift

   1. If Control Value is False, (Conversion of True - Conversion of False)/Conversion of False

   2. If Control Value is True, (Conversion of False - Conversion of True)/Conversion of True

Across Campaigns

1. Calculating the conversion % of a campaign

   1. Calculate the Number of leads with a campaign touch that converted - X

   2. Calculate the Number of leads with a campaign touch that did not convert - Y

   3. Conversion % of a campaign = X/(X+Y)

2. Calculating the conversion % of all campaigns

   1. Calculate the Number of leads with any of the campaigns touched that converted - X

   2. Calculate the Number of leads with any of the campaigns touched that did not convert - Y

   3. Conversion % = X / (X+Y)

3. Calculating the conversion % of selected campaigns

   1. Calculate the Number of leads with any of the selected campaigns touched that converted - X

   2. Calculate the Number of leads with any of the selected campaigns touched that did not convert - Y

   3. Conversion % = X/(X+Y)

4. Lift is calculated,

   1. When comparing against a base value of the average of all campaigns, (Conversion of a campaign- Conversion of all other campaigns)/(Conversion of all other campaigns)

   2. When comparing against a base value of the average of selected campaigns, (Conversion of a campaign- Conversion of selected other campaigns)/(Conversion of selected other campaigns)

   3. When comparing against a base value of a campaign, (Conversion of a campaign- Conversion of base campaign)/(Conversion of base campaign)

Differences in Difference (Test vs Control Group Impact Analysis) :

In RevSure, the Difference-in-Differences (DiD) Analysis capability extends the existing Pre & Post Analysis widget to enable causal impact measurement between Test and Control groups. By toggling DiD ON, users can evaluate whether a change in a Response Variable is attributable to a Treatment, beyond what would have happened naturally over time.

What is Difference-in-Differences?

Difference-in-Differences (DiD) is a quasi-experimental technique that compares:

• The change over time in an outcome for a Test group, against

• The change over time in the same outcome for a Control group.

This allows RevSure to isolate the incremental impact of a marketing or GTM intervention, while controlling for baseline trends that affect both groups.

Intuition:

If the Test group improves more than the Control group from Pre → Post, the excess improvement can be attributed to the Treatment.

When to Use DiD Analysis

Use DiD when:

• You have a clear intervention or treatment (campaign, spend increase, event, etc.)

• You can define a Test group that received the treatment

• You have a reasonable Control group that did not receive the treatment

• Randomized experiments are not feasible, but historical data is available

Key Concepts

Treatment

The action or intervention being evaluated.

Examples:

• Digital Paid Spend increase

• New campaign launch

• Event execution

• Competitor activity

Response Variable

The business outcome is impacted by the treatment.

Examples:

• Pipeline Generated

• Lead Generation

• Booking Value

Test Group

Entities (leads, opportunities, accounts, etc.) that received the treatment.

Control Group

Entities that did not receive the treatment serve as the counterfactual.

How DiD Works in RevSure

1. Enable DiD Analysis

• A DiD toggle is available within the Pre & Post Analysis widget

• Default state: OFF

• When turned ON, a configuration modal opens

DiD Configuration Modal

Step 1: Name the Configuration

Users can assign a name to the DiD setup for easier reuse and sharing.

Step 2: Define Test and Control Groups

Users choose how to configure the Control group using radio buttons:

Option 1: Default Control Group

• The control group is automatically defined as everyone not included in the Test group

• Users only configure:

  • Filters

  • Date filters

  • Consider only the first movement flag

• Applied only to the Test group

Helpful UI cues:

• Label indicating the size of the Test group

Option 2: Configure Control Group

• Users explicitly define both the Test and Control groups

• Control group filters must be mutually exclusive of the Test group

• While configuring Control filters, users can only select values that do not overlap with Test selection

Helpful UI cues:

• Label indicating the size of the Control group

Additional behavior:

• The treatment metric for the control group is blank by default

• Users may optionally define a different Treatment metric at the group level

Validation & Safeguards

• Overlapping Test and Control filters trigger a warning

• The run button is disabled for invalid configurations

• Missing date filters show inline validation errors

Output & Results

Table View

Results are displayed in a structured table with:

• Two collapsible sections:

  • Test Group

  • Control Group

Each section includes:

Calculated DiD Metrics

• DiD Effect (Difference in Totals)

• DiD Effect (Difference in Averages)

These quantify the incremental impact attributable to the Treatment.

Statistical Methodology

Estimation Approach

RevSure estimates the DiD effect using Ordinary Least Squares (OLS) regression:

• The DiD coefficient captures the incremental impact of the Treatment

• It measures how much more (or less) the Test group changed relative to the Control group

Statistical Significance

• A two-sided t-test is applied to the DiD coefficient

• Uses classical OLS standard errors

• Determines whether the observed impact is statistically significant

Interaction with Global Filters

• Global filters are disabled when DiD is enabled

• This avoids ambiguity and ensures Test/Control definitions remain explicit

Summary

The Difference-in-Differences Analysis in RevSure enables robust, causal measurement of marketing and GTM interventions by extending Pre-Post analysis to Test vs Control comparisons. With built-in safeguards, flexible group configuration, and statistically grounded outputs, DiD helps users move from correlation to confident impact attribution.