The Economic Value Of Punchh Loyalty Program — An Intervention Analysis Investigation
Authors: Xin Heng
Executive Summary
By applying the intervention analysis technique on Punchh’s customer behavior data, we are able to quantify the economic value of the Punchh-powered loyalty program. For a particular business client, we have found that, by pausing the signup campaigns, they have failed to acquire 22% of new customers. After holding off the other loyalty campaign activities, the per-capita opportunity cost of customer value is $20 (with a one-year forecasting horizon).
Loyalty Management is an essential component of the marketing group in any organization. A company spends tens or hundreds of dollars to acquire a new customer. In order to turn such an investment into a long term profit, the engagement marketing managers must leverage any available new ideas and technologies at hand to continuously drive user satisfaction and customer values [1].
Punchh is the industry-leading marketing platform for restaurants and retailers. With 150+ enterprise customers, Punchh spans 45,000 stores in 12 countries totaling $20B loyalty GMV across 55 million consumers. More often than not, we hear the CMO’s asking: what is the true value of my loyalty management strategy? Are my customers spending more? Are they coming more frequently? More fundamentally, is my loyalty program ROI positive?
There are a few economic approaches that can be applied to address this business problem. One major data science area in the industry is experimentation, i.e. change something and measure the effect. In randomized controlled trials, for example, the company can hold out a small percentage of the customers that are completed excluded from the loyalty management program. When the experimentation period is ended, the marketing analyst would measure the difference in the customers’ purchase behavior, extrapolating that into a comparison of the whole loyalty program being fully turned on vs. turned off. It is a simple and elegant solution but may suffer from the Hawthorne Effect [2] or the opposite of it — once they know they do not receive certain loyalty perks, they might choose either to work extra hard to earn their loyalty milestones or being completely discouraged. The second approach is to select two equivalent sets of geographic regions and run different treatments. In other words, one set of geo’s is the control group while the other set is the test group. When the experiment is over, it is the same math as the first approach and would suffer a lot less in terms of “I got to know from my neighbor that she received many loyalty perks but not me”. Having said that, selecting two equivalent groups of geolocations is usually be a very challenging task — one may think this set of 20 cities is equivalent to the other set. However, they will likely show substantial divergence in customer behavior during the AA period, when the experiment technically has not even started.
The Punchh analyst team was recently given a rare opportunity to conduct such a measurement using the third technique. One of our business partners decided to hold back all their loyalty campaigns for several weeks. Instead of studying something when it is turned on, we are now able to do the opposite — quantify the value of the loyalty program when it is turned off. The economic approach is called Intervention Analysis [3]. And Google has coined it as Causal Impact [4]. The article is thus serving two purposes: it quantifies the value of a loyalty program using Punchh’s unique data set. It also demonstrates how to use this fundamental econometric approach to complete the study.
Data and the Event
In Punchh, we record every loyalty customer’s purchase and redemption behavior as it occurs in real time. Data points that are relevant to this research include customer transaction amount & timestamp, and new customer’s join date. We aggregate the above data at the weekly level, e.g. how many new customers are joined in a week, and how many loyalty visits have occurred during the same week.
We have identified the event time points as the following. Note: to further protect our client’s identity, we shifted the entire data by a random day amount and multiple all the metrics by an undisclosed factor.
Intervention event: Company X has paused all their loyalty campaigns on October 15, 2017
Pre-intervention period: 2014-Jan-05 to 2017-Oct-14 (197 weeks)
Intervention period: 2017-Oct-15 to 2017-Dec-02 (7 weeks)
Loyalty performance metrics in the study:
1. total signups: total number of new member signup events
2. total loyalty visits: total number of visits (online or in-store) from the loyalty members
3. loyalty revenue: total dollar value of loyalty members’ transactions
Figure 1 shows the time series of weekly loyalty visits with the vertical line being the intervention event date. Over the past four years, the company has enjoyed a substantial increase in loyalty activities: visits of the loyalty members have grown from 15,000 range to 25,000, i.e. a 66% increase. There are two dips in the end period of every year, indicating Thanksgiving and the Christmas season. There is also a seasonality pattern, i.e. a steady increase of loyalty visits until the mid-year before it starts to taper off until the end of the year. We will take advantage of the seasonality patterns in the later section.
In Table 1, we present several interesting summary statistics numbers for three loyalty performance metrics.
Take signups as an example: the median signups in a week are 8,986, but it can go as high as 21,717 — a 140% spike, or as low as 4,212. The swings in loyalty visits and loyalty revenue are less wild. In addition, the signups metric is showing right skewness, i.e. the mean is larger than the median. To our surprise, both loyalty visits and loyalty revenue show zero skew — something worth a future deep dive.
Methods
In the intervention analysis, the key step is to come up with high-quality counterfactual predictions. It can be easily explained with the use of Figure 3.
When creating the counterfactual market response, we use all the available historical data up to the intervention event date (the dotted vertical line in Figure 3). The US clicks is the metric in this example. We would then make time-series forecasts pretending there was no intervention event, and the market would have continued as normal, i.e. the blue dotted line. The delta between the black solid line (after the intervention date) and the blue dotted line is all we need to estimate the effect size of the intervention event, which is subsequently used to quantify other business metrics. We will show several examples later.
They are a wide variety of time series forecasting techniques in the market that can undertake this counterfactual prediction. We choose to use seasonal ARMA [5] as the workhorse model while using the weekly data frequency and year-over-year seasonality. We can demonstrate the model using this simplified formula:
y(t)= ARMA(lagged terms of y(t))
Here, y is the metric under study, e.g. weekly signups. A useful notion of ARMA model is ARMA(p, q) × (P, Q)s. Here, p, q are the orders of the so-called AR and MA terms. P, Q are those for the seasonal part. Coefficient s is the seasonality period, i.e. s = 52 because there are 52 weeks in a year. In econometrics, we say that the non-seasonal parts (p and q) control the short-term patterns while the seasonal parts (P and Q) control the dynamics of the correlations over multiple seasons. We will not dive deeply into the model approach, as one can easily find it in any econometrics textbook.
Other techniques that we take advantage of include:
- y(t) metric is almost always non-stationary, and thus we take a first-order seasonal difference.
- For the simplicity purpose and because of the low skewness of the 3 performance metrics, we decide not to take the log(y) operation on the variables.
- A company usually conducts certain yearly marketing activities. And a city always has year over year events or festivals. As an outsider, it is formidable to obtain the event calendar and insert it into the time series model. Thus, the 52-week seasonal model is an excellent proxy to mimic such yearly events without having to know that event calendar — a practical trick.
- One can introduce step functions to “mask” those one-off outlier events in a time series. The gotcha is how one can call out an event as an outlier, which is not worth studying.
We have implemented the above ARMA model in R, with the following workhorse function:
- D = 1 and d = 0 indicate that we are only taking a first-order seasonal difference
- Variable: “xreg” is a named matrix of independent, explanatory variables and is equal to NULL throughout the study
- We have explained that the time series has a year-over-year seasonality, i.e. frequency = 52, and we ensure seasonal equals to TRUE.
We demonstrate the result of the intervention analysis in Figure 4. By following the procedure explained with Figure 3, the red solid line is the in-sample fit of the loyalty visits time series up to the invention event date at 2017–10–15. It is able to capture both the year-over-year seasonal pattern as well as the Thanksgiving and Christmas dips. With the fitted model, we make predictions for the next 7 weeks in the intervention period. We denote them using the red dotted line. Thanksgiving is within the intervention period, and the fitted ARMA model is able to predict a dip in loyalty visits correctly. The black solid line is the real loyalty_visits values throughout the pre-intervention and intervention periods.
In Table 2, we document the realized and predicted loyalty visits during the intervention period. Without a doubt, the predicted (counterfactual) market response is substantially higher than that of the realized value. The difference is 30%, i.e. (sum(yhat) / sum(y)-1)100%. In other words, when turning off the loyalty campaigns, we estimate that the business has lost 30% of loyalty transactions. In both Figure 4 and Table 2, we also show the 80% error band, corroborating our observation that the predicted values are significantly higher than that of the realized values.
We can conduct a quick estimate of the opportunity cost of losing 30% of loyalty visits. We stick with the one-year forecasting horizon. We estimate that the average customer lifetime value (CLV) at the one-year horizon is $68, by applying Punchh’s proprietary CLV prediction engine. With a 30% drop in customer visits, the loss in CLV would be $20. Note that we have not taken into consideration the significant increase in churn probability; otherwise, the opportunity cost in customer value is believed to be even bigger.
Let’s now switch to a new loyalty performance metric: signups. The signup campaigns are also an important component of the business’s marketing strategy. Table 3 and Figure 5 show the realized and predicted weekly signups. By using the same arithmetic, we have found that the business has lost 22% new customers which they were supposed to acquire. Furthermore, Figure 5 also shows that the realized values (black solid line) in the intervention period is touching the lower boundary of the predicted error band, indicating the statistical significance of the effect.
Punchh’s CLV prediction engine can make a conservative estimate on the value of a new customer, which is $12. From Figure 5, we see that the weekly signups are approximately 12,500. The opportunity cost of subdued signup performance is $33,000 per week, i.e. 12,500 * 22% * $12.
Regarding acquiring new customers, the last question is whether the signup campaigns used to be a profitable strategy. Assuming the business did not turn off other customer acquisition activities (e.g., TV, Facebook), the 22% drop in signups would be attributed to the holdback of the Punchh-powered signup campaigns. $12, as previously mentioned, is the conservative estimate of the one-year value of a new customer. We can thus attribute $2.64 to the Punchh-powered signup campaigns. In a signup offer, the discount value applied on a signup coupon would be roughly $3.0 [6]. Thus, in any regard, it would be a wise idea to maintain the signup coupon campaign activities — one of the most ROI-positive customer acquisition channels.
Lastly, we address a finance question: has my business gained any (short term) revenue advantage by stopping all the loyalty campaigns? The hypothesis would be: we normally apply a hefty $ incentive in a loyalty campaign. By turning it off, we should save the cost and thus increase the profit. “loyalty revenue”, as a performance metric is indeed the measure of the gross profit from a loyalty member’s transaction, i.e. total receipt amount — discount amount. Figure 6 can help answer this question. It is not surprising that the hypothesis is not valid: there is no financial gain of turning off the loyalty program. The predicted values (red dotted line) and the realized values of the loyalty revenue, during the intervention period, have no difference.
Conclusion
Experimentation is becoming an essential part of the marketing org’s daily operations. It is a very effective means to measure the business value of many marketing strategies. In this work, we have an in-depth discussion of the intervention analysis technique and apply it to study a significant change point of an anonymous client’s loyalty program. We have found that, by holding off the loyalty program, they have failed to acquire 22% of new customers. The per-capita, annualized opportunity cost of customer value is $20. Lastly, there is no financial benefit of saving the promotional discount money by pausing the loyalty campaigns.
About the authors
Xin Heng is the Senior Director, Head of Data Science in Punchh, where his primary responsibility is to build world-class data solutions to drive the growth of our business partners. His team is working on AI & BI products and big data engineering tools.
Reference
[1] Punchh’s engaging marketing products: https://punchh.com/loyalty
[2] The Wikipedia explanation of the Hawthorne Effect: https://en.wikipedia.org/wiki/Hawthorne_effect
[3] Penn State’s introduction to the intervention analysis: https://newonlinecourses.science.psu.edu/stat510/node/76/
[4] Google’s explanation of causal impact: https://ai.google/research/pubs/pub41854
[5] Penn State’s introduction to the seasonal ARMA model: https://newonlinecourses.science.psu.edu/stat510/node/67/
[6] $3 cost may be an overestimate of the signup campaign cost. We will address this issue in a separate article.
