Does the Punchh referral program drive signups and customer values?

Authors: Ritika Gill, Pratibha Pagaria

Referrals are one of the most reliable and trusted factors in customer acquisition resulting in remarkably high profits. It is an effective way to acquire anonymous customers by the existing ones. Many questions arise for businesses like whether they are using the right type of referral program: Does referral drive signups? Does referral have an impact on customer lifetime value (CLV)? Is the impact statistically meaningful?

Historically, restaurant chain owners have had limited knowledge of their customers. To overcome this problem, Punchh (www.punchh.com) has introduced a modern CRM platform for the brick-and-mortar business world that optimizes your loyalty programs and marketing campaigns. We can effectively target different customer segments, based on real-time customer data wherein businesses can engage customers using different ways like offers, gaming, online ordering, payments, gift cards, referrals, social media, surveys, feedback, reviews, and ratings.

According to Nielsen[1] and Referral SaaSquatch[2], referral plays a significant role in the guest acquisition and says that every referring customers make an average of 2.68 invites. The studies show that referral programs have significant value but businesses are still facing issues in defining an appropriate referral program. To understand what referral program would suit business one should know their priorities clear as there are different kinds of programs used in industry like direct wherein existing customer refers anonymous and both get rewards, tangible where you offer a tangible incentive to your customer, dual-sided where business ensures that both parties referrer and the referee gets benefits for the referral action performed. The dual-sided reward is a most industry-wide adopted program based on their situation and intentions.

Dual-sided is further categorized based on a business’s goal as Type A and Type B reward. Type A is when a guest does signup using an invite code, and irrespective of his contribution to business both referral and the referee gets the benefit. Uber is an instance of two-sided incentive wherein they are offering $25 on signup for both the referrer and referee. Type B is the case where a referral gets the benefit only when the referee adds value to the business. Amazon Prime and PayPal are among the few businesses wherein referral will receive the benefit if the referred customer signs up and adds some revenue. Punchh also uses a Type B referral program. To understand if the program really created an impact or not, a detailed study was done. Below are some of my findings from the study.

On analyzing Punchh’s 80+ businesses and 30M+ active users within 6 months of business go-live, we have found that the total signups using invite code account for 9% of the overall signups. This percentage of signups varied from 3% to 55% based on offers gifted to the guest where the highest was for an American casual dining restaurant chain. A further drill-down analysis was conducted to check on various restaurant types, inferring that average referred signups account for 9%, 12% and 9% for fast casual, casual dining and quick serve, respectively.

The natural follow-up question is: do referrals have any impact on loyalty? There may be guests who visit the store because of referral benefits however doesn’t stay associated with the business for a longer duration. How businesses can measure the referred guests loyalty? To assess this comparison was done among guests who did signup using an invite code called “referred” versus other guests termed as “organic”.

Out of the multiple techniques used to measure the effectiveness, this blog talks about the proportional test, survival analysis, and ways to handle the censored data. First, the proportional test helps one to validate the statistical significance between two numbers or percentage. For example, there is a survey in school to check how many male and female are underweight?

From the table above, it is seen that female has a higher percentage of underweight as compared to male but is this statistically significant? The proportional test used with a 5 % level of significance gives a p-value of 0.01 (less than the significance level) which represents the difference between 17% and 23% is statistically significant. We used the above test to check the significance of referrals while converting an anonymous guest to a loyalty guest for 80+ businesses.

Below is a sample data set with 2 anonymized businesses (A and B). For example, Business B had ~2.7K loyalty guests out of 3.5K signup events for referrals vs. 51K out of 70K for organic. This implies that conversion to loyalty guest from signups is 76% for referred vs. 74% for organic. Now, both referred and organic guests are two different groups with different sample sizes how one can say that 76% and 74% conversion is significant? For this, the proportional test is used.

• Signups: the count of guests who signed up during a specific duration
• Loyalty: a guest is considered a loyalty customer when (s)he made her/his 1st visit
• Conversion: the percentage of guests, who converted to loyalty (loyalty’s / signups)

Proportion test is testing if the success probabilities from two groups are the same, or equal to a certain value. The sampling method used is random sampling where the samples are independent and a probability must lie between 0 and 1. For example, each trial here is a sequence of n independent Bernoulli trials and the conversion of a trial is either 0 or 1.

The R libraries and functions stats and prop.test were used during the analysis: prop.test(x, n, alternative = “two.sided”, conf.level = 0.95)

• x: the count of successes
• n: the total number of trials
• conf.level: the confidence interval. 95% is a convention in the statistics community. In industry, 80% and 90% are also routinely used.
• alternative: it specifies the type of hypothesis

By applying the prop test on all businesses combined, we have determined that with a 5% level of significance, referred guest are 18% more likely to convert to loyalty than organic — a big win for Punchh’s referral signup program.

Proportional test explains significance in change, but it has a potential survival bias problem. If a guest signed up and converted to loyalty after the study period was ended, it would not be considered in a prop test. Then how should a business account for such situations? How should a business alleviate survival bias and censorship problem?

Censoring, Survival Bias and Survival Analysis

In survival analysis[3], the focus is on the time the event has occurred and is tracked over the specific time period and whenever the observation is left incomplete it is called censoring.

For example, when a study is conducted to measure the impact of the drug on the mortality rate for a specific time period, it may happen that an individual withdraws from the study or is still alive after the study ends. The situation when information about the subject cannot be gathered completely is referred to as censorship[4]. Censorship can be classified as right, left and interval; wherein the right and left represents the data point is above or below a certain point.

Now, when the data is censored we have to use another technique to remove the bias from the study. Cox Proportional[5], a type of regression model which explains how dependent variable changes with a change in independent variables is used. In our study:

• Event: Conversion of a guest to loyalty
• Dependent variable: Number of days taken
• Independent variable: Signup channel (organic or referred), Signup month

Note: in the below example, only 2 independent variables are considered, but Punchh is using multiple independent variables in their real work.

Sample data set used to apply the Cox proportional model:

• Signup channel 0 is ‘referred’ guests; 1 is for ‘organic’ guests
• Signup Date is date guest joins the business
• Signup Month is a month of signup
• First visit date is the first actual visit after signup. NA means (s)he has not visited after the signup event
• Number of days is the time between signup and first visit
• Status 0 is censored and 1 is uncensored
• Guest converted to loyalty after the study ends or churns during the study is censored (A and B)

Thankfully, R has provided the survival package and the coxph function, so we did not need to reinvent the wheel. Our study period is for three months:

result = coxph(Surv(days_till_loyalty, status, type = “right”) ~ signup_channel + signup_month, data = data)

From the above output report, we can see that p-value is 4.41e-11 for the “Signup Channel” factor, indicating that there is a significant difference between the signup channels. The “Signup Month” factor, by comparison, is not statistically significant (p-value = 0.49). The value from “Signup Channel” : exp(coef) = 1.1274 indicates a referral guest has 11.3%, i.e. (1–1/1.1274) shorter time to convert to loyalty. Note: 11.3% has a smaller magnitude than the 18% quoted from the above prop test; however, they are not the same performance metrics.

CLV of referred versus organic guests

Who are or will be your most valuable customers? This question is essential for businesses nowadays and the answer to this is Customer lifetime value (CLV). CLV is a forward-looking metric that estimates how much a customer is worth over the entire time. One-year forward-looking horizon is normally considered as an industrial convention. With an accurate CLV prediction, the business can carry out many marketing operations. Punchh machine learning group has built a world-class AI product that can provide the customer-level CLVs for all our business clients. It has a very rich set of features including user purchase behaviors, user demographics, business metadata, and their complicated combinations.

To check the impact of referrals, we performed the analysis for businesses with different restaurant types. We found that the average CLV of a referred guest in casual dining is $9 higher than that of organic signup (one-year horizon).

Conclusion

From this analysis, we have found strong evidence that the Punchh referral program has been a significant win in terms of increasing the customer signup probability and driving the customer lifetime values. We have also demonstrated how the Punchh analytics team is able to use rigorous statistical and econometric techniques to quantify the benefits of Punchh products to our clients.

About the author

Ritika Gill is a data analyst at Punchh, who is specialized in statistical analysis.

Pratibha Pagaria is senior manager of analytics, where she leads a growing team of data analysts and helps clients in driving insights from data.

Reference

[1] The Nielsen Company: Personal recommendations and consumer opinions posted online are the most trusted forms of advertising globally: https://www.nielsen.com/content/dam/corporate/us/en/newswire/uploads/2009/07/pr_global-study_07709.pdf

[2] Referral Squatch: Referral marketing statistics 2017: https://www.referralsaasquatch.com/infographic-state-of-referral-marketing-statistics/

[3] Wikipedia on survival analysis: https://en.wikipedia.org/wiki/Survival_analysis

[4] Wikipedia on censoring in statistics: https://en.wikipedia.org/wiki/Censoring_(statistics)

[5] British Journal of Cancer(2003) MJ Bradburn, TG Clark, SB Love, and DG Altman: Survival analysis Introduction to concepts and methods: https://www.nki.nl/media/837544/bradburn2003a.pdf