Predictive Modelling: Turning Everyday Signals into a Churn Risk Score.

A note on the data: The dataset used here is synthetic (B2B SaaS churn dataset). Treat the patterns as a demonstration of method and the decisions this kind of analysis supports, not as findings about any real company.

The short version

This demo report is for CEOs and leadership teams for a B2B SaaS business, and it answers four questions.

1. Where does our revenue come from?

2. Where are we losing it?

3. Can we see churn coming?

4. Is the current early-warning system working?

Method Overview: 50 000 B2B SaaS accounts, one row per account. Firstly I cleaned and checked the data, compared churn across operational and demographic signals, then trained a gradient boosting model to score every account’s churn risk. Scores are out-of-fold (5-fold cross-validation), so no account is scored by a model that has seen it. Inactivity is deliberately excluded from the model to avoid leakage

---

What I’m working with

The data is a single snapshot of B2B SaaS accounts: 50 000 rows, one per account, 18 columns. Below is the schema grouped by what each field is for.

Field group	Columns	Why it matters
Identity	Customer_ID, Company_Name	Join key back to your CRM
Segment	Industry, Country, Company_Size, Account_Manager	The cuts leadership usually discusses
Commercial	Plan_Tier, Billing_Cycle, Monthly_Recurring_Revenue	Revenue weighting
Lifecycle	Signup_Date, Last_Active_Date	Tenure and inactivity
Billing health	Payment_Status (Active, Pending, Failed)	Involuntary-churn signal
Engagement	Active_Seats_Used, Total_Seats_Allowed, Feature_Adoption_Rate	Product usage depth
Support	Support_Tickets_30d	Friction and distress
Existing model	Churn_Risk_Score (Low, Medium, High)	The legacy early-warning system
Outcome	Is_Churned (0 or 1)	What I’m trying to predict

I add four derived columns: a numeric feature-adoption percentage, days since last activity, seat utilisation (seats used ÷ seats allowed) and an ordered plan tier (Starter < Pro < Enterprise).

snapshot = pd.Timestamp("2026-02-28")
df["adoption"] = df["Feature_Adoption_Rate"].str.rstrip("%").astype(float)
df["days_inactive"] = (snapshot - pd.to_datetime(df["Last_Active_Date"])).dt.days
df["seat_util"] = df["Active_Seats_Used"] / df["Total_Seats_Allowed"]

If you want to replicate with your own data, the same shape works for any product with billing, usage and support data.

---

Can I trust the data?

Before drawing conclusions, I ran a quick audit: missing values, duplicate IDs, impossible dates and impossible seat counts.

checks = {
    "missing_values": int(df.isna().sum().sum()),
    "duplicate_ids": int(df["Customer_ID"].duplicated().sum()),
    "active_before_signup": int((pd.to_datetime(df["Last_Active_Date"]) < pd.to_datetime(df["Signup_Date"])).sum()),
    "seats_over_limit": int((df["Active_Seats_Used"] > df["Total_Seats_Allowed"]).sum()),
}

Check	Result
Missing values (all columns)	0
Duplicate Customer_ID	0
Last activity before signup	59
Seats used > seats allowed	0
Average MRR, annual vs monthly	Annual R665 · Monthly R666

The data is clean overall. Four things are worth knowing:

• 59 rows have a last-activity date earlier than the signup date. That’s a tiny share and changes no conclusion, but the source system should be fixed.

• Annual and monthly plans show the same average MRR. Either annual-prepay discounts aren’t captured or MRR is list price, so I treat every revenue figure here as list MRR.

• Plan tier is almost fully determined by company size (1 to 50 staff is Starter, 51 to 500 is Pro, 501+ is Enterprise). Plan and size are effectively one segment.

• Inactivity is a result of churn, not a predictor of it. Every retained account was active within the last 13 days, and every churned account had been inactive for 30 days or more.

Status	Min days inactive	Max	Mean
Retained	0	13	6.5
Churned	30	179	103.7

Leakage warning:

Because inactivity simply reflects that an account has already left, using it to “predict” churn would make a model look brilliant and be useless in production. I excluded it from everything that follows. This is the most common way churn models fool their builders, so audit your own columns for it first. Ask of every feature: would I know this before the customer cancelled?

---

Where the money is and where it leaks

Here is the business at a glance.

KPI	Value
Accounts	50 000
Total MRR	R33.27M
Implied ARR	R399.21M
Average MRR per account	R665
Churned accounts	916
Logo churn rate	1.83%
MRR lost to churn	R396K
Revenue churn rate	1.19%

Logo churn of 1.8% looks healthy. Splitting by plan shows why that number hides the real story.

Plan	Accounts	MRR	Churn Rate	MRR Lost	% of Accounts	% of MRR	% of MRR Lost
Starter	30 008	R2.67M	2.3%	R61K	60.0%	8.0%	15.3%
Pro	14 941	R5.39M	1.2%	R63K	29.9%	16.2%	15.8%
Enterprise	5 051	R25.21M	1.1%	R273K	10.1%	75.8%	68.9%

The unit economics are what make this vivid.

Average MRR is roughly R89 for a Starter account, R361 for Pro and R4 991 for Enterprise. Losing one Enterprise account costs the same monthly revenue as losing about 56 Starter accounts. Starter churns about twice as fast as the other tiers (2.3% vs 1.2% and 1.1%), yet it barely moves revenue.

So what?

Revenue protection is an Enterprise problem. Logo retention is a Starter problem. They need different playbooks: high-touch for Enterprise, automated and scaled for Starter.

---

What drives churn, and what doesn’t

To find the signals, I band each candidate metric (adoption, seat use, ticket count) and measure churn inside each band, alongside payment status.

df["adoption_band"] = pd.cut(df["adoption"], [0, 40, 50, 60, 70, 85, 100])
df.groupby("adoption_band", observed=True)["Is_Churned"].mean()
# repeat for pd.cut(seat_util), Support_Tickets_30d bins and Payment_Status

• Failed payment: 14.5% churn, about 13 times the 1.1% of healthy accounts. This is the single biggest and most fixable lever.

• Feature adoption has a steep gradient: 8.2% churn at 30 to 40% adoption, falling to 0.2% above 85%.

• Seat utilisation follows the same pattern: accounts using 30% or less of their seats churn about 37 times more often than accounts using more than 80%.

• Support tickets are a distress signal: 6 to 8 tickets in 30 days means 6.6% churn (about 3.5 times the average), versus 0.2% for 0 to 1 tickets.

---

The cuts that don’t matter

I ran the same analysis across industry, country and account manager, the splits most leadership reviews are built around. Churn is essentially flat across all three, with every group sitting between about 1.5% and 2.1%.

It means you shouldn’t allocate retention effort along those lines, and a performance review that blames one account manager’s “bad book” is likely reading noise.

So what?

Churn isn’t random, and it isn’t demographic. Four everyday operational signals tell me who is at risk long before the cancellation. Route retention work by account health, not by industry, country or owner.

---

Why the legacy score fails

The business already has a Churn_Risk_Score. I cross-tabulated its labels against what actually happened.

Legacy Label	Retained	Churned	Churn Rate	Share of all Churners
Low	48 792	737	1.5%	80.5%
Medium	286	168	37.0%	18.3%
High	6	11	64.7%	1.2%

The score flags only 471 accounts as Medium or High. Of those, 179 churned, so when it raises a flag it’s right about 38% of the time, roughly 21 times the 1.8% base rate. It simply raises too few flags: those 179 are only about 20% of all 916 churners, and 80% of lost customers were labelled “Low risk.”

In data-science terms, the legacy score has decent precision and terrible recall. It’s a coverage problem, not an accuracy problem, and that’s good news, because coverage is fixable by scoring every account on a continuous scale instead of three coarse buckets.

---

Building a better health score

The model is a gradient boosting classifier trained on nine features: payment status, feature adoption, seat utilisation, support tickets, MRR, plan, billing cycle, industry and country. I exclude inactivity to avoid leakage. Crucially, every account is scored out-of-fold with 5-fold cross-validation, so no account is ever scored by a model that has seen it.

from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import StratifiedKFold, cross_val_predict
from sklearn.metrics import roc_auc_score
 
features = ["payment", "adoption", "seat_util", "tickets", "mrr",
            "plan", "billing", "industry", "country"]  # no days_inactive
X, y = prep(df[features]), df["Is_Churned"]
 
model = HistGradientBoostingClassifier(random_state=42)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
df["p_churn"] = cross_val_predict(model, X, y, cv=cv, method="predict_proba")[:, 1]
print(roc_auc_score(y, df["p_churn"]))   # 0.859 on the original data

The snippet is a simplified version of the approach, and prep stands in for your own encoding of the categorical columns. On the original data the out-of-fold AUC is 0.859. In plain terms: pick one churned and one retained account at random, and the model ranks the churner as riskier about 86% of the time.

AUC is abstract, so I also measure what a customer success team cares about: if there’s only capacity to review X% of accounts, how many churners do I catch, and how many of the accounts reviewed actually churn?

Accounts Reviewed (Highest Risk First)	Churners Caught (Recall)	Flagged Accounts that Churn (Precision)	Lift vs Random
Top 5% (2 500 accounts)	44%	about 16%	8.8x
Top 10% (5 000 accounts)	57%	about 10%	5.7x
Top 20% (10 000 accounts)	72%	about 7%	3.6x
Legacy score (471 accounts, about 1%)	20%	38%	about 20x per account, but far too narrow

Because the new score ranks every account, leadership can choose how wide to cast the net based on team capacity. Reviewing the top 10% catches about three times as many churners as the legacy score.

What that costs:

Catching 57% of 916 churners is about 522 accounts, found by reviewing 5,000. Roughly 1 in 10 flagged accounts actually churns; the other nine don’t. That is still 5.7 times better than picking at random, but it means the review has to be cheap per account (automated outreach for most, human time reserved for the high-MRR few). Compare the cost of a review with the MRR a save protects before deciding how wide to go.

How much to trust these numbers:

916 churners is enough to rank accounts sensibly, but it is not a large number, so the recall and lift figures are single estimates with real uncertainty around them. I haven’t put confidence intervals on them here; a bootstrap over the out-of-fold predictions would give a range, and the prospective check described below is the better test.

---

What the model relies on

Payment status, adoption, support load and seat usage account for about 88% of the model’s importance. Industry and country together contribute about 5%, which confirms what the banded charts showed.

Caveat:

Tickets and adoption are recorded at the snapshot, so some of the signal may reflect churn already in progress. Validate prospectively before rollout: score today, then check outcomes in 90 days.

---

Revenue at risk today

Now I apply the score to customers who are still here. I define “high risk” as the top 10% of risk among retained accounts and total the MRR they hold.

Plan	High Risk Accounts	MRR at Risk	ARR at Risk	Legacy Says "Low"
Starter	3 933	R348K	R4.18M	94%
Pro	751	R269K	R3.22M	94%
Enterprise	227	R1.11M	R13.29M	90%
Total	4 911	R1.72M	R20.69M	94%

Two things stand out. First, Enterprise holds R1.11M of at-risk MRR in just 227 accounts, while Starter needs 3,933 accounts to reach R348K. Second, 90% or more of these high-risk accounts are currently labelled “Low” by the legacy score, so nobody is working on them.

I also compute a probability-weighted expected loss: the sum of each retained account’s churn probability times its MRR. It comes to R398K per month, which is the model’s estimate of the MRR at risk across the accounts still retained.

---

The priority call list

The most useful output for a customer success team is a named list. These are the five highest-MRR Enterprise accounts flagged as high risk.

Customer ID	Company	MRR	Payment	Adoption	Seat Use	Tickets (30d)	Legacy	Health Risk
CUST-5974	Omni Labs 279	R8 489	Pending	56%	55%	6	Low	4.1%
CUST-47202	Quantum Tech 713	R8 457	Failed	67%	76%	4	Low	9.5%
CUST-49193	Secure Scale 104	R8 333	Failed	78%	85%	3	Low	4.1%
CUST-16224	Apex Sphere 213	R8 318	Failed	52%	47%	5	Low	20.1%
CUST-2048	Vibe Sphere 272	R8 312	Failed	73%	64%	3	Low	4.4%

How to read the “Health risk” column.

It is a ranking, not a forecast for the individual account. Overall churn is only 1.8%, so even a high-ranked account usually carries a small absolute probability. An account at 4.1% is still more than twice the base rate and sits in the top 10% of retained accounts, and a failed payment adds to the case for a call. The score decides who gets looked at first, and the drivers (payment, tickets, adoption) tell the account manager what to say.

---

A quick win: failed-payment recovery

Failed payments are the strongest churn signal and the cheapest to fix, with dunning emails, a card updater and an account manager call. The scenario below estimates the monthly revenue saved if a share of currently failed, still-retained accounts is brought back to the healthy-account churn rate.

The arithmetic:

There are 1 735 retained accounts with a failed payment, holding R1.26M MRR. Their churn rate is 14.5% versus 1.1% for healthy accounts, a gap of 13.4 percentage points. Fully closing the gap would save about R169K of MRR; recovering only a share of it gives:

Recovery Rate	MRR Saved per Month	ARR Saved
25%	R42K	R506K
50%	R84K	R1.01M
75%	R127K	R1.52M

This is a conservative, single-period estimate. It ignores the lifetime value of retained accounts and any upsell, so the real value is likely higher.

---

From score to system: the architecture and automation

The value comes from putting the score where people already work. Here is the minimum viable pipeline.

1. Extract (nightly): Pull billing status, seat counts, feature usage and ticket counts from the billing system, product analytics and help-desk into one account-level table keyed on Customer_ID.

2. Score: Run the model on every active account and write back a health_score (0 to 100) and risk_band field.

3. Sync: Write both fields to the CRM account record so they’re visible in the same view the account manager already uses.

4. Trigger: Fire a webhook when an account enters the top risk band or its payment status flips to Failed.

5. Route: By plan, since the economics differ. Enterprise goes to a named owner, Pro gets a playbook task, Starter gets an automated email sequence.

An illustrative webhook payload for step 4:

{
  "event": "account.risk_band_changed",
  "customer_id": "CUST-47202",
  "plan": "Enterprise",
  "mrr": 8457,
  "risk_band": "high",
  "health_score": 91,
  "drivers": ["payment_failed", "tickets_30d>=4"],
  "route_to": "enterprise_save_desk"
}

Including the drivers in the payload matters more than the score itself. A customer success manager who sees “payment failed, 4 tickets” knows what to say on the call. A manager who only sees “91” doesn’t.

---

Recommendations

#	Recommendation	Owner	KPI To Track	Timeline
1	Retire the legacy risk score and adopt the four-signal health score (payment, adoption, tickets, seats) in the CRM	RevOps / Data	% of churners flagged in advance (target above 60%, today 20%)	30 days
2	Failed-payment playbook: automated dunning on day 0, 3 and 7, a card updater, and an account manager call for Pro and Enterprise	Finance + CS	Failed-to-recovered rate; churn among failed accounts	30 days
3	Enterprise save desk: a named owner for each high-risk Enterprise account, an executive sponsor call and a 30-day success plan	CS leadership	Enterprise MRR churn; save rate	60 days
4	Adoption programme: guided onboarding, in-app nudges for core features, seat-activation campaigns	Product + CS	Accounts above 70% adoption and above 60% seat use	90 days
5	Scaled Starter motion: automated, health-based emails instead of account manager time	Growth	Starter logo churn (today 2.3%)	90 days
6	Stop segmenting retention work by industry, country or account manager; route by health score instead	CS ops	n/a	Immediate

---

Limitations and how to reproduce this

This analysis is a demo, and it has limits worth being honest about.

• Data issues. Resolve the 59 date conflicts and store net (discounted) MRR for annual contracts.

• Adding time-series data. Monthly usage, tickets and NPS would let the score predict future churn rather than describe the current state.

• Validate prospectively. Score all accounts now, then measure 90-day churn by risk decile before relying on the model.

• Quantify the uncertainty. With 916 churners, the recall and lift figures are point estimates. A bootstrap would give a range.

• Remember the data is synthetic. The point is the method and the decisions it supports.

To run this on your own data, you need at minimum one row per account with: an account ID, plan or segment, MRR, billing or payment status, a seat or licence count (used and allowed), a feature-adoption or usage-depth measure, a 30-day support ticket count, and a churn flag. Keep activity-date fields out of the model, as explained above. If you can export a monthly history of those fields, even better.

---

Want this built on your data?

If your team is sitting on churn, retention or revenue data and the early-warning system isn’t catching what it should, build this kind of analysis end to end, from the data checks to the playbook leadership can act on, and into the CRM so it actually gets used.