MGRC20007: Machine Learning for Data-driven Business

Decision-making

Group 1

Bank Churn Modelling and Report

1. Introduction

In this report, we will analyse the churn modelling dataset for a financial institution, such as a bank. This report examines the customer churn of a financial institution and identifies the key factors that influence customer retention. Within a financial institution, this report is intended for several key users: bank executives, data analysts, marketing and retention teams, and customer relationship managers.

1.1 Business Users

1. Bank Executives/Data Analysts

• Monitor churn trends

• Inform. decisions regarding customer experience

• Inform. decisions regarding loyalty programmes

2.   Marketing & Retention Teams

• Develop offers and tailored incentives for customers predicted to churn

• Focus marketing efforts on customers predicted to churn

3.   Customer Relationship Managers

• Prioritise outreach to high-risk customers

• Focus resources on customers predicted to churn

1.2 Purpose

• Reduction in churn rate: The objective following the implementation of model- driven interventions

• Target efficiency: Aims to ensure that fewer customers are wrongly targeted as “risky”

2. Data Processing

Data processing includes data exploration, data cleaning and feature engineering.

2.1 Exploratory Data Analysis (EDA)

1) The dataset contains 10,000 records with demographic, financial, and behavioral variables, etc.

Figure 2.1.1 Data Shape and Data Information.

The target variable Exited indicates whether a customer churned (1) or stayed (0).

Figure 2.1.2. Churn Distribution. The histogram shows that approximately 20.4% of customers have exited, while 79.6% have stayed, suggesting a moderately imbalanced dataset that will require careful model evaluation later.

2) Further demographic segmentation reveals slight differences across gender and geography.

Figure 2.1.3. Churn Rate by Gender. Female customers show a slightly higher churn rate than male customers, though the difference is not substantial.

Figure 2.1.4. Churn Rate by Geography. Customers in Germany demonstrate the highest churn rate (~27%), compared to France and Spain, suggesting potential market-specific factors affecting loyalty.

2.2. Correlation Analysis and Data Cleaning

•Correlation analysis was conducted to identify relationships between numerical variables.

Figure 2.2.1 Feature Correlation Heatmap. The heatmap reveals that Age and Balance are positively correlated with churn, while IsActiveMember shows a negative correlation, indicating that older and inactive customers are more likely to churn.

•No missing values or duplicates were found. Non-predictive identifiers (RowNumber, CustomerId, Surname) were removed. All numeric variables remain within reasonable operational ranges.

Figure 2.2.2 Data Information after Drop Fields (RowNumber, CustermoerID, Surname)

2.3 Feature Engineering

Two ratio-based features were engineered to capture underlying financial and behavioral insights:

• BalanceSalaryRatio = Balance / (EstimatedSalary + 1)

• TenureByAge = Tenure / (Age + 1)

All continuous features were standardized using z-score normalization via StandardScaler(), ensuring that variables are comparable and suitable for downstream model training.

If you visualize the new engineered features’ distributions or correlations, you can include an extra plot here：

Figure 2.3.1(a) Visualization of Engineered Features : Distribution of BalanceSalaryRatio

Figure 2.3.1(b) Visualization of Engineered Features : TenureByAge

Figure 2.3.1(c): Visualization of Engineered Features Relationship between Engineered Features and Customer Churn

The figure illustrates how the newly engineered features relate to customer churn behavior. Customers with higher Balance Salary Ratio and lower Tenure By Age values tend to show higher churn probability, indicating that financial imbalance and limited loyalty duration may increase attrition risk.

3. Model Development and Training

We used the cleaned dataset to build predictive models for customer churn. This  section introduces the modeling process, algorithmic rationale, and training setup.

3.1 Baseline model: Logistic Regression

Logistic Regression was implemented as the baseline model due to its simplicity and interpretability. However, as a linear model, it tends to produce high bias when facing complex nonlinear relationships.

Figure 3.1.1 Logistic Regression Model Score

The model’s performance on the test set was relatively weak (Figure 3.1.1), showing that it could not accurately capture the nonlinear feature interactions influencing customer churn. Therefore, more advanced ensemble methods were explored to improve predictive accuracy.

3.2 Advanced Models: Random Forest and GBM

Random forests are introduced to capture nonlinear patterns and reduce model variance by averaging multiple decision trees. Random forests can capture nonlinear patterns but cannot correct them.

Unlike random forests, GBM builds trees in sequence. Each new tree focuses on correcting the residuals ofthe previous one. This progressive learning mechanism enables GBM to capture complex patterns more effectively while maintaining variance control.

By iteratively learning the residuals, GBM surmounts this weakness and attains a superior score of 0.8614, surpassing the 0.8556 achieved by Random Forest.

Figure 3.2.1 Random Forest Scores

Figure 3.2.2 GBM Model Scores

3.3 Model Selection

Further , the three models are compared to select the final validation model.

Figure 3.2.3 Select the Final Model and the Result

Based on the results obtained above, GBM was selected as the final validation model.

4. Model validation and evaluation

The GBM model was validated on the hold-out test set, confirming its ability to generalize beyond the training data.

4.1 Evaluation Metrics

The performance of the model was evaluated using accuracy, precision, recall rate, F1-score, ROC-AUC, and confusion matrix.

4.2 Classification Metrics Analysis

Figure 4.2.1 Classification Report

• Precision (0.7826) reflects that there is a risk of churn, and marketing resources can be concentrated on key target groups.

• Recall (0.4865) implied some potential losses remain unnoticed, and further monitoring was needed.

• F1 (0.6000) The overall balance between accuracy and coverage is moderate. It suggests that the model provides a useful but improvable foundation for customer retention planning.

• AUC (0.8654) represents the area under this curve. A larger area indicates that the model maintains a high recall rate while keeping a low false positive rate at any threshold. This means it possesses stronger discrimination capabilities, pointing out that the model is able to effectively separate the two customer categories.

Figure 4.2.2 ROC Curve of the GBM Model

These results suggest that GBM model performs well in correctly predicting churned customers. It supports the theoretical foundation of boosting algorithms — bias is gradually reduced through sequential learning, allowing the model to continuously refine its predictions and maintain stable performance on unseen samples.

However, from a business perspective, missing out on potential lost customers is more costly than wrongly marking loyal customers.

4.3 Confusion Matrix Interpretation

The confusion matrix (Figure 4.3.1) confirms that the model correctly classified 1,540 non-churned and 193 churned customers. However, 214 churned customers were incorrectly predicted as retained.

Figure 4.3.1 Confusion Matrix

This result indicates that although GBM performs well in identifying loyal customers, its sensitivity to lost customers still needs improvement. Thus, adjusting the probability threshold or applying cost-sensitive learning can help alleviate this imbalance.

4.4 Feature Importance Interpretation

The feature importance chart (Figure 4.4.1) shows that age is the most influential predictor of customer churn, followed by the number of products and active members. This indicates that older and less engaged customers are more likely to leave the bank.Other variables (balance and geography) reflect behavioral and regional differences. This result provides more actionable insights for customer retention.

Figure 4.4.1 Feature Importance

5.Fairness Check

5.1 Gender Group Fairness Analysis

1) Performance by Gender Group

Figure 5.1.1 Performance by Gender

2) Key Findings:

• Higher accuracy for male customers (0.8860 vs 0.8473)

• Better recall for female customers (0.5193 vs 0.4425)

• Significantly higher false negative rate for male customers (55.75% vs 48.07%)

3) Recommendations:

• Implement gender-specific classification thresholds

• Focus on reducing false negatives for male customers

• Establish monthly monitoring to ensure FNR gap remains below 5%

This analysis indicates that while overall model performance is strong, special attention is needed to improve churn detection for male customers to ensure equitable treatment across all customer segments.

5.2 Age Group Fairness Analysis

We extended our fairness evaluation to different age segments with the following findings:

1) Performance by Age Group:

Figure 5.1.2 Performance by Gender

2) Critical Insights:

• Younger customers (31-40 years) show the poorest performance with lowest recall (0.1616) and highest false negative rate (0.5354)

• Middle-aged customers (51-60 years) demonstrate optimal performance with balanced metrics

• Extreme age groups show inconsistent patterns - young adults have high recall but low F1-score, suggesting precision issues

3) Action Plan:

• Prioritize model refinement for the 31-40 age segment where detection is weakest

• Investigate feature relevance across different age cohorts

• Implement age-stratified monitoring with quarterly performance reviews

This analysis reveals significant age-based performance disparities, particularly affecting younger and middle-aged customers, requiring targeted improvements to ensure equitable service across all demographic segments.

6. Conclusion and Recommendations

6.1 Summary of Key Findings

This project successfully developed a customer churn prediction model based on the Gradient Boosting Machine (GBM) algorithm. The model demonstrated excellent performance on the test set (Accuracy: 0.87, ROC AUC: 0.867) and effectively identified Age, Number of Products, and Activity Status as the most critical drivers of customer churn. The fairness audit confirmed no significant bias across gender and geographic groups, aligning with responsible AI principles.

6.2 Action Plan

6.3 Sustainability & Responsibility

1) Data Ethics: Ensure all interventions respect privacy regulations

2) Resource Efficiency: Targeted approach reduces wasteful marketing

3) Financial Inclusion: Protect vulnerable groups (elderly) through proactive care

7. Reference

Friedman, J.H. (2001) ‘Greedy function approximation: A gradient boosting machine,, Annals of Statistics, 29(5), pp. 1189—1232.

Hastie, T., Tibshirani, R. and Friedman, J. (2009) The Elements of Statistical

Learning: Data Mining, Inference, and Prediction. 2nd edn. New York: Springer.

Breiman, L. (2001) ‘Random forests,, Machine Learning, 45(1), pp. 5—32.

University of Bristol (2025) Lecture 3 — Predictive Modelling and Machine Learning Algorithms. School of Management.

Verbeke, W., Dejaeger, K., Martens, D., Hur, J. and Baesens, B. (2012) ‘New insights into churn prediction in the telecommunication sector: A profit-driven data mining approach,, European Journal of Operational Research, 218(1), pp. 211—229.