Credit Scoring Models Enhancement Using Support

Vector Machines
Authors Name/s per 1st Affiliation (Author) Authors Name/s per 2nd Affiliation (Author) Authors Name/s per 3rd Affiliation (Author)
Jinxin Xu Yu Cheng Liyang Wang
Department of Cox Business School The Fu Foundation School of Olin Business School
Southern Methodist University Engineering and Applied Science Washington University in St. Louis
Dallas, TX Columbia University St. Louis, MO, USA
[email protected] New York, NY, USA [email protected]
[email protected]
Authors Name/s per 5th Affiliation (Author)
Authors Name/s per 4th Affiliation (Author) Zhenglin Li
Kaixian Xu Department of Computer Science and
Tandon School of Engineering Engineering
New York University Texas A&M University
New York, NY, USA College Station, TX, USA
[email protected] zhenglin [email protected]

Abstract— In recent decades, financial credit scoring models making by lenders, thereby minimizing the risk of financial
are essential for assessing borrower creditworthiness in financial losses stemming from loan defaults. By accurately evaluating
industries, guiding lending decisions and risk management. the credit risk of applicants, lenders can tailor their lending
Traditional methods such as logistic regression and decision terms, such as interest rates and credit limits, to mitigate
trees, while effective, often fall short in capturing complex data potential losses while extending credit access to deserving
interactions and dealing with high-dimensional, diverse datasets. borrowers.
As a researcher who works in US bank, this paper explores
enhancement of credit scoring models through advanced machine Traditional credit scoring approaches, such as logistic
learning techniques, specifically focusing on comparison models regression and decision trees, have long been the industry
with support vector machines (SVM), random forests, and neural standard due to their simplicity and interpretability [1].
networks. These methods today are heralded for their robustness However, these models exhibit limitations in capturing the
in high dimensional data environments and superior capability in complexity inherent in credit data. One major limitation is their
the modeling non-linear relationships. The study involves reliance on linear relationships and predefined decision
comparative analysis using real-world credit data (German credit boundaries, which may fail to capture the nonlinear patterns
card data) to assess the performance of traditional versus and interactions present in credit data [2]. Additionally,
machine-learning-based models. Results indicate the SVM model traditional models may struggle to incorporate alternative data
significantly outperform traditional models in accuracy,
sources, such as social media activity or transactional data,
precision, recall, F1-Score and AUC. SVM accuracy is 75% than
which can provide valuable insights into an individual’s
70% in Logistic Regression model. Other model metrics are 5%
to 7% higher than other metrics. The adoption of SVM and other creditworthiness. Moreover, traditional credit scoring models
machine learning techniques promises a shift towards more often suffer from data sparsity and imbalances, particularly
dynamic, precise, and reliable credit scoring systems, enabling when dealing with rare credit events or subpopulations. This
lenders to make more informed decisions with reduced risk of can result in biased predictions and reduced model
defaults. This paper contributes to the ongoing evolution of real performance, especially for minority or underserved
credit assessment methodologies, emphasizing the integration of communities.
machine learning to address the limitations of conventional credit
The rapid advancement of machine learning techniques has
scoring approaches.
sparked interest in enhancing credit scoring models by
Keywords- Machine Learning; Credit Scoring Model; Support addressing the limitations of traditional approaches. Machine
Vector Machine; Risk Assessment learning algorithms, such as random forest, support vector
machines, and neural networks, offer several advantages over
traditional methods. Machine learning algorithms excel in
I. INTRODUCTION handling complex, high-dimensional data and can capture
Credit scoring models are pivotal tools in the financial nonlinear relationships and interactions more effectively [3],
industry, facilitating the assessment of creditworthiness for [4]. They are also capable of leveraging large volumes of
borrowers seeking loans or credit lines. These models analyze a diverse data sources, including structured and unstructured
multitude of factors, including credit history, income, debt data, to improve predictive accuracy. Furthermore, machine
levels, and payment behavior, to quantify the risk of default learning techniques enable the development of more adaptive
associated with extending credit. The significance of credit and dynamic credit scoring models that can continuously learn
scoring models lies in their ability to enable informed decision and evolve with changing market conditions and borrower
behavior. This adaptability is particularly valuable in today’s leveraging advanced modelling techniques and handling large
fast-paced and dynamic lending environment. The primary volumes of diverse data sources effectively.
objective of this study is to explore the potential of machine
learning techniques in enhancing credit scoring models. The B. Comparison of Machine Learning Algorithms for Credit
study aims to evaluate the performance of traditional credit In recent years, AI technology can assist financial
scoring approaches versus machine learning-based models institutions with credit scoring and risk prediction for
using real-world credit data. By addressing these objectives, the customers, as well as regulatory compliance and compliance
aim is to contribute to the ongoing discourse on the evolution process automation [6]. Additionally, a multitude of machine
of credit scoring methodologies utilizing Artificial Intelligence learning algorithms have been applied to credit scoring tasks,
and Machine Learning. each offering unique strengths and capabilities. This section
provides a comparative analysis of several prominent machine
II. RELATED WORK learning algorithms in the context of credit risk assessment,
drawing insights from relevant studies and empirical research.
A. Overview of Traditional Credit Scoring Methods
a) Random Forest: Random forest is a popular ensemble
Traditional credit scoring methods have served as the learning technique that combines multiple decision trees to
foundation of credit risk assessment in the financial industry improve predictive accuracy and robustness. Studies by
for decades. These methods, including logistic regression and Zaytsev, (2020) have demonstrated the effectiveness of
decision trees, have been extensively studied and widely random forest in credit scoring, particularly in handling high-
adopted due to their simplicity, interpretability, and proven dimensional data and capturing complex interactions among
track record in credit risk modelling.
predictor variables [7]. Random forest models exhibit
a) Logistic Regression: Logistic regression is one of the resilience to overfitting and noise, making them well-suited
most commonly used statistical techniques for credit scoring for credit risk assessment tasks with large and diverse datasets.
[5]. It models the probability of default (PD) as a function of b) Gradient Boosting Machines (GBM): Gradient
various predictor variables, such as credit history, income, and boosting machines, such as XGBoost and LightGBM, are
debt-to-income ratio. The coefficients estimated by logistic powerful algorithms that sequentially build a series of weak
regression provide insights into the relative importance of each learners to iteratively minimize prediction errors. Research by
predictor variable in predicting credit risk. Notable studies by Schmitt, (2022) has highlighted the superior predictive
such as Dumitrescu et al. (2021) have highlighted the performance of gradient boosting machines in credit scoring
effectiveness of logistic regression in credit scoring, tasks, outperforming traditional methods and other machine
particularly in predicting binary outcomes such as loan learning algorithms [8]. GBM models excel in capturing
defaults [2]. nonlinear relationships and feature interactions, making them
b) Decision Trees: Decision trees are another popular highly effective for complex credit risk modelling scenarios.
method for credit risk assessment, known for their intuitive c) Support Vector Machines (SVM): Support vector
decision-making process and ability to handle nonlinear machines are a class of supervised learning algorithms that
relationships in data. Decision trees recursively partition the construct hyperplanes in high-dimensional feature spaces to
feature space into subsets based on the values of predictor separate data points into different classes. Studies by Huang et
variables, ultimately leading to a tree-like structure where each al. (2007) have investigated three strategies to construct the
leaf node represents a predicted outcome. Tripathi et al. (2022) hybrid SVM-based credit scoring models to evaluate the
introduced the classification and regression tree (CART) applicant’s credit score from the applicant’s input features [9].
algorithm, which has been widely applied in credit scoring SVM models offer robustness to noise and outliers, making
tasks [6]. Decision trees offer transparency and ease of them suitable for credit risk assessment tasks with imbalanced
interpretation, making them valuable tools for credit risk or noisy datasets.
assessment.
d) Neural Networks: Neural networks, particularly deep
While traditional credit scoring methods have learning architectures like multi-layer perceptrons (MLP) and
demonstrated efficacy in certain scenarios, they exhibit convolutional neural networks (CNN), have gained
limitations in capturing the complex interactions and prominence in credit scoring due to their ability to learn
nonlinearities present in credit data. These limitations can lead complex patterns from raw data. Research by Ruiz et al.
to suboptimal predictive performance and hinder the ability of (2020) has explored the application of neural networks in
lenders to accurately assess credit risk. Moreover, traditional credit risk assessment, showcasing their capacity to
methods may struggle to incorporate alternative data sources, automatically extract hierarchical features and capture subtle
such as social media activity or transactional data, which can relationships in credit data. Neural network models exhibit
provide valuable insights into borrower behavior. flexibility and scalability, enabling them to handle large-scale
In the light of these challenges, there is growing interest in credit scoring tasks with diverse data sources [10]–[13].
exploring advanced methodologies, such as machine learning,
to enhance the predictive accuracy and robustness of credit
scoring models. Machine learning algorithms offer the
potential to overcome the limitations of traditional methods by
 III. METHODOLOGY IV. EXPERIMENT AND RESULT

A. Selection of Machine Learning Algorithm A. Data Collection

Selecting appropriate machine learning algorithms is a To illustrate the application of SVM in credit scoring, we
crucial step in developing credit scoring models that utilize a dataset commonly used in this field, such as the
accurately assess credit risk. This section outlines the “German Credit Data” which contains 1,000 entries with 20
methodology for selecting and evaluating machine learning attributes and a binary label indicating good or bad credit
algorithms, including random forest, support vector machines risks. Each entry in this dataset corresponds to an individual
(SVM), and neural networks, for credit risk assessment tasks. who has obtained a loan from a bank. Individuals are
This study utilized Support Vector Machine (SVM) for credit categorized as either good or bad credit risks based on a
risk assessment tasks in credit scoring. specific set of characteristics. Age (numerical) Gender
(options: male, female) Occupation (numeric: 0 - unskilled
B. Support Vector Machine (SVM)
and non-resident, 1 - unskilled and resident, 2 - skilled, 3 -
a) Introduction to SVM in Credit Scoring: Support highly skilled) Residential accommodation (options:
Vector Machines (SVM) are powerful supervised learning ownership, rental, or no cost) Savings accounts (text - low,
models used for classification and regression tasks. In the middle, affluent, wealthy) Checking account balance
context of credit scoring, SVMs are particularly valued for expressed in DM (Deutsch Mark). The credit amount is
their ability to handle high-dimensional data and their specified as a numerical value in Deutsche Marks (DM).
effectiveness in binary classification problems, such as Duration (numeric, in month) Purpose (text: automobile,
distinguishing between defaulters and non-defaulters in loan furniture/equipment, radio/TV, household appliances, repairs,
applications. education, business, vacation/others Risk (Value target - Good
SVM works by finding the hyperplane that best separates or Bad Risk).
different classes in the feature space. In a two-dimensional
space, this hyperplane can be thought of as a line dividing a
plane in two parts where each class lays on either side. For
non-linearly separable data, SVM uses a kernel trick to
transform the data into a higher dimension where a hyperplane
can be used to separate the classes.
Choose an appropriate kernel function that maps the input
data to a higher-dimensional space. The Radial Basis Function
(RBF) is a popular choice for mapping input data to a higher-
dimensional space, especially when using SVM. This kernel
function is very effective for non-linear classification or
regression problems. The RBF kernel, often known as the
Gaussian kernel, is defined by the formula:

K (x, x′) = exp (−γ∥ x − x′∥ 2)

Here (x, x′) are 2 vectors in the input space, while ∥x−x′∥
denotes the Euclidean distance between the vectors and γ is a
parameter that controls the width of the kernel, influencing the
behavior of the kernel function significantly.
b) Application of SVM in Credit Scoring: SVMs can
enhance traditional credit scoring models by providing a
robust alternative that is less likely to overfit compared to
more complex models like neural networks, especially when
the number of features is high relative to the number of data
Fig. 1. Model Process
points. Here is operational follow chart of SVM and applied to
credit scoring.
c) Insights and Business Implications: The application of
SVM in credit scoring can significantly aid financial B. Feature Selection and Engineering
institutions by improving the accuracy of risk assessment, Feature selection and engineering play a crucial role in
thereby potentially lowering the risk of defaults and enhancing developing effective credit scoring models by identifying
credit decision-making processes. SVM’s capacity to handle relevant predictors and creating informative features that
large feature sets efficiently makes it especially suitable for capture the underlying patterns in credit data. This section
today’s data-rich environment in finance. outlines the methodology for feature selection and engineering
techniques tailored to credit risk assessment tasks.
 Feature selection aims to identify the most relevant would be appropriate for inclusion in a scientific publication to
predictors from a large pool of potential features, reducing clearly convey the results of the empirical analysis.
model complexity and improving predictive performance. The
Fig. 2 summarizes the performance of the SVM model on
experiment is conducted step by step:
the testing dataset. Metrics include accuracy, precision, recall,
• Evaluate feature importance using statistical
F1-score, and area under the ROC curve (AUC).
measures such as correlation, mutual information, or
chi-square tests. Select features based on their
correlation with the target variable or their ability to
discriminate between different credit risk levels.
• Use machine learning models, such as recursive
feature elimination (RFE) or forward/backward
selection, to iteratively evaluate subsets of features
and identify the optimal feature subset that
maximizes predictive accuracy.
• Incorporate feature selection directly into the model
training process by leveraging algorithms with built-
in feature selection mechanisms, such as Lasso
regularization or tree-based methods like random
forest or gradient boosting.
Feature engineering involves creating new features or Fig. 2. Performance Metrics of The SVM on the German Credit Data Test
transforming existing ones to capture additional information Set
and improve the predictive power of credit scoring models.
The experiment is conducted step by step:
• Apply mathematical transformations, such as D. Comparative Analysis of SVM and Logistic Regression
logarithmic transformations or polynomial features,
Table 1 compares the SVM model to a baseline logistic
to numerical variables to capture nonlinear
regression model to highlight the improvement in predictive
relationships and improve model fit.
performance.
• Create interaction terms by combining pairs or higher
order combinations of features to capture synergistic TABLE I. COMPARATIVE PERFORMANCE OF SVM AND
effects or interactions between predictor variables. LOGISTIC REGRESSION MODELS
• Aggregate information across multiple variables to
create summary statistics or composite features, such Model Accuracy Precision Recall F1-Score AUC
as average credit utilization rate or total debt-to- Logistic Regression 70% 71% 63% 67% 0.74
income ratio, to provide a holistic view of borrower SVM (Linear Kernel) 75% 77% 68% 72% 0.79
behavior.
E. Feature Importance Analysis
• Incorporate temporal features, such as time since last
delinquency or frequency of recent credit inquiries, This table outlines the most significant features
to capture changes in borrower behavior over time contributing to the SVM model predictions, based on the
and improve the timeliness of credit risk predictions. magnitude of their coefficients in the linear SVM.
TABLE II
• Engineer features tailored to specific domains or
FEATURE IMPORTANCE BASED ON SVM COEFFICIENTS
business contexts, such as industry-specific risk
Feature Coefficient
indicators or behavioral proxies for creditworthiness,
Account Balance: No Account 1.25
to capture unique aspects of credit risk not captured
Duration of Credit (month) -0.88
by standard variables.
Payment Status of Previous Credit: No Problems 0.85
SVM offer a valuable tool for enhancing credit scoring
Purpose: New Car -0.75
models, providing a balance between complexity and
Credit Amount 0.70
performance, especially suitable for datasets where traditional
methods may falter due to dimensionality or non-linearity F. Discussion of Results
issues. Their application can lead to more robust, scalable, and
The results presented in these tables highlight the
effective credit scoring systems in the financial industry.
effectiveness of the SVM model in credit scoring. Figure 2
shows robust performance metrics, with a significant
C. Model Performance Metrics
improvement over traditional methods such as logistic
To effectively present the results of implementing an regression, as detailed in Table 1. Table 2’s insights into
SVM model on the “German Credit Data” for credit scoring feature importance are crucial for financial institutions as they
enhancement, we would include detailed data tables that provide actionable intelligence on which factors are most
outline the performance metrics, model comparison, and predictive of credit risk, allowing for better risk management
feature importance analyses. Below are example tables that
and decision-making processes. These tables collectively The adoption of SVM and other advanced machine
provide a comprehensive overview of the SVM learning techniques marks a pivotal shift towards more
implementation, demonstrating its viability and enhanced dynamic, precise, and reliable credit scoring systems. These
performance in credit scoring applications. This form of systems not only enhance the ability of financial institutions to
structured data presentation is critical in academic and assess borrower creditworthiness but also contribute to a
professional publications to support claims with empirical reduction in the risk of defaults. Moreover, the ability of SVM
evidence clearly and concisely. to effectively handle high-dimensional and diverse datasets
underscores their potential to revolutionize credit scoring
G. Area Under the Curve (AUC-ROC) Plot
processes by incorporating a broader range of data types and
The Area Under the Curve (AUC) graph, commonly sources. The study highlights the significant benefits of
depicted as a Receiver Operating Characteristic (ROC) curve, integrating artificial intelligence and machine learning into
is a fundamental tool in evaluating the performance of binary financial practices. Future research should continue to explore
classification models, including those used in credit scoring. It the scalability of these models and their implementation across
illustrates the trade-off between the True Positive Rate (TPR) different types of credit products to fully realize their potential
and the False Positive Rate (FPR) at various classification in enhancing global credit accessibility.
thresholds. In credit scoring, where the objective is to predict In conclusion, our findings advocate for a broader
whether a borrower will default on a loan based on certain integration of machine learning into credit scoring practices.
features, the ROC curve and its AUC hold significant Such integration not only addresses the limitations inherent in
importance: traditional models but also aligns with the ongoing digital
The ROC curve allows us to assess how well the model transformation in the financial sector. As the industry
discriminates between good and bad credit risks. A higher continues to evolve, it is imperative that credit scoring
TPR and lower FPR indicate better discrimination, suggesting methodologies keep pace, leveraging the power of machine
that the model is effective at identifying true positives (actual learning to foster more informed lending decisions and robust
defaults) while minimizing false positives (incorrectly labeled risk management strategies. Future work should explore the
as defaults). The ROC curve helps in selecting an appropriate scalability of these models and their application across
classification threshold based on the specific needs and different contexts and datasets, further solidifying the role of
constraints of the credit institution. For example, if the cost of machine learning in shaping the future of credit risk
missing a default (false negative) is high, the threshold can be assessment.
adjusted to prioritize sensitivity (TPR), sacrificing some
specificity (FPR). The AUC value serves as a quantitative REFERENCES
measure of model performance, allowing for direct
comparison between different models or variations of the same [1] E. Dumitrescu, S. Hue, C. Hurlin, and S. Tokpavi, “Machine learning
model. Higher AUC values indicate superior model for´ credit scoring: Improving logistic regression with non-linear
decisiontree effects,” European Journal of Operational Research, vol.
performance in distinguishing between good and bad credit 297, no. 3, pp. 1178–1192, 2022.
risks. By analyzing the shape of the ROC curve and the [2] E.-I. Dumitrescu, S. Hue, C. Hurlin,´ et al., “Machine learning or
corresponding AUC, lenders can gain insights into the risk econometrics for credit scoring: Let’s get the best of both worlds,” 2021.
characteristics of their credit portfolio. A steep ROC curve and [3] M. M. Amaro, Credit scoring: comparison of non-parametric techniques
a high AUC suggest that the model is capable of accurately against logistic regression. PhD thesis, Lisbon Portugal, 2020.
assessing credit risk across a wide range of scenarios, [4] T. Hu, W. Zhu, and Y. Yan, “Artificial intelligence aspect of
providing lenders with confidence in their decision-making transportation analysis using large scale systems,” in 2023 6th Artificial
Intelligence and Cloud Computing Conference (AICCC), pp. 54–59, 2023.
process. The ROC curve and AUC are valuable tools for
[5] X. Dastile, T. Celik, and M. Potsane, “Statistical and machine learning
model validation and ongoing monitoring. Changes in the models in credit scoring: A systematic literature survey,” Applied Soft
shape of the ROC curve or fluctuations in the AUC over time Computing, vol. 91, p. 106263, 2020.
may indicate shifts in the underlying credit landscape or the [6] D. Tripathi, A. K. Shukla, B. R. Reddy, G. S. Bopche, and D.
need for model recalibration. Chandramohan, “Credit scoring models using ensemble learning and
classification approaches: a comprehensive survey,” Wireless Personal
V. CONCLUSIONS Communications, vol. 123, no. 1, pp. 785–812, 2022.
[7] M. Schmitt, “Deep learning vs. gradient boosting: Benchmarking
This study provides a ML and Finance combined analysis stateof-the-art machine learning algorithms for credit scoring,” arXiv
of the enhancement of credit scoring models, with a particular preprint arXiv:2205.10535, 2022.
emphasis on the application of SVM. By comparing the [8] V. Zaytsev, “Selection and evaluation of relevant predictors for credit
performance of SVM to traditional methods like logistic scoring in peer-to-peer lending with random forest based methods,”
2020.
regression and decision trees, we have demonstrated that SVM
[9] C.-L. Huang, M.-C. Chen, and C.-J. Wang, “Credit scoring with a data
offer significant improvements in accuracy, precision, recall, mining approach based on support vector machines,” Expert systems
F1-Score, and AUC. Specifically, our results show that SVM with applications, vol. 33, no. 4, pp. 847–856, 2007.
achieve higher accuracy and superior performance metrics [10] A. Ruiz-Tagle Palazuelos, E. L. Droguett, and R. Pascual, “A novel deep
across the board, making them a more robust choice for capsule neural network for remaining useful life estimation,”
handling the complexities of credit scoring in the modern Proceedings of the Institution of Mechanical Engineers, Part O: Journal
of Risk and Reliability, vol. 234, no. 1, pp. 151–167, 2020.
financial landscape.
[11] Y. Yan, “Influencing factors of housing price in new york-analysis: [13] Z. Li, H. Yu, J. Xu, J. Liu, and Y. Mo, “Stock market analysis and
Based on excel multi-regression model,” 2022. prediction using lstm: A case study on technology stocks,” Innovations
[12] W. Zhu and T. Hu, “Twitter sentiment analysis of covid vaccines,” in in Applied Engineering and Technology, pp. 1–6, 2023.
2021 5th International Conference on Artificial Intelligence and Virtual
Reality (AIVR), pp. 118–122, 2021.