Although Support Vector Machines have been used to develop highly accurate classification and regression models in various real-world problem domains, the most significant barrier is that SVM generates black box model that is difficult to understand. The procedure to convert these opaque models into transparent models is called rule
extraction. This thesis investigates the task of extracting comprehensible models from trained SVMs, thereby alleviating this limitation. The primary contribution of the thesis is the proposal of various algorithms to overcome the significant limitations of SVM by
taking a novel approach to the task of extracting comprehensible models. The basic contribution of the thesis are systematic review of literature on rule extraction from SVM, identifying gaps in the literature and proposing novel approaches for addressing the gaps.
The contributions are grouped under three classes, decompositional, pedagogical and eclectic/hybrid approaches. Decompositional approach is closely intertwined with the internal workings of the SVM. Pedagogical approach uses SVM as an oracle to re-label training examples as well as artificially generated examples. In the eclectic/hybrid approach, a combination of these two methods is adopted.
The thesis addresses various problems from the finance domain such as bankruptcy prediction in banks/firms, churn prediction in analytical CRM and Insurance fraud detection. Apart from this various benchmark datasets such as iris, wine and WBC for classification problems and auto MPG, body fat, Boston housing, forest fires and pollution for regression problems are also tested using the proposed appraoch. In addition, rule extraction from unbalanced datasets as well as from active learning based approaches has been explored. For classification problems, various rule extraction methods such as FRBS, DT, ANFIS, CART and NBTree have been utilized. Additionally for regression problems, rule extraction methods such as ANFIS, DENFIS and CART have also been employed. Results are analyzed using accuracy, sensitivity, specificity, fidelity, AUC and t-test measures. Proposed approaches demonstrate their viability in extracting accurate, effective and comprehensible rule sets in various benchmark and real world problem domains across classification and regression problems. Future directions have been indicated to extend the approaches to newer variations of SVM as well as to other problem domains.
Table of Contents
Chapter 1: Introduction to Rule Extraction
1.1 Rule Extraction: Motivation
1.2 Rule Extraction: Significance
1.2.1 Provision of User Explanation Capability
1.2.2 Transparency
1.2.3 Data Exploration
1.3 Rule Extraction: A Taxonomy
1.3.1 Decompositional Approaches
1.3.2 Pedagogical Approaches
1.3.3 Eclectic/Hybrid Approaches
1.4 Rule Quality Criteria
1.5 Thesis Overview
1.6 Datasets Used
1.7 Experimental Setup
1.8 Thesis Outline
Chapter 2: Rule Extraction from SVM: An Introduction
2.1 Rule Extraction from SVM
2.1.1 Decompositional Approaches
2.1.2 Pedagogical Approaches
2.1.3 Eclectic/Hybrid Approaches
2.2 Gaps Identified in Literature
2.3 Outline of the Proposed Approaches
Chapter 3: Fuzzy Rules Extraction using SVM for solving Bankruptcy prediction in banks problem
3.1 Motivation
3.2 Proposed Fuzzy Rule Extraction Technique
3.2.1 Extraction of Support Vectors from SVM
3.2.2 Rule Generation
3.3 Literature Review of Bankruptcy Prediction in Banks and Firms
3.4 Results and Discussions
3.5 Conclusions
Chapter 4: Rule Extraction from SVM using Feature Selection for Solving Classification and Regression Problems
4.1 Motivation
4.2 Proposed Rule Extraction Approach
4.2.1 Feature Selection using SVM-RFE
4.2.2 Building SVM/SVR models
4.2.3 Rule Generation
4.3 Problems Analyzed
4.3.1 Auto MPG Dataset
4.3.2 Body Fat Dataset
4.3.3 Boston Housing Dataset
4.3.4 Forest Fires Dataset
4.3.5 Pollution Dataset
4.4 Results and Discussions
4.4.1 Classification Problems
4.4.2 Regression Problems
4.4.3 Overall Observations
4.5 Conclusions
Chapter 5: Rule Extraction from SVM for Data Mining on Unbalanced Datasets
5.1 Motivation
5.2 Customer Relationship Management (CRM)
5.3 Churn Prediction Problem
5.4 Proposed Eclectic/Hybrid Rule Extraction Approach
5.4.1 Feature Selection using SVM-RFE
5.4.2 Support Vector Extraction using SVM
5.4.3 Rule Generation using NBTree
5.5 Dataset Description
5.6 Data Imbalance Problem
5.6.1 Literature Review of Techniques Dealing with Unbalanced Data
5.6.2 Random Under-Sampling
5.6.3 Random Over-Sampling
5.6.4 SMOTE (Synthetic Minority Over-sampling Technique)
5.7 Results and Discussions
5.8 Conclusions
Chapter 6: Modified Active Learning Based Approach for Rule Extraction from SVM
6.1 Motivation
6.2 Modified Active Learning Based Approach for Rule Extraction
6.2.1 Feature Selection Phase
6.2.2 Active Learning Phase
6.2.3 Rule Generation Phase
6.3 Finance Application Analyzed
6.3.1 Fraud Detection in Automobile Insurance Dataset
6.3.2 Pre-Processing
6.3.3 Literature Survey of Fraud Detection Problems
6.4 Results and Discussions
6.4.1 Churn Prediction using SVM+NBTree
6.4.2 Insurance Fraud Detection using SVM+NBTree
6.4.3 Churn Prediction using SVM+DT
6.4.4 Insurance Fraud Detection using SVM+DT
6.4.5 Overall Observations
6.5 Conclusions
Chapter 7: Rule Extraction from SVR for Solving Regression Problems
7.1 Motivation
7.2 Proposed Eclectic/Hybrid Rule Extraction Technique
7.2.1 Extraction of Support Vectors and SVR Predictions
7.2.2 Rule Generation
7.3 Problems Analyzed
7.4 Results and Discussions
7.5 Conclusions
Chapter 8: Overall Conclusions and Future Directions
Objective & Research Scope
This thesis addresses the "black box" limitation of Support Vector Machines (SVMs) by proposing novel algorithms for rule extraction. The primary objective is to convert opaque, highly accurate SVM models into transparent, comprehensible models (e.g., if-then rules, decision trees) to provide better interpretability, particularly for domain-specific applications like bankruptcy prediction, customer churn, and insurance fraud detection.
- Systematic literature review on rule extraction from SVM and identification of research gaps.
- Development of decompositional, pedagogical, and eclectic/hybrid rule extraction methodologies.
- Application of proposed techniques to real-world finance and benchmark datasets.
- Implementation of feature selection (SVM-RFE) to improve rule quality and comprehensibility.
- Evaluation using metrics such as accuracy, sensitivity, specificity, fidelity, and AUC.
Auszug aus dem Buch
1.2 Rule Extraction: Significance
Andrews et al. (1995) describe the motivation behind rule extraction from neural networks. A brief review of the arguments of Andrews et al. (1995) will help to establish aims and significance for rule extraction from SVM techniques.
1.2.1 Provision of user explanation capability
In symbolic artificial intelligence (AI), the term “explanation” refers to an explicit structure which can be used internally for reasoning and learning, externally for the explanation of results to the user. Gallant (1988) observes that an explanation capability enables a novice user to gain insights into the problem at hand. Davis et al. (1977) argues that even limited explanation can positively influence acceptance of the system by the user. Traditionally, researchers have experimented with various forms of user explanation, in particular rule traces. It is obvious that explanations based on rule traces are too rigid and inflexible (Gilbert, 1989) because rules may not be equally useful to the user. Further, the granularity of the rule traces’ explanation is often inappropriate (Gilbert, 1989; Andrews et al., 1995).
1.2.2 Transparency
The creation of a “user explanation” capability is the primary objective for extracting rules from neural networks and SVMs, with the provision of “transparency” of the internal states of a system. Transparency means that internal states of the machine learning system are both accessible and can be interpreted unambiguously. Such capability is mandatory if neural network or SVM based solutions are to be accepted into “safety-critical” problem domains such as air traffic control, operations of power plants, medical diagnosis, etc (Andrews et al., 1995).
Summary of Chapters
Chapter 1: Introduction to Rule Extraction: Discusses the need for model transparency in machine learning, introduces the taxonomy of rule extraction approaches, and outlines the thesis goals.
Chapter 2: Rule Extraction from SVM: An Introduction: Provides a comprehensive literature review and categorizes SVM rule extraction into decompositional, pedagogical, and eclectic/hybrid groups.
Chapter 3: Fuzzy Rules Extraction using SVM for solving Bankruptcy prediction in banks problem: Presents a decompositional approach to extract fuzzy if-then rules for bank bankruptcy prediction.
Chapter 4: Rule Extraction from SVM using Feature Selection for Solving Classification and Regression Problems: Explores a pedagogical approach using SVM-RFE for feature selection to improve rule quality for both classification and regression.
Chapter 5: Rule Extraction from SVM for Data Mining on Unbalanced Datasets: Introduces an eclectic approach specifically for handling medium-scale, unbalanced data in customer churn prediction.
Chapter 6: Modified Active Learning Based Approach for Rule Extraction from SVM: Proposes a modified active learning method (MALBA) to improve rule extraction performance in unbalanced finance applications.
Chapter 7: Rule Extraction from SVR for Solving Regression Problems: Extends the eclectic rule extraction framework to Support Vector Regression (SVR) models.
Chapter 8: Overall Conclusions and Future Directions: Summarizes the thesis findings and suggests potential future research areas such as incremental SVM learning.
Keywords
Support Vector Machines, Rule Extraction, Bankruptcy Prediction, Customer Churn, Insurance Fraud, Data Mining, Unbalanced Datasets, Feature Selection, SVM-RFE, Fuzzy Logic, Decision Trees, NBTree, Machine Learning, Interpretability, Regression Problems.
Frequently Asked Questions
What is the primary motivation for this research?
The main motivation is that SVMs, while highly accurate, act as "black box" models. This makes them difficult to understand in domains where transparency is crucial, such as bankruptcy prediction and fraud detection.
What are the main approaches used for rule extraction?
The thesis utilizes three primary categories: decompositional (intertwined with internal model workings), pedagogical (treating the model as a black box), and eclectic/hybrid (a combination of both).
What is the primary objective of this work?
The goal is to develop algorithms that extract comprehensible, symbolic, and high-fidelity descriptions from trained SVM and SVR models without compromising their predictive power.
Which scientific methods are applied in this thesis?
The work employs several techniques including Fuzzy Rule Based Systems (FRBS), Decision Trees (DT), CART, ANFIS, DENFIS, and Naive-Bayes Trees (NBTree), alongside SVM-RFE for feature selection.
What topics are covered in the main section of the thesis?
The thesis covers bankruptcy prediction in banks, customer churn prediction in analytical CRM, insurance fraud detection, and general regression analysis using extracted rules.
Which keywords best characterize this work?
The key themes include Support Vector Machines, rule extraction, finance domain applications, data imbalance, and model transparency.
How does the author handle unbalanced datasets?
The research employs several balancing techniques, including random undersampling, random oversampling, combined undersampling/oversampling, and the SMOTE technique.
What role does feature selection play in this research?
Feature selection using SVM-RFE is used to reduce data dimensionality, which helps in generating smaller, more interpretable rule sets and improving model performance.
- Citar trabajo
- Mohammed Farquad (Autor), 2010, Rule Extraction from Support Vector Machine, Múnich, GRIN Verlag, https://www.grin.com/document/193686
