A Novel Performance Metric for Building an Optimized Classifier
Problem statement: Typically, the accuracy metric is often applied for optimizing the heuristic or stochastic classification models. However, the use of accuracy metric might lead the searching process to the sub-optimal solutions due to its less discriminating values and it is also not robust to...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Science Publications
2011
|
| Subjects: | |
| Online Access: | http://ir.unimas.my/id/eprint/3432/ http://ir.unimas.my/id/eprint/3432/1/A%20Novel%20Performance%20Metric%20for%20Building%20an%20Optimized%20Classifier.pdf |
| Summary: | Problem statement: Typically, the accuracy metric is often applied for optimizing the
heuristic or stochastic classification models. However, the use of accuracy metric might lead the
searching process to the sub-optimal solutions due to its less discriminating values and it is also not
robust to the changes of class distribution. Approach: To solve these detrimental effects, we propose a
novel performance metric which combines the beneficial properties of accuracy metric with the
extended recall and precision metrics. We call this new performance metric as Optimized Accuracy
with Recall-Precision (OARP). Results: In this study, we demonstrate that the OARP metric is
theoretically better than the accuracy metric using four generated examples. We also demonstrate
empirically that a naïve stochastic classification algorithm, which is Monte Carlo Sampling (MCS)
algorithm trained with the OARP metric, is able to obtain better predictive results than the one trained
with the conventional accuracy metric. Additionally, the t-test analysis also shows a clear advantage of
the MCS model trained with the OARP metric over the accuracy metric alone for all binary data sets.
Conclusion: The experiments have proved that the OARP metric leads stochastic classifiers such as
the MCS towards a better training model, which in turn will improve the predictive results of any
heuristic or stochastic classification models. |
|---|