Intelligent imputation method for mix data-type missing values to improve data quality
Missing data is a widespread data quality issue across various domains. A common challenge is the occurrence of missing data during the data input process. Numerous studies have proposed methods to impute missing values for data across multiple fields. However, certain domains present unique chal...
| Main Author: | |
|---|---|
| Format: | Thesis |
| Language: | English |
| Published: |
2024
|
| Subjects: | |
| Online Access: | http://psasir.upm.edu.my/id/eprint/119916/ http://psasir.upm.edu.my/id/eprint/119916/1/119916.pdf |
| _version_ | 1848868080450535424 |
|---|---|
| author | Alabadla, Mustafa R. A. |
| author_facet | Alabadla, Mustafa R. A. |
| author_sort | Alabadla, Mustafa R. A. |
| building | UPM Institutional Repository |
| collection | Online Access |
| description | Missing data is a widespread data quality issue across various domains. A common
challenge is the occurrence of missing data during the data input process. Numerous
studies have proposed methods to impute missing values for data across multiple
fields. However, certain domains present unique challenges due to the involvement of
attributes from multiple scientific disciplines, such as biology, chemistry, and medical
which complicates the imputation process. Current machine learning models struggle
to handle both missing values and inaccuracies simultaneously, particularly when
dealing with large datasets. These challenges are further compounded by the data type
constraints imposed by these algorithms. Furthermore, most of the current approaches
focused on the imputation method alone without giving enough attention to the
cleansing and pre-processing phase which can be crucial for the imputation method
mechanism. Besides that, software tools for applying missing data imputation
approaches are limited. Hence, there is a need for the inclusion of intelligence
approaches in data imputation in the case of determining which independent variables
are the best set to impute missing values in dependent variables. To find optimum
variables, Machine Learning approach needs to be utilized. In this research, an
imputation approach using Extremely Randomized Trees (Extra Trees) of ensemble
machine learning methods named (ImputeX) is proposed. This method has the ability
to impute both categorical and continuous data features for large datasets. In addition,
an application is presented for public users to utilize the proposed method using
standard and autonomous data imputation. The proposed imputation method was
compared with existing imputation methods including MissForest, K-NNI,
HyperImpute, Multivariate Imputation by Chained Equations (MICE), Multiple
Imputation with Denoising Autoencoders (MIDAS), and SoftImpute. From these
results, it was observed that the proposed method improves the execution time by 35%
compared to recent imputation methods and increases the accuracy by 0.5% at 10%
missing ratio reaching 15% of accuracy improvement at 90% missing ratio. While the
presented application has achieved the best performance compared to current software
tools such as R package, Statistical Package for the Social Sciences (SPSS), Stata, and
Microsoft Excel. The significance of this research is to develop an intelligent method
that can deal with both missing values and accuracy in large datasets while minimizing
time consumed. Through the presentation of an accurate and reliable imputation
method, this research helps to improve data quality. Additionally, it contributes to data
science by improving the data cleaning procedure, which is a step in the data preprocessing
stage. |
| first_indexed | 2025-11-15T14:46:42Z |
| format | Thesis |
| id | upm-119916 |
| institution | Universiti Putra Malaysia |
| institution_category | Local University |
| language | English |
| last_indexed | 2025-11-15T14:46:42Z |
| publishDate | 2024 |
| recordtype | eprints |
| repository_type | Digital Repository |
| spelling | upm-1199162025-10-09T07:40:57Z http://psasir.upm.edu.my/id/eprint/119916/ Intelligent imputation method for mix data-type missing values to improve data quality Alabadla, Mustafa R. A. Missing data is a widespread data quality issue across various domains. A common challenge is the occurrence of missing data during the data input process. Numerous studies have proposed methods to impute missing values for data across multiple fields. However, certain domains present unique challenges due to the involvement of attributes from multiple scientific disciplines, such as biology, chemistry, and medical which complicates the imputation process. Current machine learning models struggle to handle both missing values and inaccuracies simultaneously, particularly when dealing with large datasets. These challenges are further compounded by the data type constraints imposed by these algorithms. Furthermore, most of the current approaches focused on the imputation method alone without giving enough attention to the cleansing and pre-processing phase which can be crucial for the imputation method mechanism. Besides that, software tools for applying missing data imputation approaches are limited. Hence, there is a need for the inclusion of intelligence approaches in data imputation in the case of determining which independent variables are the best set to impute missing values in dependent variables. To find optimum variables, Machine Learning approach needs to be utilized. In this research, an imputation approach using Extremely Randomized Trees (Extra Trees) of ensemble machine learning methods named (ImputeX) is proposed. This method has the ability to impute both categorical and continuous data features for large datasets. In addition, an application is presented for public users to utilize the proposed method using standard and autonomous data imputation. The proposed imputation method was compared with existing imputation methods including MissForest, K-NNI, HyperImpute, Multivariate Imputation by Chained Equations (MICE), Multiple Imputation with Denoising Autoencoders (MIDAS), and SoftImpute. From these results, it was observed that the proposed method improves the execution time by 35% compared to recent imputation methods and increases the accuracy by 0.5% at 10% missing ratio reaching 15% of accuracy improvement at 90% missing ratio. While the presented application has achieved the best performance compared to current software tools such as R package, Statistical Package for the Social Sciences (SPSS), Stata, and Microsoft Excel. The significance of this research is to develop an intelligent method that can deal with both missing values and accuracy in large datasets while minimizing time consumed. Through the presentation of an accurate and reliable imputation method, this research helps to improve data quality. Additionally, it contributes to data science by improving the data cleaning procedure, which is a step in the data preprocessing stage. 2024-05 Thesis NonPeerReviewed text en http://psasir.upm.edu.my/id/eprint/119916/1/119916.pdf Alabadla, Mustafa R. A. (2024) Intelligent imputation method for mix data-type missing values to improve data quality. Doctoral thesis, Universiti Putra Malaysia. http://ethesis.upm.edu.my/id/eprint/18488 Machine learning Data cleaning (Data processing) Missing data (Statistics) |
| spellingShingle | Machine learning Data cleaning (Data processing) Missing data (Statistics) Alabadla, Mustafa R. A. Intelligent imputation method for mix data-type missing values to improve data quality |
| title | Intelligent imputation method for mix data-type missing values to improve data quality |
| title_full | Intelligent imputation method for mix data-type missing values to improve data quality |
| title_fullStr | Intelligent imputation method for mix data-type missing values to improve data quality |
| title_full_unstemmed | Intelligent imputation method for mix data-type missing values to improve data quality |
| title_short | Intelligent imputation method for mix data-type missing values to improve data quality |
| title_sort | intelligent imputation method for mix data-type missing values to improve data quality |
| topic | Machine learning Data cleaning (Data processing) Missing data (Statistics) |
| url | http://psasir.upm.edu.my/id/eprint/119916/ http://psasir.upm.edu.my/id/eprint/119916/ http://psasir.upm.edu.my/id/eprint/119916/1/119916.pdf |