IDENTIFICATION AND ANALYSIS OF CONFOUNDING VARIABLES AND SIMPSON’S PARADOX - Chattopadhyay, Ishani (2022)
This dissertation harnesses machine learning algorithms and model agnostic tools to explore the counter intuitive relationship between protein intake from legumes and pass rate in Malawi. This dissertation focuses on an exploratory analysis to study approaches towards creating sub-groups based on K-means clustering algorithm in order to identify Simpson’s Paradox. The curious case of negative relationship between protein intake from legumes and pass rate in Malawi, has been addressed through identification of confounders by harnessing logistic regression and chi-square tests. Random Forest Model and Partial Dependency Plots have been utilised to study the relationship between protein intake from legumes and pass rates by creating sub-groups of the confounders in order to isolate the effect of these confounders. This dissertation follows a waterfall method that dives deeper into identification of confounders whenever a sub group indicates a negative relationship between legumes and pass rates. The dissertation tries to answer certain trends in the relationship and possible ways to understand the problem. The analysis helps identify areas that could be explored further in order to provide better amenities to improve the standard of living of the poorer areas in Malawi.