Machine Learning for Early Detection of Chronic Diseases Using Clinical Data

Abstract

Chronic diseases like type 2 diabetes (T2D), chronic kidney disease (CKD) and cardiovascular disease (CVD) are insidious, insidious diseases that develop over years, and if they are diagnosed early enough, lifestyle and/or drug intervention can prevent irreversible damage. This paper presents and tests machine learning models based on routinely collected information from the electronic health record (EHR) to predict incident T2D, CKD, and CVD within 3 years. A retrospective cohort of 48,320 adults was divided into training (70%), validation (10%) and test (20%) sets, with 62 clinical features being extracted from the 2 years before. The logistic regression, random forest, gradient boosting (XGBoost), multilayer perceptron and stacked ensemble were compared to a rule-based baseline. The best test AUROC is 0.884 for T2D, 0.903 for CKD, and 0.823 for CVD for gradient boosting, and 0.004 to 0.005 on top of that for the stacked ensemble. The gradient-boosting model identified 65.8% of cases with T2D, 71.1% with CKD, and 51.2% with CVD with 90% specificity, and retained the same level of sensitivity when the prediction time interval is increased from 6 to 36 months. Using isotonic calibration reduced the expected calibration error from 0.031 to 0.007. SHAP analysis revealed three key features that drove the risk of T2D, namely HbA1c, fasting glucose and BMI, which, when removed, resulted in a loss of 0.093 AUROC. There was a lower level of performance at community clinics (0.861) than at urban clinics (0.889): this demonstrates the need to monitor subgroups. A notion of a deployment workflow with drift monitoring is proposed. These findings indicate that routine data may be used for early warning screening but further prospective research is needed before clinical implementation.