Improving accuracy of estimating glomerular filtration rate using artificial neural network: model development and validation
Journal of Translational Medicine volume 18, Article number: 120 (2020)
The performance of previously published glomerular filtration rate (GFR) estimation equations degrades when directly used in Chinese population. We incorporated more independent variables and using complicated non-linear modeling technology (artificial neural network, ANN) to develop a more accurate GFR estimation model for Chinese population.
The enrolled participants came from the Third Affiliated Hospital of Sun Yat-sen University, China from Jan 2012 to Jun 2016. Participants with age < 18, unstable kidney function, taking trimethoprim or cimetidine, or receiving dialysis were excluded. Among the finally enrolled 1952 participants, 1075 participants (55.07%) from Jan 2012 to Dec 2014 were assigned as the development data whereas 877 participants (44.93%) from Jan 2015 to Jun 2016 as the internal validation data. We in total developed 3 GFR estimation models: a 4-variable revised CKD-EPI (chronic kidney disease epidemiology collaboration) equation (standardized serum creatinine and cystatin C, age and gender), a 9-variable revised CKD-EPI equation (additional auxiliary variables: body mass index, blood urea nitrogen, albumin, uric acid and hemoglobin), and a 9-variable ANN model.
Compared with the 4-variable equation, the 9-variable equation could not achieve superior performance in the internal validation data (mean of difference: 5.00 [3.82, 6.54] vs 4.67 [3.55, 5.90], P = 0.5; interquartile range (IQR) of difference: 18.91 [17.43, 20.48] vs 20.11 [18.46, 21.80], P = 0.05; P30: 76.6% [73.7%, 79.5%] vs 75.8% [72.9%, 78.6%], P = 0.4), but the 9-variable ANN model significantly improve bias and P30 accuracy (mean of difference: 2.77 [1.82, 4.10], P = 0.007; IQR: 19.33 [17.77, 21.17], P = 0.3; P30: 80.0% [77.4%, 82.7%], P < 0.001).
It is suggested that using complicated non-linear models like ANN could fully utilize the predictive ability of the independent variables, and then finally achieve a superior GFR estimation model.
Glomerular flirtation rate (GFR) has been well recognized as the best overall indicator of kidney function, which is widely used in diagnosis, treatment and prognosis of chronic kidney disease (CKD) . GFR can be measured by renal or serum clearance of exogenous filtration markers such as inulin and iohexol, but the so-called measured GFR (mGFR) values are cumbersome and costly to be derived in clinical routine. Therefore, investigators have developed widely used GFR estimation equations using established filtration markers (e.g., serum creatinine and cystatin C) in association with demographical variables (e.g., age, gender and race) [2,3,4,5,6,7]. The global organization kidney disease: improving global outcomes (KDIGO) has recommended to use estimated GFR (eGFR) as the initial test in clinical practice and epidemiological survey . By 2017, many countries have been reporting eGFR with serum creatinine measurement .
The most accepted eGFR equations are modification of diet in renal disease (MDRD)  and chronic kidney disease epidemiology collaboration (CKD-EPI) equations , which can provide acceptable GFR estimates for the North American population. However, these eGFR estimations may not perform well among Chinese population, as these equations were not developed based on Chinese population . Therefore, studies have been conducted to develop accurate equations for Chinese or Asian population . However, most of these studies focus on either establishing an ethnic factor  or developing a new equation just using traditional regression method.
In the development of GFR estimation equations, the standard procedures are using natural logarithm transformation of mGFR and filtration markers, then using ordinary least square linear or piecewise linear regression. This simple linearity might not explain the complicated relationship among kidney function, GFR and filtration markers [1, 12]. Moreover, the potential predictive power of auxiliary variables (demographical variables and other laboratory test variables) was not sufficiently utilized, as no interaction terms were incorporated into the equations. Studies have shown that using complicated non-linear modeling technology may improve the performance of GFR estimation [13,14,15,16]. Therefore, we used artificial neural network (ANN), a powerful and common methodology in machine learning, to develop a more accurate eGFR model for Chinese population, and validated this model and compared its performance with standard regression equation models.
Study design and study participants
Patients diagnosed with CKD in the Third Affiliated Hospital of Sun Yat-sen University during January 2012 to June 2016 were recruited consecutively into this study. Participates were excluded for any of the following reasons: (1) age < 18 years; (2) having acute kidney function deterioration, skeletal muscle atrophy, edema, pleural effusion or ascites, heart failure, malnutrition, amputation, or ketoacidosis; (3) taking trimethoprim or cimetidine; or (4) receiving dialysis at the time of study.
The institutional review board at the Third Affiliated Hospital of Sun Yat-sen University approved this study. A written informed consent was obtained from all participants.
GFR was measured by 99mTc-DTPA renal dynamic imaging, which had been recalibrated to a dual plasma sample 99mTc-DTPA GFR. Renal dynamic imaging was obtained with a Millennium TMMPR SPECT using the General Electric Medical System (Discovery VH, GE Healthcare). Serum samples from each participant were collected on the same day of performing GFR measurement and assayed on a Hitachi 7180 auto-analyzer (Hitachi reagents from Roche Diagnostics) in a single laboratory at the Department of Laboratory in the Third Affiliated Hospital of Sun Yet-sun University. Creatinine was measured by an enzymatic method and then recalibrated to the isotope dilution mass spectrometry reference method [17, 18]. We also recalibrated serum cystatin C to the standard reference material (ERM-DA471) . The laboratory test variables were extracted from the analysis report and recorded manually.
Development of revised CKD-EPI equations and ANN model
The revised equations were derived using the same method for developing the CKD-EPI equation by Inker and colleagues . We first developed an equation for GFR estimation using a combination of conventional 4 variables including age, sex, serum creatinine (Scr) and serum cystatin C (Scys), then we further developed a 9-variable equation by incorporating 5 more auxiliary variables including body mass index (BMI), blood urea nitrogen (BUN), albumin (ALB), uric acid (UA) and hemoglobin (HGB). For both equations the dependent variable was mGFR. mGFR and independent variables Scr, Scys, and BUN were log-transformed, so the correlation between mGFR and the independent variables became nearly linear. We developed the equations with 4- and 9-variable which fit the piecewise linear splines with a knot of both Scr and Scys by using splines Package in R software (version 3.5.0, R Development Core Team). The method for determining the knot of spline of Scr and Scys was described in Additional file 1.
We also developed an ANN model with the same 9 independent variables for GFR estimation. Prior to ANN development, we performed data cleaning and pre-processing on the development data, including outlier deleting and variable normalization. We used only 1 hidden layer with 4 neurons, and the activation functions in all hidden neurons were set as Leaky ReLU (alpha = 0.1) . The ANN was trained by Stochastic gradient descent (SGD) optimizer, and the whole development of ANN was implemented under Keras framework in Python (version 3.6.6, Python Software Foundation). The detailed ANN model development was described in Additional file 2.
Model evaluation and statistical analysis
The performance indicators of GFR estimation include bias, precision and accuracy Bias and precision were defined as the median and the interquartile range (IQR) of the difference of eGFR minus mGFR, respectively. Accuracy was assessed as P30 (percentage of eGFR within ± 30% of mGFR). Besides the model evaluation on overall cohort, we also performed the identical evaluation procedures on subgroups divided by mGFR. Data from patients from Jan 2015 to Jun 2016 were used for internal validation on the performance of the derived models. We also performed a sensitivity analysis by developing and internally validating the 3 GFR estimation models based on random split datasets.
Complete-case analysis was used to handle the missing data. The 95% confidence intervals were calculated with bootstrap methods (2000 bootstraps) [21,22,23]. Wilcoxon signed rank test was used to compare the bias between models, whereas Permutation test for comparison of precision, and McNemar test for comparison of P30. All statistical analysis was performed using MATLAB software (version 2018b, MathWorks).
Characteristics of participants
Among the initially enrolled 2997 CKD patients during 2012 and 2016, 970 with incomplete data and 75 with irregular recordings were excluded (details are available in Fig. 1). Finally, 1952 participants were included in the model development or validation, including 1075 (55.1%) participants from Jan 2012 to Dec 2014 assigned into the development dataset to derive the revised equations and ANN, whereas 877 (44.9%) from Jan 2015 to Jun 2016 assigned as the internal validation dataset to independently evaluate the performance of the derived models.
Table 1 summarizes the characteristics of both development and internal validation datasets. For the development dataset, 57.3% were male; mean age was 55.6 years (standard deviation [SD] 14.5); mean mGFR was 71.0 (SD, 27.4) mL/min/1.73 m2, serum creatinine 1.5 (SD, 1.3) mg/dL, and serum cystatin C 1.5 (SD, 0.9) mg/L. For internal validation dataset, 59.0% were male; mean age was 57.4 (SD, 13.4) years; mGFR was 68.8 (27.1) mL/min/1.73 m2, serum creatinine 1.3 (SD, 0.9) mg/dL, and serum cystatin C 1.3 (SD, 0.7) mg/L. There were few participants with mGFR less than 30 mL/min/1.73 m2 in both development and internal validation dataset (6.5% and 6.3% respectively).
Formulation of revised CKD-EPI equations and ANN model
The knots of serum creatinine for female and male participants were 0.7 and 0.9 mg/dL, respectively, whereas the knots of serum cystatin C were both 0.9 mg/L. The formulations of revised 4-variable and 9-variable CKD-EPI equations were shown in Table 2. Additional file 3 shows how to implement the 9-variable ANN model.
Performance of models in the internal validation dataset
The performance of three derived models was summarized in Table 3. The bias (median difference) between mGFR and eGFR of the revised 4-variable CKD-EPI equation is 4.67 [95% CI 3.55–5.90], the precision 20.11 [18.46–21.80] mL/min/1.73 m2, and the accuracy (or P30) 75.8% [72.9–78.6%].
Compared with the revised 4-variable CKD-EPI equation, the 9-variable equation has similar bias (5.00 [3.82–6.54] mL/min/1.73 m2, P = 0.5) and P30 (76.6% [73.7–79.5%], P = 0.4), and a slightly better precision (18.91 [17.43–20.48] mL/min/1.73 m2, P = 0.05).
The bias of 9-variable ANN model is 2.77 [1.82–4.10] mL/min/1.73 m2, which is much smaller than that of 4-variable revised CKD-EPI equation (P = 0.007). The P30 of ANN model is 80.0% [77.4–82.7%], significantly higher than the two equation (P < 0.001). However, it is similar in precision between ANN and the 4-variable equation (P = 0.3, see Table 3).
The model performance in subgroups by mGFR was similar with the overall performance. In both subgroups of mGFR < 60 mL/min/1.73 m2 and mGFR ≥ 60 mL/min/1.73 m2, the 9-variable ANN model consistently achieved superior P30 than the two revised equations. However, in subgroup of mGFR ≥ 90 mL/min/1.73 m2 the ANN model tended to be more biased (median of difference − 2.91 [− 4.60 to − 1.32] mL/min/1.73 m2) (Table 3).
The sensitivity analysis based on random split datasets showed that the 9-variable ANN model has significantly superior P30 and precision and similar bias compared with the 4-variable CKD-EPI equation (see Additional file 4).
Accurate evaluation of GFR is important for assessing the severity of CKD, predicting prognosis and deciding proper therapeutic interventions. Since publication of Cockcroft-Gault (CG) Equation in 1976 , many studies have been conducted to derive actionable models to estimate GFR. The major barrier of accurately estimate individual’s GFR is non-GFR determinants of filtration markers [1, 12, 24, 25], which degrade the ideal linear correlation between GFR and filtration markers. Under the consideration of cost and convenience, such unmeasured non-GFR determinants are unable to be incorporated into the GFR estimation models, instead auxiliary variables (demographical variables and other laboratory test variables) are used as surrogates. The frequently used demographical variables are age, gender and race, whereas the frequently used other laboratory test variables are blood urea nitrogen and albumin.
However, other laboratory test variables in the linear equations seem to have limited predictive ability to estimate GFR compared with filtration markers and demographical variables. The 6-variable MDRD equation has two additional variables Serum urea nitrogen and Albumin than the simplified 4-variable MDRD equation, but the performance of the two equations are nearly the same [3,4,5]. In the development of CKD-EPI equation in 2012, no other laboratory test variables or interaction terms are incorporated into the final equation as their predictive ability are not statistically significant during variable selection .
In our study, we developed two revised CKD-EPI equations. One equation incorporated 4 variables: standardized serum creatinine and cystatin C, age and gender, which are the standard variable combination during developing the GFR estimation model. We further incorporated more auxiliary variables as in theory it is beneficial using more independent variables when developing prediction models. Besides blood urea nitrogen and albumin, we also incorporated body mass index, uric acid and hemoglobin, and finally developed a 9-variable revised CKD-EPI equation. However, the two revised equations turned out to have similar performance of GFR estimation. The reason behind this phenomenon is the simple linear regression cannot sufficiently utilize the potential predictive power of these auxiliary variables. When we used the same 9 variables to develop a ANN model, compared with the revised 4-variable CKD-EPI equation, the 9-variable ANN model significantly reduce bias and improve P30 accuracy.
The mathematical theory of ANN is the universal approximation theorem [26, 27], which means that ANN is able to approximate any continuous even uncontinuous functions. When the network size of ANN increases, the capacity of ANN will become more powerful. Furthermore, ANN doesn’t require any assumptions about distribution of variables and can handle with the multi-collinearity among independent variables . Therefore, ANN can capture not only the complicated correlations between GFR and independent variables, but also any interactions between independent variables, so it can make GFR estimations based on these sophisticated relationships.
In future GFR estimation studies, it is a major trend to incorporated more variables into GFR estimation models, such as potential novel filtration markers β-Trace Protein (BTP) and β2-Microglobulin (B2M) [29,30,31], and even biomarkers from proteomics and metabolomics [30,31,32]. Our study suggests that it is beneficial to use complicated models to fully utilize the predictive ability of these variables to achieve a good performance of GFR estimation.
There are limitations in our study. First, all study participants were from one medical center in China, and most are CKD patients. The generalizability of the study may be limited to CKD patients, and the performance of the developed ANN still requires extra validation on diverse populations. Second, the gold standard mGFR was measured by 99mTc-DTPA renal dynamic imaging, and then recalibrated to a dual plasma sample 99mTc-DTPA GFR. It is widely accepted that using iohexol or iothalamate will achieve a more accurate mGFR compared with 99mTc-DTPA . Third, the sizes of development dataset as well as internal validation dataset are relatively small, especially there were few participants with mGFR ≤ 30 mL/min/1.73 m2. Fourth, although ANN model is superior in the accuracy, it is difficult to interpret, and the relationship and interaction between independent variables are still unknown.
In conclusion, we introduced up to 9 variables into GFR estimation and developed revised CKD-EPI 4-variable and 9-variable equations, and a 9-variable ANN model, respectively. Compared with the 4-variable equation, the 9-variable equation could not achieve superior performance, but the 9-variable ANN model significantly reduces bias and improve P30 accuracy. It is suggested that using complicated non-linear models like ANN could fully utilize the predictive ability of the additional auxiliary variables. However, the proposed ANN model still requires extra and careful validation in more diverse cohort data.
Availability of data and materials
The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.
Chronic kidney disease
Glomerular filtration rate
Measured glomerular filtration rate
Estimated glomerular filtration rate
Kidney disease: improving global outcomes
Modification of diet in renal disease
Chronic kidney disease epidemiology collaboration
Artificial neural network
Body mass index
Serum cystatin C
Blood urea nitrogen
Levey AS, Inker LA. GFR as the “Gold Standard”: estimated, measured, and true. Am J Kidney Dis. 2016;67:9–12.
Cockcroft DW, Gault MH. Prediction of creatinine clearance from serum creatinine. Nephron. 1976;16:31–41.
Levey AS, Bosch JP, Lewis J, et al. A more accurate method to estimate glomerular filtration rate from serum creatinine: a new prediction equation. Ann Intern Med. 1999;130:461–70.
Levey AS, Greene T, Kusek JW, Beck GJ. A simplified equation to predict glomerular filtration rate from serum creatinine. J Am Soc Nephrol. 2000;11:155A.
Levey AS, Coresh J, Greene T, et al. Using standardized serum creatinine values in the modification of diet in renal disease study equation for estimating glomerular filtration rate. Ann Intern Med. 2006;145:247–54.
Levey AS, Stevens LA, Schmid CH, et al. A new equation to estimate glomerular filtration rate. Ann Intern Med. 2009;150:604–12.
Inker LA, Schmid CH, Tighiouart H, Eckfeldt JH, Feldman HI, Greene T, Kusek JW, Manzi J, Van Lente F, Zhang YL, et al. Estimating glomerular filtration rate from serum creatinine and cystatin C. N Engl J Med. 2012;367:20–9.
group KdIgoKCw. KDIGO 2012 clinical practice guideline for the evaluation and management of chronic kidney disease. Kidney Int Suppl. 2012;2013(3):1–150.
Miller WG, Jones GRD. Estimated glomerular filtration rate; laboratory implementation and current global status. Adv Chronic Kidney Dis. 2018;25:7–13.
Delanaye P, Cavalier E, Mariat C, Krzesinski J-M, Rule AD. Estimating glomerular filtration rate in Asian subjects: where do we stand? Kidney Int. 2011;80:439–40.
Li DY, Yin WJ, Yi YH, Zhang BK, Zhao J, Zhu CN, Ma RR, Zhou LY, Xie YL, Wang JL, Zuo SR. Development and validation of a more accurate estimating equation for glomerular filtration rate in a Chinese population. Kidney Int. 2019;95(3):636–46.
Levey AS, Inker LA, Coresh J. GFR estimation: from physiology to public health. Am J Kidney Dis. 2014;63:820–34.
Liu X, Chen YR, Li NS, Wang C, Lv LS, Li M, Wu XM, Lou TQ. Estimation of glomerular filtration rate by a radial basis function neural network in patients with type-2 diabetes mellitus. BMC Nephrol. 2013;14(1):181.
Liu X, Pei X, Li N, Zhang Y, Zhang X, Chen J, Lv L, Ma H, Wu X, Zhao W, Lou T. Improved glomerular filtration rate estimation by an artificial neural network. PLoS ONE. 2013;8:e58242.
Liu X, Li NS, Lv LS, Huang JH, Tang H, Chen JX, Ma HJ, Wu XM, Lou TQ. A comparison of the performances of an artificial neural network and a regression model for GFR estimation. Am J Kidney Dis. 2013;62(6):1109–15.
Liu X, Li N, Lv L, Fu Y, Cheng C, Wang C, Ye Y, Li S, Lou T. Improving precision of glomerular filtration rate estimating model by ensemble learning. J Transl Med. 2017;15:231.
Myers GL, Miller WG, Coresh J, Fleming J, Greenberg N, Greene T, Hostetter T, Levey AS, Panteghini M, Welch M, Eckfeldt JH. Recommendations for improving serum creatinine measurement: a report from the laboratory working group of the national kidney disease education program. Clin Chem. 2006;52:5–18.
Ceriotti F, Boyd JC, Klein G, Henny J, Queraltó J, Kairisto V, Panteghini M. Reference intervals for serum creatinine concentrations: assessment of available data for global application. Clin Chem. 2008;54:559–66.
Grubb A, Blirup-Jensen S, Lindström V, Schmidt C, Althaus H, Zegers I. First certified reference material for cystatin C in human serum ERM-DA471/IFCC. Clin Chem Lab Med. 2010;48:1619–21.
Maas AL, Hannun AY, Ng AY: Rectifier nonlinearities improve neural network acoustic models. In: ICML (International Conference on Machine Learning), vol 3; Atlanta, Georgia, USA. 2013.
Efron B, Tibshirani RJ. An introduction to the bootstrap. Boca Raton: CRC Press; 1994.
Davison AC, Hinkley DV. Bootstrap methods and their application. Cambridge: Cambridge University Press; 1997.
Carpenter J, Bithell J. Bootstrap confidence intervals: when, which, what? A practical guide for medical statisticians. Stat Med. 2000;19:1141–64.
Van Biesen W, Nagler EV. A Swiss army knife for estimating kidney function: why new equations will not solve the real problem. Nephrol Dial Transplant. 2016;31:685–7.
Stevens LA, Coresh J, Greene T, Levey AS. Assessing kidney function—measured and estimated glomerular filtration rate. N Engl J Med. 2006;354:2473–83.
Cybenko G. Approximation by superpositions of a sigmoidal function. Math Control Signals Systems. 1989;2:303–14.
Csáji BC. Approximation with artificial neural networks. Fac Sci Etvs Lornd Univ Hung. 2001;24:48.
Dreiseitl S, Ohno-Machado L. Logistic regression and artificial neural network classification models: a methodology review. J Biomed Inform. 2002;35:352–9.
Inker LA, Tighiouart H, Coresh J, Foster MC, Anderson AH, Beck GJ, Contreras G, Greene T, Karger AB, Kusek JW, et al. GFR estimation using β-trace protein and β2-microglobulin in CKD. Am J Kidney Dis. 2016;67:40–8.
Filler G, Huang S-HS, Lindsay RM. The search for more reliable estimated GFR biomarkers. Am J Kidney Dis. 2016;67:5–8.
Karger AB, Inker LA, Coresh J, Levey AS, Eckfeldt JH. Novel filtration markers for GFR estimation. EJIFCC. 2017;28:277–88.
Inker LA, Levey AS, Coresh J. Estimated glomerular filtration rate from a panel of filtration markers-hope for increased accuracy beyond measured glomerular filtration rate? Adv Chronic Kidney Dis. 2018;25:67–75.
Soveri I, Berg UB, Björk J, Elinder C-G, Grubb A, Mejare I, Sterner G, Bäck S-E. Measuring GFR: a systematic review. Am J Kidney Dis. 2014;64:411–24.
The authors want to thanks all the participants and clinical staffs for their cooperation.
This study was supported by grants from the National Key R&D Program of China (Grant No. 2018YFC1315400, 2018YFC0910500), the National Natural Science Foundation of China (Grant No.81873631, 81370866, 81070612), the China Postdoctoral Science Foundation (Grant No. 201104335), the Third Affiliated Hospital of Sun Yat-Sen University, Clinical Research Program (Grant No.YHJH201806), the Neil Shen’s SJTU Medical Research Fund, SJTU-Yale Collaborative Research Seed Fund, Science and Technology Commission of Shanghai Municipality (Grant No. 17DZ22512000).
Ethics approval and consent to participate
The institutional review board at the Third Affiliated Hospital of Sun Yat-sen University approved this study. A written informed consent was obtained from all participants.
Consent for publication
All the authors declared no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Li, N., Huang, H., Qian, HZ. et al. Improving accuracy of estimating glomerular filtration rate using artificial neural network: model development and validation. J Transl Med 18, 120 (2020). https://doi.org/10.1186/s12967-020-02287-y