Development and validation of a prediction model with missing predictor data: a practical approach

doi:10.1016/j.jclinepi.2009.03.017

Journal of Clinical Epidemiology

Volume 63, Issue 2, February 2010, Pages 205-214

https://doi.org/10.1016/j.jclinepi.2009.03.017 Get rights and content

Abstract

Objective

To illustrate the sequence of steps needed to develop and validate a clinical prediction model, when missing predictor values have been multiply imputed.

Study Design and Setting

We used data from consecutive primary care patients suspected of deep venous thrombosis (DVT) to develop and validate a diagnostic model for the presence of DVT. Missing values were imputed 10 times with the MICE conditional imputation method. After the selection of predictors and transformations for continuous predictors according to three different methods, we estimated regression coefficients and performance measures.

Results

The three methods to select predictors and transformations of continuous predictors showed similar results. Rubin's rules could easily be applied to estimate regression coefficients and performance measures, once predictors and transformations were selected.

Conclusion

We provide a practical approach for model development and validation with multiply imputed data.

Introduction

Interest in multivariable prediction models for diagnostic and prognostic research has grown over the past decade. Prediction models enable physicians explicitly to convert combinations of multiple predictor values to an estimated absolute risk of disease presence (in case of diagnosis) or the occurrence of a disease-related event (in case of prognosis). Prediction models are developed with data of patients from a development set, often using multivariable regression analysis. The models are accordingly validated in new, similar patients (validation set) [1], [2].

Missing observations are almost universally encountered in clinical data sets, no matter how strictly studies have been designed or how hard investigators try to prevent them. The easiest way to deal with missing values is to exclude all patients with a missing value on any of the considered variables. Such a complete case analysis may sacrifice useful information and may cause biased results [3], [4]. Imputation based on observed patient characteristics (conditional imputation) has been advocated to deal with the missing values [3]. To take the uncertainty of the imputed values into account, missing values should be imputed multiple (m) times, for which several iterative algorithms are available. The resulting m completed data sets are each analyzed separately by standard methods and the m results are combined into one final point estimate and variance, with the standard error equal to the square root of the variance [3], [5].

Combining the m results is straightforward, when a single analysis is considered. The m point estimates are averaged and the m variances can be combined taking the variability between the m data sets into account with a components-of-variance argument (Rubin's rules) [3].

The development of a prediction model follows a sequence of steps [6], including selection of predictors, selection of transformations for continuous predictors [7], [8], and estimation of the regression coefficients. Hence, model development with multiply imputed data is not straightforward and seldom illustrated. Here, we demonstrate the development and validation of a prediction model obtained with logistic regression in the presence of multiply imputed data. Continuous predictors are modeled with transformations if necessary. In the second model, the continuous predictors are dichotomized. Further, three different methods to select predictors and transformations are applied. We encountered also another practical problem, typical with real life data. Two continuous predictors were recorded partly as dichotomous and partly as continuous. We impute the continuous values by using the observed value for the dichotomous variable and the distribution of the continuous variable where available. Two empirical data sets on the diagnosis of deep venous thrombosis (DVT) are used with minor to major percentages of missing predictor values, one data set to develop the model [9] and one to validate the model [10].

Section snippets

Empirical data

We used the data of 2,086 consecutive primary care patients suspected of DVT. The data originated from a large cross-sectional diagnostic study that was performed between January 1, 2002 and January 1, 2006 among over 100 primary care physicians in The Netherlands. For specific details and main results of the diagnostic study, we refer to the literature [9], [10]. In brief, suspicion of DVT was based on swelling, redness, or pain of the lower extremities. Information was systematically

Model development in general

When developing a prediction model, various issues and choices need to be addressed. We discuss briefly three common steps in the development of prediction models. First, the number of candidate predictors is commonly too large to include them all in the prediction model. The data to hand can be used to select predictors, for instance with a backward elimination procedure.

Second, the shape of the relation of continuous predictors with the outcome variable can be studied with nonlinear functions

Model development

We examine first only patients with complete cases and then the completed data using multiple imputation. We illustrate the model development methods in the presence of multiply imputed data as described in Section 3.2 with the data sets on the diagnosis of DVT. We consider a model that could contain continuous predictors that are modeled with the MFP algorithm and a second model with only dichotomous predictors. Continuous predictors (calf circumference, d-dimer, age, and duration of symptoms)

Discussion

Missing data are commonplace in clinical studies. The main message we wish to bring out here is that good statistical methods are available to enable credible, practical analyses of such data sets. It is often unclear from reports whether a prediction model was developed or validated in the presence of missing data. Authors usually ignore cases with missing observations and perform complete case analyses. More recently, awareness has been growing of the usefulness of multiple imputation

References (41)

K.G. Moons et al.
Using the outcome for imputation of missing predictor values was preferred
J Clin Epidemiol
(2006)
E.W. Steyerberg et al.
Internal validation of predictive models: efficiency of some procedures for logistic regression analysis
J Clin Epidemiol
(2001)
T.G. Clark et al.
Developing a prognostic model in the presence of missing data: an ovarian cancer case study
J Clin Epidemiol
(2003)
A.C. Justice et al.
Assessing the generalizability of prognostic information
Ann Intern Med
(1999)
D.G. Altman et al.
What do we mean by validating a prognostic model?
Stat Med
(2000)
D.B. Rubin
Multiple imputation for non response in surveys
(1987)
R.A. Little
Regression with missing X's; a review
J Am Stat Assoc
(1992)
J.L. Schafer
Analysis of incomplete multivariate data
(1997)
F.E. Harrell et al.
Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors
Stat Med
(1996)
F.E. Harrell et al.
Regression modelling strategies for improved prognostic prediction
Stat Med
(1984)

W. Sauerbrei et al.

Building multivariable prognostic and diagnostic models: transformation of the predictors by using fractional polynomials

J R Stat Soc A

(1999)

R. Oudega et al.

Ruling out deep venous thrombosis in primary care. A simple diagnostic algorithm including D-dimer testing

Thromb Haemost

(2005)

D. Toll et al.

A new diagnostic rule for deep vein thrombosis: safety and efficiency in clinically relevant subgroups

Fam Pract

(2008)

J.L. Schafer

Multiple imputation: a primer

Stat Methods Med Res

(1999)

S. van Buuren et al.

Multiple imputation of missing blood pressure covariates in survival analysis

Stat Med

(1999)

P. Royston et al.

Regression using fractional polynomials of continuous covariates: parsimonious parametric modelling (with discussion)

Appl Stat

(1994)

F.E. Harrell

Regression modeling strategies. With applications to linear models, logistic regression, and survival analysis

(2001)

D.J. Spiegelhalter

Probabilistic prediction in patient management and clinical trials

Stat Med

(1986)

J.B. Copas

Regression, prediction and shrinkage

J R Stat Soc B

(1983)

H.C. van Houwelingen et al.

Construction, validation and updating of a prognostic model for kidney graft survival

Stat Med

(1995)

Cited by (207)

Developing and externally validating multinomial prediction models for methotrexate treatment outcomes in patients with rheumatoid arthritis: results from an international collaboration
2024, Journal of Clinical Epidemiology
In rheumatology, there is a clinical need to identify patients at high risk (>50%) of not responding to the first-line therapy methotrexate (MTX) due to lack of disease control or discontinuation due to adverse events (AEs). Despite this need, previous prediction models in this context are at high risk of bias and ignore AEs. Our objectives were to (i) develop a multinomial model for outcomes of low disease activity and discontinuing due to AEs 6 months after starting MTX, (ii) update prognosis 3-month following treatment initiation, and (iii) externally validate these models.
A multinomial model for low disease activity (submodel 1) and discontinuing due to AEs (submodel 2) was developed using data from the UK Rheumatoid Arthritis Medication Study, updated using landmarking analysis, internally validated using bootstrapping, and externally validated in the Norwegian Disease-Modifying Antirheumatic Drug register. Performance was assessed using calibration (calibration-slope and calibration-in-the-large), and discrimination (concordance-statistic and polytomous discriminatory index).
The internally validated model showed good calibration in the development setting with a calibration-slope of 1.01 (0.87, 1.14) (submodel 1) and 0.83 (0.30, 1.34) (submodel 2), and moderate discrimination with a c-statistic of 0.72 (0.69, 0.74) and 0.53 (0.48, 0.59), respectively. Predictive performance decreased after external validation (calibration-slope 0.78 (0.64, 0.93) (submodel 1) and 0.86 (0.34, 1.38) (submodel 2)), which may be due to differences in disease-specific characteristics and outcome prevalence.
We addressed previously identified methodological limitations of prediction models for outcomes of MTX therapy. The multinomial approach predicted outcomes of disease activity more accurately than AEs, which should be addressed in future work to aid implementation into clinical practice.
The novel SALT-M score predicts 1-year post-transplant mortality in patients with severe acute-on-chronic liver failure
2023, Journal of Hepatology
Twenty-eight-day mortality ranges from 30-90% in patients with acute-on-chronic liver failure grades 2/3 (severe ACLF). Though liver transplantation (LT) has demonstrated a survival benefit, the scarcity of donor organs and uncertainty regarding post-LT mortality among patients with severe ACLF may cause hesitancy. We developed and externally validated a model to predict 1-year post-LT mortality in severe ACLF, called the Sundaram ACLF-LT-Mortality (SALT-M) score, and estimated the median length of stay (LoS) after LT (ACLF-LT-LoS).
In 15 LT centers in the US, we retrospectively identified a cohort of patients with severe ACLF transplanted between 2014-2019, followed up to Jan’2022. Candidate predictors included demographics, clinical and laboratory values, and organ failures. We selected predictors in the final model using clinical criteria and externally validated them in two French cohorts. We provided measures of overall performance, discrimination, and calibration. We used multivariable median regression to estimate LoS after adjusting for clinically relevant factors.
We included 735 patients, of whom 521 (70.8%) had severe ACLF (120 ACLF-3, external cohort). The median age was 55 years, and 104 with severe ACLF (19.9%) died within 1-year post-LT. Our final model included age >50 years, use of 1/≥2 inotropes, presence of respiratory failure, diabetes mellitus, and BMI (continuous). The c-statistic was 0.72 (derivation) and 0.80 (validation), indicating adequate discrimination and calibration based on the observed/expected probability plots. Age, respiratory failure, BMI, and presence of infection independently predicted median LoS.
The SALT-M score predicts mortality within 1-year after LT in patients with ACLF. The ACLF-LT-LoS score predicted median post-LT stay. Future studies using these scores could assist in determining transplant benefits.
Liver transplantation (LT) may be the only life-saving procedure available to patients with acute-on-chronic liver failure (ACLF), but clinically instability can augment the perceived risk of post-transplant mortality at 1 year. We developed a parsimonious score with clinically and readily available parameters to objectively assess 1-year post-LT survival and predict median length of stay after LT. We developed and externally validated a clinical model called the Sundaram ACLF-LT-Mortality score in 521 US patients with ACLF with 2 or ≥3 organ failure(s) and 120 French patients with ACLF grade 3. The c-statistic was 0.72 in the development cohort and 0.80 in the validation cohort. We also provided an estimation of the median length of stay after LT in these patients. Our models can be used in discussions on the risks/benefits of LT in patients listed with severe ACLF. Nevertheless, the score is far from perfect and other factors, such as patient’s preference and center-specific factors, need to be considered when using these tools.
Vital signs-based deterioration prediction model assumptions can lead to losses in prediction performance
2023, Journal of Clinical Epidemiology
Vital signs-based models are complicated by repeated measures per patient and frequently missing data. This paper investigated the impacts of common vital signs modeling assumptions during clinical deterioration prediction model development.
Electronic medical record (EMR) data from five Australian hospitals (1 January 2019–31 December 2020) were used. Summary statistics for each observation's prior vital signs were created. Missing data patterns were investigated using boosted decision trees, then imputed with common methods. Two example models predicting in-hospital mortality were developed, as follows: logistic regression and eXtreme Gradient Boosting. Model discrimination and calibration were assessed using the C-statistic and nonparametric calibration plots.
The data contained 5,620,641 observations from 342,149 admissions. Missing vitals were associated with observation frequency, vital sign variability, and patient consciousness. Summary statistics improved discrimination slightly for logistic regression and markedly for eXtreme Gradient Boosting. Imputation method led to notable differences in model discrimination and calibration. Model calibration was generally poor.
Summary statistics and imputation methods can improve model discrimination and reduce bias during model development, but it is questionable whether these differences are clinically significant. Researchers should consider why data are missing during model development and how this may impact clinical utility.
External validation and updating of prognostic models for predicting recovery of disability in people with (sub)acute neck pain was successful: broad external validation in a new prospective cohort
2023, Journal of Physiotherapy
Can existing post-treatment prognostic models for predicting neck pain recovery (primarily in terms of disability and secondarily in terms of pain intensity and perceived improvement) be externally validated and updated at the end of the treatment period and at 6 and 12 weeks of follow-up in a new Dutch cohort of people with neck pain treated with guideline-based usual care physiotherapy?
External validation and model updating in a new prospective cohort of three previously developed prognostic models.
People with (sub)acute neck pain and registered for primary care physiotherapy treatment.
Recovery of disability, pain intensity, and perceived recovery at 6 and 12 weeks and at the end of the treatment period.
Discriminative performance (c-statistic) of the disability model at 6 weeks was 0.73 (95% CI 0.69 to 0.77) and reasonably well calibrated after intercept recalibration. The disability model at 12 weeks and at the end of the treatment period showed discriminative c-statistic performance values of 0.69 (95% CI 0.64 to 0.73) and 0.68 (95% CI 0.63 to 0.72), respectively, and was well calibrated. Pain models and perceived recovery models did not reach acceptable performance. Cervical mobility added value to the disability models and pain catastrophising to the disability and pain models at 6 weeks.
Broad external validation of the disability model was successful in people with (sub)acute neck pain and clinicians may use this model in clinical practice with reasonable accuracy. Further research is required to assess the disability model’s clinical impact and generalisability, and to identify additional valuable model predictors.
https://osf.io/a6r3k/
Prognostic score and sex-specific nomograms to predict survival in resectable lung cancer: A French nationwide study from the Epithor cohort database
2023, The Lancet Regional Health - Europe
Prognostic assessment in patients undergoing cancer treatments is of paramount importance to plan subsequent management. In resectable lung cancer availability of an easy-to use nomogram to predict long-term outcome would be extremely useful to identify high-risk patients in the era of perioperative targeted and immune therapies.
We retrieved clinical, surgical and pathological data of all consecutive patients included in Epithor, the database of French Society of Thoracic and Cardiovascular Surgery, and operated on between 2003 and 2020 for non-small cell lung cancer in a curative intent. The primary endpoint was overall survival up to 5 years. We assessed prognostic significance of available variables using Cox modelling, in the whole dataset, and in men and in women separately, and performed temporal validation. Finally, we constructed two sex-specific nomograms. Survivals by fifths of score were assessed in the development and temporal validation sets.
The study included 62,633 patients (43,551 men and 19,082 women). Median survival time was 9.2 years. Nine factors had strong prognostic impact and were used to construct nomograms. The optimism-corrected c statistic for the prognostic score was 0.689 in the development sample, and 0.726 (95% CI 0.718–0.735) in the temporal validation sample. All differences between adjacent fifths of score were significant (P < 0.0001). Figures of 3-year OS by fifths of score were 92.2%, 83.0%, 74.3%, 64.0%, and 43.4%, respectively, in the development set and 93.3%, 88.4%, 81.0%, 73.7%, 55.7% in the temporal validation set. Performance of score was maintained when stratifying by stage of diseases.
In the present work, we report evidence that long-term overall survival after resection of NSCLC can be predicted by an easy to construct and use composite score taking into account both host and tumour related factors.
Epithor is funded by FSTCVS.
Development and validation of a prediction model for self-reported mobility decline in community-dwelling older adults
2022, Journal of Clinical Epidemiology
Citation Excerpt :
Nonlinearity between continuous predictors and the outcome was examined using fractional polynomials before multivariable modeling. Missing data were imputed using multiple imputation by chained equation technique [18,19] (Appendix 3). Fifty imputed datasets were created.
The aim of this study is to develop and validate two models to predict 2-year risk of self-reported mobility decline among community-dwelling older adults.
We used data from a prospective cohort study of people aged 65 years and over in England. Mobility status was assessed using the EQ-5D-5L mobility question. The models were based on the outcome: Model 1, any mobility decline at 2 years; Model 2, new onset of persistent mobility problems over 2 years. Least absolute shrinkage and selection operator logistic regression was used to select predictors. Model performance was assessed using C-statistics, calibration plot, Brier scores, and decision curve analyses. Models were internally validated using bootstrapping.
Over 18% of participants who could walk reported mobility decline at year 2 (Model 1), and 7.1% with no mobility problems at baseline, reported new onset of mobility problems after 2 years (Model 2). Thirteen and 6 out of 31 variables were selected as predictors in Models 1 and 2, respectively. Models 1 and 2 had a C-statistic of 0.740 and 0.765 (optimism < 0.013), and Brier score = 0.136 and 0.069, respectively.
Two prediction models for mobility decline were developed and internally validated. They are based on self-reported variables and could serve as simple assessments in primary care after external validation.

View all citing articles on Scopus

: This work is supported by the Netherlands Organization for Scientific Research Grant ZON-MW 917.46.360 (Y. Vergouwe and K.G.M. Moons); UK Medical Research Council (P. Royston); and Cancer Research UK (D.G. Altman).

View full text

Original ArticleDevelopment and validation of a prediction model with missing predictor data: a practical approach

Abstract

Objective

Study Design and Setting

Results

Conclusion

Introduction

Section snippets

Empirical data

Model development in general

Model development

Discussion

J Clin Epidemiol

J Clin Epidemiol

J Clin Epidemiol

Assessing the generalizability of prognostic information

Ann Intern Med

What do we mean by validating a prognostic model?

Stat Med

Multiple imputation for non response in surveys

Regression with missing X's; a review

J Am Stat Assoc

Analysis of incomplete multivariate data

Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors

Stat Med

Regression modelling strategies for improved prognostic prediction

Stat Med

Building multivariable prognostic and diagnostic models: transformation of the predictors by using fractional polynomials

J R Stat Soc A

Ruling out deep venous thrombosis in primary care. A simple diagnostic algorithm including D-dimer testing

Thromb Haemost

A new diagnostic rule for deep vein thrombosis: safety and efficiency in clinically relevant subgroups

Fam Pract

Multiple imputation: a primer

Stat Methods Med Res

Multiple imputation of missing blood pressure covariates in survival analysis

Stat Med

Regression using fractional polynomials of continuous covariates: parsimonious parametric modelling (with discussion)

Appl Stat

Regression modeling strategies. With applications to linear models, logistic regression, and survival analysis

Probabilistic prediction in patient management and clinical trials

Stat Med

Regression, prediction and shrinkage

J R Stat Soc B

Construction, validation and updating of a prognostic model for kidney graft survival

Stat Med

Original Article
Development and validation of a prediction model with missing predictor data: a practical approach