Assessing Markers From Ambulatory Laboratory Tests for Predicting High-Risk Patients

Study Design: Observational study of a patient cohort over 2 years.

Methods: We used administrative claims and EHR data over a 2-year period for a population of continuously insured patients in an integrated health system who had at least 1 ambulatory visit during the first year. We performed regression tree analyses to develop risk markers from frequently ordered outpatient laboratory tests. We added these risk markers to demographic and Charlson Comorbidity Index models and 3 models from the Johns Hopkins Adjusted Clinical Groups system to predict individual cost, inpatient admission, and high-cost patients. We evaluated the predictive and discriminatory performance of 5 lab-enhanced models.

Results: Our study population included 120,844 patients. Adding laboratory markers to base models improved R2 predictions of costs by 0.1% to 3.7%, identification of high-cost patients by 3.4% to 121%, and identification of patients with inpatient admissions by 1.0% to 188% for the demographic model. The addition of laboratory risk markers to comprehensive risk models, compared with simpler models, resulted in smaller improvements in predictive power.

Conclusions: The addition of laboratory risk markers can significantly improve the identification of high-risk patients using models that include age, gender, and a limited number of morbidities; however, models that use comprehensive risk measures may be only marginally improved.

Am J Manag Care. 2018;24(6):e190-e195Takeaway Points

  • Laboratory tests that are frequently ordered by physicians in outpatient practices contain valuable data for individual risk assessment.
  • Ranges of blood chemistries and hematology results define a set of model markers that have clinical face validity and potential utility for care management.
  • Adding the laboratory-based markers to risk levels derived from claims, prescriptions, and enrollment data improves the prediction of individual cost, the prediction of inpatient admission, and the prospective identification of high-cost patients.
  • For practices, a simple model that includes demographics and laboratory information may provide a basic tool to evaluate patient panels.

Most predictive models in healthcare have relied upon diagnosis information from health insurance claims or other administrative data. Such claims-based predictive models have been used extensively by health plans and government agencies for provider profiling and payment, underwriting, and prioritizing patients for care management.1 Although claims remain an important source of risk data, the widespread implementation of electronic health records (EHRs) and other clinical information technology systems offers a new source of data on disease severity and health status, as most EHRs contain information not captured in claims, such as laboratory values, vital signs, and clinical assessments.2

In the inpatient setting, laboratory tests have been used to assess the risk of mortality across a range of conditions, including acute myocardial infarction, congestive heart failure, diabetes, ischemic and hemorrhagic stroke, pneumonia, and septicemia.3-7 These predictive assessments of mortality risk have incorporated blood chemistries, hematology, and blood gases. Predictive models for mortality performed better after adding laboratory risk markers, but similar models predicting 30-day readmission did not improve as much.8

Another case for laboratory data has been made for case-mix adjustment of inpatient admissions using diagnosis-related groups (DRGs).9,10 Clinical laboratory results combined with inpatient administrative data incrementally improved the ability of DRGs to explain the length of inpatient stays; however, Medicare Severity DRGs and other DRG versions do not incorporate laboratory data for inpatient classification.

Laboratory tests can be powerful predictors among certain patient populations. For example, patients with diabetes who maintained reduced glycated hemoglobin (A1C) levels (ie, had better glycemic control) had lower annual costs than patients with higher levels.11

The goal of this study was to develop and evaluate an approach for transforming common outpatient laboratory tests into risk measures that could be useful when added to population-level predictive models. Our objective was to determine result ranges for several candidate blood tests that were associated with increased costs in the year after the tests were performed. We hypothesized that certain ranges of component results from blood tests in the base year would be associated with higher healthcare costs and increased inpatient utilization during the subsequent year. We also hypothesized that laboratory risk markers based on component ranges would improve predictive risk models for these outcomes, including models with demographic and Charlson Comorbidity Index (CCI) risk markers and 3 models from the Johns Hopkins Adjusted Clinical Groups (ACG) system.


Data Source and Study Population

We obtained data from HealthPartners, Inc (Bloomington, Minnesota), a health insurer and large integrated delivery system. Its database contains structured EHR data, including encounter diagnoses and laboratory test results; administrative data that included benefit eligibility files; and claims data with International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) diagnoses, Current Procedural Terminology (CPT) procedure codes from inpatient and outpatient settings, and filled prescriptions with National Drug Codes from outpatient pharmacies. HealthPartners provided these data for patients who were receiving care at facilities owned by the healthcare system.

Our study population included 120,844 patients who were continuously enrolled in 2012 and 2013 and had at least 1 visit to 1 of 5 HealthPartners outpatient clinics in the Minneapolis-St. Paul metropolitan area in 2012.

Data Preparation

To harmonize the coding of test orders across HealthPartners’ entities, we mapped the internal HealthPartners codes to Logical Observation Names Identifiers and Codes (LOINC). LOINC is a common language for identifying health measurements, observations, and documents, and it is commonly used for laboratory orders and findings.12

The assignment of LOINC was a 2-step process. We first used the Regenstrief LOINC Mapping Assistant to suggest potential LOINC, which were turned over to a pathologist for final review in the second step.13 All laboratory tests that we selected for this study were mapped to LOINC.

Selection of Laboratory Tests and Creation of Risk Markers

We identified 23 blood chemistries and hematology counts from 4 test panels (ie, the basic metabolic panel, lipid panel, liver function tests, complete blood counts) and extracted A1C, alanine aminotransferase (ALT), albumin, alkaline phosphatase (ALK), aspartate aminotransferase (AST), bicarbonate, blood urea nitrogen (BUN), calcium, chloride, creatinine, glucose, hematocrit, hemoglobin, high-density lipoprotein cholesterol (HDL-C), low-density lipoprotein cholesterol (LDL-C), platelet, potassium, sodium, total bilirubin, total cholesterol, total protein, triglycerides, and white blood cell results from the EHR data. Our literature review suggested that these tests are commonly ordered in office-based clinical practices, and our study population confirmed this theory, as 49% had at least 1 result for any of the 23 tests in 2012. We extracted CPT codes for test orders from claims submitted to HealthPartners and confirmed that essentially all results for the tests of interest were present in the EHR data. We compared results with reference ranges for healthy persons,14 manually excluded implausible results that were extremely far outside the reference ranges, and selected the most recent results to create patient-level risk markers. Patient-level annual costs were calculated from claims incurred in 2013.

We used a 3-step process to develop laboratory-based risk markers. First, we conducted a regression tree analysis for each of the 23 tests to determine result ranges that were prospectively associated with increased annual costs using the caret package in R software.15 A strategy of individually analyzing laboratory covariates has been used to discriminate the risk of inpatient mortality.16 We minimized the impact of high-cost claimants by truncating individual costs at $250,000, which was equivalent to the 99.9th percentile in the study population. Second, several regression tree analyses generated multiple result ranges, and we condensed them into ordinal “low” and “high” levels to create binary markers. High levels of risk consisted of low test results, high values, or both. To prevent model overfitting, we required the high-risk groups to contain at least 1% of patients who had results for a test. Third, we created binary markers for low and high risk levels. Tested patients had either a low- or a high-risk marker assigned to their test results. High risk indicated a potential for high cost in the future period; low risk indicated that a patient’s condition was nonsevere or under control or that the test was performed for diagnostic reasons. Patients who did not have tests had all markers set to 0 so that we were able to evaluate the joint impact of laboratory-based risk markers in the entire patient population.

Outcome Measures and Predictive Models

Our prospective outcome measures were individual total annual claims costs, presence of inpatient hospitalization, and high-risk patient status, defined as being among the top 5% of claimants in 2013. We developed 5 predictive models for these 3 outcomes.

The demographic model included age and gender only; the Charlson model included age, gender, and 17 morbidity categories from the CCI, which is widely used to predict high-cost patients.16 The ACG system contributed 3 predictive models; all included morbidity markers based on diagnoses, and 1 included morbidity markers that were derived from prescription data.17 We first combined the demographic variables with 32 Aggregated Diagnosis Groups (ADGs; ie, types of morbidity from the ACG system) into an ADG model. Other researchers have validated a similar ADG model for predicting mortality.18 Second, we implemented the ACG-Dx predictive model with age, gender, 24 selected ACGs, and 116 Expanded Diagnosis Clusters (EDCs) for the patient population.19 ACGs are mutually exclusive groups that are based on patterns of ADG morbidity types and have similar resource use; EDCs are clinical groupings of diseases based on ICD-9-CM and International Classification of Diseases, Tenth Revision, Clinical Modification codes. A patient may have multiple ADGs and EDCs. Third, we implemented an ACG-DxRx model that contained all markers from the Dx version and included 65 Rx-MGs (ie, morbidity types based on prescription data). ACG-PM predictive models have been validated for predicting high-risk patients and inpatient hospitalization.1,20

Statistical Analyses

We used the health system’s data to generate individual risk scores. The models were run with and without the laboratory markers included. Our objective was to evaluate the contribution of new laboratory markers to the performance of predictive models with different levels of complexity as indicated by the number of morbidity markers.

We calibrated models for costs using ordinary least squares (OLS) and generalized linear regression and chose to report the coefficient of determination (R2) for OLS models. Models for inpatient hospitalization and high-risk outcomes were calibrated using logistic regression, and we computed sensitivity, specificity, area under the receiver operating characteristic curve (AUC), and integrated discrimination improvement (IDI) statistics to quantify the improvement in discriminatory performance due to laboratory markers. We included IDI because the original models that included ACG system variables showed high AUC values; therefore, meaningful improvements in discriminatory power might not be captured by measuring only AUC. The properties of the IDI statistic are well understood, and this statistic is increasingly used to evaluate markers that are introduced into predictive models.21,22


Characteristics of the Study Population

The study population consisted of 120,844 patients who had at least 1 ambulatory care visit at a HealthPartners-owned clinic in 2012. The mean (SD) age was 37.6 (19.2) years, 20.6% were younger than 18 years, 3.3% were 65 years or older, and 57.1% were women (Table 1). Almost all patients (99.9%) had at least 1 type of morbidity recorded, with an average (SD) of 5.9 (3.4) ADGs, and 20.0% had at least 1 comorbid condition that was included in the CCI. Mean (SD) patient total annual claims costs were $5732 ($20,208), and 5.1% were admitted to a hospital in 2013.

We extracted test orders from all outpatient medical service claims and measured laboratory data completeness. A1C results were 92% complete; calcium results, 96% complete; and all other test results, more than 98% complete.

Risk Associated With Laboratory Results

We determined whether any result ranges for the candidate tests were associated with increased costs in the year after the tests were performed. Our analysis indicated separations between “low-cost” and “high-cost” risk groups for 12 of the 23 tests. These 12 tests included sodium, chloride, bicarbonate, glucose, and calcium from the basic metabolic panel; total protein and albumin from the liver function tests; hemoglobin, hematocrit, and platelets from the complete blood count; and total cholesterol and LDL-C from the lipid panel. The other 11 laboratory tests—A1C, ALT, ALK, AST, BUN, creatinine, HDL-C, potassium, total bilirubin, triglycerides, and white blood cell results—did not show an association with low- or high-cost risk and were excluded from the predictive modeling.

High-risk group sizes ranged from 1% of patients with hematocrit results to 21% of patients with albumin results. The average costs in these 2 groups were $32,695 and $24,234, respectively. Other high-risk groups showed similar increased average annual costs. The cost separation between high- and low-risk groups ranged from $4943 for LDL-C to $23,492 for hematocrit (Table 2).

Predictive Model Performance Improvement With Added Laboratory Markers

Laboratory markers increased the prospective R2 of the demographic model for costs more than 2-fold from 2.2% to 5.9%; the IDI measures for inpatient and top-cost claimant identification were 121% and 188%, respectively (Table 3). For Charlson models, the R2 for cost increased from 10.3% to 11.4%, and the identification of inpatients and top-cost claimants, as measured by the IDIs, improved by 40% and 14%, respectively.

Overall, ACG-PM models exhibited higher prospective R2 values and showed less improvement with added laboratory markers compared with the demographic and Charlson models. The ACG-Dx and ACG-DxRx models predicted 22.2% and 24.7% of cost variation, respectively. Laboratory markers added small improvements to predicting costs across all 3 ACG system models; the R2 improvements ranged from 0.1% to 0.6%.

The lab-enhanced ADG model had an AUC of 0.820 and an IDI of 4.8% for the identification of top-cost claimants. Similarly, lab-enhanced ACG-Dx and ACG-DxRx models had AUCs (IDIs) of 0.835 (1.5%) and 0.847 (1.0%) for high-risk identification, respectively. For hospitalization predictions, the AUCs across lab-enhanced ACG system models ranged from 0.789 (ADG) to 0.799 (ACG-DxRx); IDIs ranged from 8.4% for the ADG model to 3.4% for the ACG-DxRx model.


We developed high-cost risk markers using commonly ordered outpatient laboratory test results and evaluated how these markers improved individual predictions of healthcare costs, hospitalization, and high-risk status. This analysis extends previous research that used laboratory test results to predict clinical outcomes, such as mortality and hospital admission.3-8,16 We explored the potential value of these new commonly available clinical data sources for population-based predictive models as applied to care management.

We transformed test results that were extracted from an outpatient EHR into risk markers that could be replicated in a health system; organizations should be able to derive risk thresholds that are similar to those used in this research. Our thresholds were outside the reference ranges for apparently healthy persons, although less extreme than those used in previous inpatient mortality models.22

Among the basic metabolic panel, 5 of 8 candidate tests were associated with high-cost risk. Abnormalities in electrolytes (sodium, potassium, chloride, bicarbonate, BUN) can occur in patients with congestive heart failure and kidney disease, and both conditions are linked with higher costs.23,24 Although creatinine is used clinically to determine chronic kidney disease stages, we found an association between creatinine and higher costs for less than 1% of tested patients, which was lower than our threshold. Hyperglycemia, as demonstrated by elevated glucose, was associated with increased cost risk, but A1C showed no association. Some tests that are commonly used to stage disease or guide treatment (eg, creatinine and A1C) were not predictive of prospective cost in our analysis. A previous analysis found that these tests were associated with 5-year costs among patients with diabetes11; our results may be accounted for by the 2-year duration of this study.

Among tests from the liver function test panel, 2 of 6 tests were associated with high-cost risk. Low levels of total protein and albumin are linked with liver disease and malnutrition.25 Three components from the complete blood count were associated with high risk in the subsequent year. Low hemoglobin and hematocrit values indicate anemia, and low values of platelets can be diagnostic of thrombocytopenia. Among the 4 tests included in the lipid panel, low total cholesterol and LDL-C levels were associated with high risk. Several conditions, including cancer, may contribute to low cholesterol levels.26 We did not identify high total cholesterol and LDL-C levels, 2 clinical risk factors for coronary artery disease,27 as ranges that contribute to high-cost risk. Patients who had lipid tests may have already been treated with statins, so they had decreased risk.

Our second objective was to examine whether laboratory-based risk markers improved predictive models for total healthcare costs, top-cost claimants, and inpatient hospitalization. We explored the added value to models that varied in complexity in terms of the number and scope of morbidity markers, ranging from a demographic-only model to a Charlson model with 17 morbidity categories to 3 complex models from the ACG system. This approach enabled us to examine the impact of laboratory-based risk markers across a range of models and inform organizations that may have access to sources of data with different limitations (eg, stand-alone laboratory centers may not have the complete clinical picture of referred patients). In all cases, laboratory-based markers improved the prediction of costs and the identification of high-cost claimants and patients with inpatient admission compared with original models. Model performance improved greatly when laboratory risk markers were added to the demographic and Charlson models and modestly when laboratory-based markers were added to ACG-Dx and ACG-DxRx models with large sets of morbidity markers derived from diagnoses and medication data found in claims or EHRs. Importantly, for health systems or healthcare practices with limited resources for predictive modeling, our results demonstrate that a simple model with laboratory markers may provide a tool to evaluate individuals and patient panels.


Our research has several limitations, including that (1) the development of laboratory-based risk markers could be refined by integrating patient characteristics (ie, age and sex) and multiple tests in regression tree analyses; (2) our study population contained mainly working-age insured patients; therefore, our exploratory research should be replicated in other populations (eg, elderly patients); (3) temporal changes in test results could contain additional risk information for patients who have multiple laboratory tests in a year; (4) additional risk information could be gathered from tests that are less frequently ordered in outpatient settings, including tests that would inform about diagnoses that are potentially underreported in EHRs and claims; and (5) the model fit could conceivably be improved somewhat with alternative statistical techniques.


We explored outpatient laboratory risk markers in a large population of insured patients. Although our results with several lab-enhanced predictive models are modest, this work offers a promising perspective for independent laboratory test providers and care delivery systems that have limited morbidity data available for high-risk patient identification. More generally, organizations that apply strategies for high-risk case finding may want to consider adding laboratory-based risk markers to their models. These added clinical data may prove useful for a range of applications in the population health surveillance and care management domains.


The authors acknowledge the support of HealthPartners, Inc (Bloomington, Minnesota), in sharing the underlying data and providing the research team with technical support throughout the research.Author Affiliations: Department of Health Policy and Management, The Johns Hopkins University Bloomberg School of Public Health, Center for Population Health Information Technology (KWL, HK, JPW), Baltimore, MD; Department of Medicine, Division of General Internal Medicine (KAG), and Division of Health Sciences Informatics (HK), The Johns Hopkins University School of Medicine, Baltimore, MD; Welch Center for Prevention, Epidemiology, and Clinical Research, Johns Hopkins Medical Institutions (KAG), Baltimore, MD.

Source of Funding: This manuscript has been prepared by faculty and staff at The Johns Hopkins University. The manuscript references the Adjusted Clinical Groups (ACG) system. The Johns Hopkins University holds the copyright to the ACG system and receives royalties from the global distribution of the ACG system. The authors are members of a group of researchers who develop and maintain the ACG system with support from The Johns Hopkins University.

Author Disclosures: Dr Gudzune is a member of the ACG Technical Advisory Board. The remaining authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

Authorship Information: Concept and design (KWL, KAG, JPW); acquisition of data (KWL, JPW); analysis and interpretation of data (KWL, KAG, JPW); drafting of the manuscript (KWL, KAG, HK); critical revision of the manuscript for important intellectual content (KWL, KAG, HK, JPW); statistical analysis (KWL); obtaining funding (JPW); administrative, technical, or logistic support (JPW); and supervision (HK, JPW).

Address Correspondence to: Klaus W. Lemke, PhD, Center for Population Health Information Technology, The Johns Hopkins University Bloomberg School of Public Health, 624 North Broadway, Rm 601, Baltimore, MD 21205. Email: klemke1@jhu.edu.REFERENCES

1. Forrest CB, Lemke KW, Bodycombe DP, Weiner JP. Medication, diagnostic, and cost information as predictors of high-risk patients in need of care management. Am J Manag Care. 2009;15(1):41-48.

2. Kharrazi H, Chi W, Chang HY, et al. Comparing population-based risk-stratification model performance using demographic, diagnosis and medication data extracted from outpatient electronic health records versus administrative claims. Med Care. 2017;55(8):789-796. doi: 10.1097/MLR.0000000000000754.

3. Pine M, Jones B, Lou YB. Laboratory values improve predictions of hospital mortality. Int J Qual Health Care. 1998;10(6):491-501.

4. Tabak YP, Johannes RS, Silber JH. Using automated clinical data for risk adjustment: development and validation of six disease-specific mortality predictive models for pay-for-performance. Med Care. 2007;45(8):789-805. doi: 10.1097/MLR.0b013e31803d3b41.

5. Novack V, Pencina M, Zahger D, et al. Routine laboratory results and thirty day and one-year mortality risk following hospitalization with acute decompensated heart failure. PLoS One. 2010;5(8):e12184. doi: 10.1371/journal.pone.0012184.

6. Escobar GJ, Gardner MN, Greene JD, Draper D, Kipnis P. Risk-adjusting hospital mortality using a comprehensive electronic record in an integrated health care delivery system. Med Care. 2013;51(5):446-453. doi: 10.1097/MLR.0b013e3182881c8e.

7. De Cosmo S, Copetti M, Lamacchia O, et al. Development and validation of a predicting model of all-cause mortality in patients with type 2 diabetes. Diabetes Care. 2013;36(9):2830-2835. doi: 10.2337/dc12-1906.

8. Hammill BG, Curtis LH, Fonarow GC, et al. Incremental value of clinical data beyond claims data in predicting 30-day outcomes after heart failure hospitalization. Circ Cardiovasc Qual Outcomes. 2011;4(1):60-67. doi: 10.1161/CIRCOUTCOMES.110.954693.

9. Goldman ES, Easterling MJ, Sheiner LB. Improving the homogeneity of diagnosis-related groups (DRGs) by using clinical laboratory, demographic, and discharge data. Am J Public Health. 1989;79(4):441-444.

10. Mozes B, Easterling MJ, Sheiner LB, et al. Case-mix adjustment using objective measures of severity: the case for laboratory data. Health Serv Res. 1994;28(6):689-712.

11. McBrien KA, Manns BJ, Chui B, et al. Health care costs in people with diabetes and their association with glycemic control and kidney function. Diabetes Care. 2013;36(5):1172-1180. doi: 10.2337/dc12-0862.

12. What LOINC is. LOINC website. loinc.org/get-started/what-loinc-is. Accessed June 5, 2017.

13. RELMA. LOINC website. loinc.org/relma. Accessed June 5, 2017.

14. Nath JL. Stedman’s Medical Terminology. 2nd ed. Baltimore, MD: Wolters Kluwer; 2016.

15. Kuhn M. Building predictive models in R using the caret package. J Stat Softw. 2008;28(5):1-26. doi: 10.18637/jss.v028.i05.

16. Charlson M, Wells MT, Ullman R, King F, Shmukler C. The Charlson comorbidity index can be used prospectively to identify patients who will incur high future costs. PLoS One. 2014;9(12):e112479. doi: 10.1371/journal.pone.0112479.

17. Weiner JP, Starfield BH, Steinwachs DM, Mumford LM. Development and application of a population-oriented measure of ambulatory care case-mix. Med Care. 1991;29(5):452-472.

18. Austin PC, Walraven Cv. The mortality risk score and the ADG score: two points-based scoring systems for the Johns Hopkins Aggregated Diagnosis Groups to predict mortality in a general adult population cohort in Ontario, Canada. Med Care. 2011;49(10):940-947. doi: 10.1097/MLR.0b013e318229360e.

19. The Johns Hopkins ACG System website. hopkinsacg.org. Accessed June 5, 2017.

20. Lemke KW, Weiner JP, Clark JM. Development and validation of a model for predicting inpatient hospitalization. Med Care. 2012;50(2):131-139. doi: 10.1097/MLR.0b013e3182353ceb.

21. Pencina MJ, Fine JP, D’Agostino RB Sr. Discrimination slope and integrated discrimination improvement — properties, relationships and impact of calibration. Stat Med. 2017;36(28):4482-4490. doi: 10.1002/sim.7139.

22. Tabak YP, Sun X, Nunez CM, Johannes RS. Using electronic health record data to develop mortality predictive model: Acute Laboratory Risk of Mortality Score (ALaRMS). J Am Med Inform Assoc. 2014;21(3):455-463. doi: 10.1136/amiajnl-2013-001790.

23. Voigt J, John MS, Taylor A, Krucoff M, Reynolds MR, Gibson CM. A reevaluation of the costs of heart failure and its implications for allocation of health resources in the United States. Clin Cardiol. 2014;37(5):312-321.

24. Smith DH, Gullion CM, Nichols G, Keith DS, Brown JB. Cost of medical care for chronic kidney disease and comorbidity among enrollees in a large HMO population. J Am Soc Nephrol. 2004;15(5):1300-1306. doi: 10.1097/01.ASN.0000125670.64996.BB.

25. Wong F. Drug insight: the role of albumin in the management of chronic liver disease. Nat Clin Pract Gastroenterol Hepatol. 2007;4(1):43-51. doi: 10.1038/ncpgasthep0680.

26. Kritchevsky SB, Kritchevsky D. Serum cholesterol and cancer risk: an epidemiologic perspective. Annu Rev Nutr. 1992;12:391-416. doi: 10.1146/annurev.nu.12.070192.002135.

27. Wilson PW, D’Agostino RB, Levy D, Belanger AM, Silbershatz H, Kannel WB. Prediction of coronary heart disease using risk factor categories. Circulation. 1998;97(18):1837-1847. doi: 10.1161/01.CIR.97.18.1837.

Source link

Most Popular

To Top