Advertisement

Facts and Fallacy of Benchmark Performance Indicators

  • James P. Byrne
    Correspondence
    Corresponding author. Division of Trauma and Acute Care Surgery, Department of Surgery, Johns Hopkins Hospital, Sheikh Zayed 6107E 1800 Orleans Street, Baltimore, MD 21287, USA
    Affiliations
    Division of Acute Care Surgery, Department of Surgery, Johns Hopkins Hospital, Sheikh Zayed 6107 1800 Orleans Street, Baltimore, MD 21287, USA
    Search for articles by this author
  • Elliott R. Haut
    Affiliations
    Division of Acute Care Surgery, Department of Surgery, Johns Hopkins Hospital, Sheikh Zayed 6107 1800 Orleans Street, Baltimore, MD 21287, USA
    Search for articles by this author

      Keywords

      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Advances in Surgery
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Institute of Medicine Committee on Quality of Health Care in A
        Crossing the quality chasm: a new health system for the 21st century.
        National Academies Press (US), Washington (DC)2001
        • American College of Surgeons Committee on Trauma
        Advanced trauma life support student course manual.
        10th edition. American College of Surgeons, Chicago, IL2018
        • Champion H.R.
        • Mabee M.S.
        • Meredith J.W.
        The state of US trauma systems: public perceptions versus reality--implications for US response to terrorism and mass casualty events.
        J Am Coll Surg. 2006; 203: 951-961
        • Minei J.P.
        • Schmicker R.H.
        • Kerby J.D.
        • et al.
        Severe traumatic injury: regional variation in incidence and outcome.
        Ann Surg. 2010; 252: 149-157
        • Shafi S.
        • Nathens A.B.
        • Cryer H.G.
        • et al.
        The trauma quality improvement program of the american college of surgeons committee on trauma.
        J Am Coll Surg. 2009; 209: 521-530
        • American College of Surgeons Committee on Trauma
        Resources for Optimal care of the injured patient.
        (Available at) (Accessed: Nov 1, 2021)
        • Khuri S.F.
        • Daley J.
        • Henderson W.
        • et al.
        The department of veterans affairs' national surgical quality improvement program: the first national, validated, outcome-based, risk-adjusted, and peer-controlled program for the measurement and enhancement of the quality of surgical care.
        Ann Surg. 1998; 228: 491-507
        • Institute of Medicine (US)
        Committee to design a strategy for quality review and assurance in medicare.
        in: Lohr K.N. Medicare: a strategy for quality assurance. National Academies Press (US), Washington (DC)1990
        • Best M.
        • Neuhauser D.
        Avedis donabedian: father of quality assurance and poet.
        Qual Saf Health Care. 2004; 13: 472-473
        • Donabedian A.
        The quality of medical care.
        Science. 1978; 200: 856-864
        • Donabedian A.
        The quality of care. how can it be assessed?.
        JAMA. 1988; 260: 1743-1748
        • Merriam-Webster.com
        Benchmarking.
        (Available at) (Accessed: Nov 1, 2021)
        • Camp R.C.
        Benchmarking: the search for industry best practices that lead to superior performance.
        ASQC Quality Press, Milwaukee, WI1989
        • Cross RaI A.
        The rank xerox experience: benchmarking ten years on benchmarking — theory and practice.
        Springer Science and Business Media, Milwaukee, WI1995
        • National Institute of Standards and Technology
        Baldrige award recipients listing.
        (Available at) (Accessed: Nov 1, 2021)
        • O'Connor G.T.
        • Plume S.K.
        • Olmstead E.M.
        • et al.
        A regional intervention to improve the hospital mortality associated with coronary artery bypass graft surgery. the northern new england cardiovascular disease study group.
        JAMA. 1996; 275: 841-846
        • Hannan E.L.
        • Kilburn Jr., H.
        • Racz M.
        • et al.
        Improving the outcomes of coronary artery bypass surgery in New York State.
        JAMA. 1994; 271: 761-766
        • Khuri S.F.
        • Daley J.
        • Henderson W.
        • et al.
        The national veterans administration surgical risk study: risk adjustment for the comparative assessment of the quality of surgical care.
        J Am Coll Surg. 1995; 180: 519-531
        • Fink A.S.
        • Campbell Jr., D.A.
        • Mentzer Jr., R.M.
        • et al.
        The national surgical quality improvement program in non-veterans administration hospitals: initial demonstration of feasibility.
        Ann Surg. 2002; 236: 344-353
        • Champion H.R.
        • Copes W.S.
        • Sacco W.J.
        • et al.
        The major trauma outcome study: establishing national norms for trauma care.
        J Trauma. 1990; 30: 1356-1365
        • Shafi S.
        • Nathens A.B.
        • Parks J.
        • et al.
        Trauma quality improvement using risk-adjusted outcomes.
        J Trauma. 2008; 64: 599-604
        • Department of Health and Human Services
        Agency for healthcare research and quality.
        (Available at) (Accessed: Nov 1, 2021)
        • Agency for Healthcare Quality and Research
        Six domains of healthcare quality.
        (Available at) (Accessed: Nov 1, 2021)
        • Agency for Healthcare Quality and Research
        Types of healthcare quality measures.
        (Available at) (Accessed: Nov 1, 2021)
        • Agency for Healthcare Quality and Research
        Datasources for healtcare quality measures.
        (Available at) (Accessed: Nov 1, 2021)
        • Agency for Healthcare Quality and Research
        Healthcare cost and utilization project.
        (Available at) (Accessed: Nov 1, 2021)
        • Stulberg J.J.
        • Haut E.R.
        Practical guide to surgical data sets: healthcare cost and utilization project national inpatient sample (NIS).
        JAMA Surg. 2018; 153: 586-587
        • American College of Surgeons
        National surgical quality improvement program.
        (Available at) (Accessed: Nov 1, 2021)
        • American College of Surgeons
        ACS national trauma databank.
        2021 (Available at) (Accessed: Nov 1, 2021)
        • Agency for Healthcare Quality and Research
        Key questions when choosing health care quality measures.
        (Available at) (Accessed: Nov 1, 2021)
        • National Quality Forum
        Maintenance of NQF-endorsed performance measures.
        (Available at) (Accessed: Nov 1, 2021)
        • Institute for Healthcare Improvement
        Plan-do-study-Act (PDSA) worksheet 2021.
        (Available at:) (Accessed: November 1, 2021)
        • American Society for Quality
        What is six sigma? 2021.
        (Available at:) (Accessed November 1, 2021)
        • Pronovost P.J.
        • Berenholtz S.M.
        • Needham D.M.
        Translating evidence into practice: a model for large scale knowledge translation.
        BMJ. 2008; 337: a1714
        • Neuman H.B.
        • Kaji A.H.
        • Haut E.R.
        Practical guide to implementation science.
        JAMA Surg. 2020; 155: 434-435
        • Streiff M.B.
        • Carolan H.T.
        • Hobson D.B.
        • et al.
        Lessons from the Johns Hopkins multi-disciplinary venous thromboembolism (VTE) prevention collaborative.
        BMJ. 2012; 344: e3935
        • Greenberg C.C.
        • Byrnes M.E.
        • Engler T.A.
        • et al.
        Association of a statewide surgical coaching program with clinical outcomes and surgeon perceptions.
        Ann Surg. 2021; 273: 1034-1039
        • Hemmila M.R.
        • Cain-Nielsen A.H.
        • Jakubus J.L.
        • et al.
        Association of hospital participation in a regional trauma quality improvement collaborative with patient outcomes.
        JAMA Surg. 2018; 153: 747-756
        • Newgard C.D.
        • Fildes J.J.
        • Wu L.
        • et al.
        Methodology and analytic rationale for the American college of surgeons trauma quality improvement program.
        J Am Coll Surg. 2013; 216: 147-157
        • Khuri S.F.
        • Daley J.
        • Henderson W.
        • et al.
        Risk adjustment of the postoperative mortality rate for the comparative assessment of the quality of surgical care: results of the national veterans affairs surgical risk study.
        J Am Coll Surg. 1997; 185: 315-327
        • Kardooni S.
        • Haut E.R.
        • Chang D.C.
        • et al.
        Hazards of benchmarking complications with the national trauma data bank: numerators in search of denominators.
        J Trauma. 2008; 64 (; discussion 7-9): 273-277
        • Haukoos J.S.
        • Newgard C.D.
        Advanced statistics: missing data in clinical research--part 1: an introduction and conceptual framework.
        Acad Emerg Med. 2007; 14: 662-668
        • Newgard C.D.
        • Haukoos J.S.
        Advanced statistics: missing data in clinical research--part 2: multiple imputation.
        Acad Emerg Med. 2007; 14: 669-678
        • Sterne J.A.
        • White I.R.
        • Carlin J.B.
        • et al.
        Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls.
        BMJ (Clinical research ed). 2009; 338: b2393
        • Merlo J.
        • Chaix B.
        • Ohlsson H.
        • et al.
        A brief conceptual tutorial of multilevel analysis in social epidemiology: using measures of clustering in multilevel logistic regression to investigate contextual phenomena.
        J Epidemiol Community Health. 2006; 60: 290-297
        • New York State Department of Health
        Cardiac reporting system overview.
        (Available at) (Accessed: Nov 1, 2021)
        • Hannan E.L.
        • Kumar D.
        • Racz M.
        • et al.
        New York State's Cardiac Surgery Reporting System: four years later.
        Ann Thorac Surg. 1994; 58: 1852-1857
        • New York State Department of Health
        Adult Cardiac Surgery in New York State 2015-2017.
        (Available at) (Accessed: Nov 1, 2021)
      1. Veterans' Administration Health-Care Amendments of 1985, Pub L No 99–166, 99 Stat 941. Dec 3, 1985. Title II: Health-Care Administration, Sec. 201– 4.

        • Khuri S.F.
        • Daley J.
        • Henderson W.G.
        The comparative assessment and improvement of quality of surgical care in the Department of Veterans Affairs.
        Arch Surg. 2002; 137: 20-27
        • Maier R.V.
        Trauma: the paradigm for medical care in the 21st century.
        J Trauma. 2003; 54: 803-813
        • American College of Surgeons Committee on Trauma
        About the verification, review and consultation program.
        (Available at) (Accessed: Nov 1, 2021)
        • American College of Surgeons Committee on Trauma
        Trauma quality improvement program (TQIP).
        (Available from) (Accessed: Nov 1, 2021)
        • Hemmila M.R.
        • Nathens A.B.
        • Shafi S.
        • et al.
        The Trauma Quality Improvement Program: pilot study and initial demonstration of feasibility.
        J Trauma. 2010; 68: 253-262
        • Agency for Healthcare Quality and Research
        Never events.
        (Available at) (Accessed: Nov 1, 2021)
        • Sutedjo J.L.
        • Ng R.K.
        • Piazza G.
        • et al.
        Medicare's new regulations for deep vein thrombosis as a "never event": wise or worrisome?.
        Am J Med. 2009; 122: 975-976
        • Centers for Medicare and Medicaid Services, HHS
        Medicare program: Changes to the hospital inpatient prospective payment systems and fiscal year 2009 rates; payments for graduate medical education in certain emergency situations; changes to disclosure of physician ownership in hospitals and physician self-referral rules; updates to the long-term care prospective payment system; updates to certain IPPS excluded hospitals; and collection of information regarding financial relationships between hospitals.
        Fed Regist. 2008; 73: 48433-49084
        • Pronovost P.J.
        • Goeschel C.A.
        • Wachter R.M.
        The wisdom and justice of not paying for "preventable complications.
        JAMA. 2008; 299: 2197-2199
        • Streiff M.B.
        • Haut E.R.
        The CMS ruling on venous thromboembolism after total knee or hip arthroplasty: weighing risks and benefits.
        JAMA. 2009; 301: 1063-1065
        • Thorson C.M.
        • Ryan M.L.
        • Van Haren R.M.
        • et al.
        Venous thromboembolism after trauma: a never event?.
        Crit Care Med. 2012; 40: 2967-2973
        • Shackford S.R.
        • Cipolle M.D.
        • Badiee J.
        • et al.
        Determining the magnitude of surveillance bias in the assessment of lower extremity deep venous thrombosis: A prospective observational study of two centers.
        J Trauma Acute Care Surg. 2016; 80: 734-739
        • Haut E.R.
        • Chang D.C.
        • Pierce C.A.
        • et al.
        Predictors of posttraumatic deep vein thrombosis (DVT): hospital practice versus patient factors-an analysis of the National Trauma Data Bank (NTDB).
        J Trauma. 2009; 66: 994-999
        • Haut E.R.
        • Noll K.
        • Efron D.T.
        • et al.
        Can increased incidence of deep vein thrombosis (DVT) be used as a marker of quality of care in the absence of standardized screening? The potential effect of surveillance bias on reported DVT rates after trauma.
        J Trauma. 2007; 63: 1132-1135
        • Haut E.R.
        • Pronovost P.J.
        Surveillance bias in outcomes reporting.
        JAMA. 2011; 305: 2462-2463
        • Bilimoria K.Y.
        • Chung J.
        • Ju M.H.
        • et al.
        Evaluation of surveillance bias and the validity of the venous thromboembolism quality measure.
        JAMA. 2013; 310: 1482-1489
        • Malhotra A.K.
        • Goldberg S.R.
        • McLay L.
        • et al.
        DVT surveillance program in the ICU: analysis of cost-effectiveness.
        PLoS One. 2014; 9: e106793
        • Haut E.R.
        • Lau B.D.
        • Kraus P.S.
        • et al.
        Preventability of Hospital-Acquired Venous Thromboembolism.
        JAMA Surg. 2015; 150: 912-915
        • Croce M.A.
        • Brasel K.J.
        • Coimbra R.
        • et al.
        National Trauma Institute prospective evaluation of the ventilator bundle in trauma patients: does it really work?.
        J Trauma Acute Care Surg. 2013; 74: 354-360
        • Guo W.A.
        Pneumonia after trauma: a "never event.
        J Surg Res. 2013; 185: 539-540
        • Kuy S.
        • Gupta R.
        • Roy C.
        • et al.
        Incidence of Catheter-Associated Urinary Tract Infections With Compliance With Preventive Guidelines.
        JAMA Surg. 2020; 155: 661-662
        • Wahl W.L.
        • Zalewski C.
        • Hemmila M.R.
        Pneumonia in the surgical intensive care unit: is every one preventable?.
        Surgery. 2011; 150: 665-672
        • Acosta J.A.
        • Yang J.C.
        • Winchell R.J.
        • et al.
        Lethal injuries and time to death in a level I trauma center.
        J Am Coll Surg. 1998; 186: 528-533
        • Heaney J.B.
        • Guidry C.
        • Simms E.
        • et al.
        To TQIP or not to TQIP? that is the question.
        Am Surg. 2014; 80: 386-390
        • Byrne J.P.
        • Xiong W.
        • Gomez D.
        • et al.
        Redefining "dead on arrival": Identifying the unsalvageable patient for the purpose of performance improvement.
        J Trauma Acute Care Surg. 2015; 79: 850-857
        • Burack J.H.
        • Impellizzeri P.
        • Homel P.
        • et al.
        Public reporting of surgical mortality: a survey of New York State cardiothoracic surgeons.
        Ann Thorac Surg. 1999; 68: 1195-1200
        • Apolito R.A.
        • Greenberg M.A.
        • Menegus M.A.
        • et al.
        Impact of the New York state cardiac surgery and percutaneous coronary intervention reporting system on the management of patients with acute myocardial infarction complicated by cardiogenic shock.
        Am Heart J. 2008; 155: 267-273
        • Span P.
        A surgery standard under fire: New York times.
        (Available at) (Accessed: Nov 1, 2021)
        • Haut E.R.
        • Kalish B.T.
        • Cotton B.A.
        • et al.
        Prehospital intravenous fluid administration is associated with higher mortality in trauma patients: a National Trauma Data Bank analysis.
        Ann Surg. 2011; 253: 371-377
        • Byrne J.P.
        • Witiw C.D.
        • Schuster J.M.
        • et al.
        Association of venous thromboembolism prophylaxis after neurosurgical intervention for traumatic brain injury with thromboembolic complications, repeated neurosurgery, and mortality.
        JAMA Surg. 2021; 157: e215794
        • Alali A.S.
        • Fowler R.A.
        • Mainprize T.G.
        • et al.
        Intracranial pressure monitoring in severe traumatic brain injury: results from the American College of Surgeons Trauma Quality Improvement Program.
        J Neurotrauma. 2013; 30: 1737-1746
        • Byrne J.P.
        • Geerts W.
        • Mason S.A.
        • et al.
        Effectiveness of low-molecular-weight heparin versus unfractionated heparin to prevent pulmonary embolism following major trauma: a propensity-matched analysis.
        J Trauma Acute Care Surg. 2017; 82: 252-262
        • Joseph B.
        • Zeeshan M.
        • Sakran J.V.
        • et al.
        Nationwide analysis of resuscitative endovascular balloon occlusion of the aorta in civilian trauma.
        JAMA Surg. 2019; 154: 500-508
        • Bilimoria K.Y.
        • Liu Y.
        • Paruch J.L.
        • et al.
        Development and evaluation of the universal ACS NSQIP surgical risk calculator: a decision aid and informed consent tool for patients and surgeons.
        J Am Coll Surg. 2013; 217: 833-842
        • American College of Surgeons National Surgical Quality Improvement Program
        Surgical risk calculator.
        (Available at) (Accessed: Nov 1, 2021)
        • Augenstein V.
        • Colavita P.
        • Wormer B.
        • et al.
        CeDAR: Carolinas equation for determining associated risks.
        J Am Coll Surg. 2015; 221: S65-S66
        • Center P.S.H.
        CeDAR app is a free resource for hernia patients 2021.
        (Available at:) (Accessed November 1, 2021)
        • Bertsimas D.
        • Dunn J.
        • Velmahos G.C.
        • et al.
        Surgical risk is not linear: derivation and validation of a novel, user-friendly, and machine-learning-based predictive optimal trees in emergency surgery risk (POTTER) calculator.
        Ann Surg. 2018; 268: 574-583