Evaluation of a Measurement System to Assess ICU Team Performance*

Evaluation of a Measurement System to Assess ICU Team Performance*

by Dietz, Aaron S.; Salas, Eduardo; Pronovost, Peter J.; Jentsch, Florian; Wyskiel, Rhonda; Mendez-Tellez, Pedro Alejandro; Dwyer, Cynthia; Rosen, Michael A.  

Critical Care Medicine: December 2018 - Volume 46 - Issue 12 - p 1898–1905

Objective: Measuring teamwork is essential in critical care, but limited observational measurement systems exist for this environment. The objective of this study was to evaluate the reliability and validity of a behavioral marker system for measuring teamwork in ICUs.

 Design: Instances of teamwork were observed by two raters for three tasks: multidisciplinary rounds, nurse-to-nurse handoffs, and retrospective videos of medical students and instructors performing simulated codes. Intraclass correlation coefficients were calculated to assess interrater reliability. Generalizability theory was applied to estimate systematic sources of variance for the three observed team tasks that were associated with instances of teamwork, rater effects, competency effects, and task effects.

Setting: A 15-bed surgical ICU at a large academic hospital.

Subjects: One hundred thirty-eight instances of teamwork were observed. Specifically, we observed 88 multidisciplinary rounds, 25 nurse-to-nurse handoffs, and 25 simulated code exercises. Interventions: No intervention was conducted for this study.

Measurements and Main Results: Rater reliability for each overall task ranged from good to excellent correlation (intraclass correlation coefficient, 0.64–0.81), although there were seven cases where reliability was fair and one case where it was poor for specific competencies. Findings from generalizability studies provided evidence that the marker system dependably distinguished among teamwork competencies, providing evidence of construct validity.

Conclusions: Teamwork in critical care is complex, thereby complicating the judgment of behaviors. The marker system exhibited great potential for differentiating competencies, but findings also revealed that more context specific guidance may be needed to improve rater reliability.

A Direct Observation Checklist to Measure Respect and Dignity in the ICU

A Direct Observation Checklist to Measure Respect and Dignity in the ICU
Carrese, J A et al
Critical Care Medicine: February 2017 - Volume 45 - Issue 2 - p 263–270

Objective: Treating patients and family members with respect and dignity is a core objective of health care, yet it is unclear how best to measure this in the ICU setting. Accordingly, we sought to create a direct observation checklist to assess the “respect and dignity status” of an ICU. Design: A draft checklist based on previous work was iteratively revised to enhance accuracy and feasibility. Setting: Seven ICUs within the Johns Hopkins Health System. Subjects: A total of 351 patient-clinician encounters with 184 different patients. Interventions: Four study team members pilot tested the checklist between January and August 2015. 
Measurements and Main Results: Standard psychometric analyses were performed. The direct observation checklist exhibits strong content and face validity as well as high reliability and internal consistency. All items load on one factor that supports the unidimensionality of the total index. Furthermore, concurrent validity of the direct observation checklist is demonstrated by statistically significant differences in mean scores between ICUs, between types of clinicians, and between patients’ clinical status and mood. 
Conclusions: We rigorously developed, pilot tested, and analyzed a direct observation checklist designed to assess the extent to which patients and families in the ICU setting are treated with respect and dignity. Future research should validate this checklist in other settings and compare its results with other measures. Data gathered about individual items on the direct observation checklist could be used to target areas for training and education; doing so should help facilitate more respectful treatment of patients and their families.