Interobserver Variability in Applying a Radiographic Definition for ARDS: Materials and Methods
The study design was a survey of volunteers recruited from participants at the Toronto Mechanical Ventilation Workshop held November, 1997 and from the National Institutes of Health (NIH) ARDS Network. Chest radiographs were obtained from three sources. Two institutions that prospectively identify hypoxemic critically ill patients for the presence of ALI-ARDS (University of Washington, Harborview Medical Center; Seattle, WA, and University of California Moffitt-Long Hospital; San Francisco, CA) contributed a random selection of chest radiographs from intubated patients with a Pao2/Flo2 < 300. In addition, randomly selected chest radiographs, from patients enrolled in a recently completed study of mechanical ventilation in patients at risk for developing ARDS, were also contributed. Radiographs could have come from any time during the patient’s course of intubation as long as they met the hypoxemia threshold on the day they underwent the radiograph. In selecting the radiographs, the investigators did not know which patients had received diagnoses of ALI-ARDS, cardiogenic pulmonary edema, pneumonia, or any other condition at the contributing center; therefore, this information could not affect radiograph selection. All were routine, posteroanterior, portable radiographs, taken for clinical use, and no attempt was made to standardize the technique. The radiographs from Seattle and San Francisco were 10 X 14-inch digitized computed radiographs, and the Toronto radiographs were 14 X 17-inch standard analog radiographs. Patient age, institution, and other identifying information was obscured by tape.
Eighteen participants read the radiographs at the Toronto meeting, and 3 others received the series of radiographs by mail. All readers who volunteered to participate in the study were included. Identical instructions, provided to each reader, stated:
“All radiographs were taken from intubated patients with
Pao2/Flo2 < 300. Does this chest radiograph fulfill the
AECC definition for ALI and ALI-ARDS, ‘bilateral infiltrates consistent with pulmonary edema’? Note that the
American-European Consensus Conference definition
specifically included mild and patchy infiltrates.”
No clinical history or additional information was provided. No time constraint was placed on the readers. Responses reported as “positive” indicated that the chest radiographs fulfilled the definition of the AECC. The readers were asked to indicate aspects of the radiographs that made the definition difficult to apply to a specific radiograph.
Data were analyzed to determine the percentage of readers that interpreted each radiograph as positive, the percentage of radiographs read as positive or negative by each reader, and to measure interobserver variability (K-statistic). The approximate normal test was used to test for statistical significance between K-statistic values. All analyses were performed on a computer (IBM-PC; Danbury, CT) using appropriate software (SAS; SAS Institute; Cary, NC).