Reader Evaluation of Radiology Reports

A radiology resident (R.A.S.) (R2 level) performed manual review of the 3233 reports for speech recognition errors. Additional independent evaluation was conducted by two board-certified radiologists (W.L., L.L.) with more than 20 years of experience, who annotated a random subsample of 100 reports for errors.

Specific examples of speech recognition errors are detailed in the Table. Error severity was marked as clinically significant or not clinically significant according to the ontology of Chang et al (3) and as used in recent studies (9,14). Clinically significant errors are considered to change the meaning of the report and risk misinterpretation by the clinician. Examples included nonsense phrases (“The lungs nuclear” instead of “the lungs are clear”), omission of important words (“Intracranial hemorrhage” instead of “No intracranial hemorrhage”), and internally inconsistent statements (“left occipital lesion” referred to as “left parietal lesion” later in the report). The location and severity were recorded for each error in each report.

Types of Speech Recognition Error