Algorithm Validation

Several authors have adopted various approaches to validate results from phenotyping algorithms. For example, Spratt et al26 adopted a stratified sampling approach based on Begg and Greenes method, whereas Newton et al27 adopted an iterative approach where information obtained at each step is used to fine-tune and improve the final phenotype algorithm method. In our study, we followed the iterative approach suggested by Newton et al and evaluated the performance of the proposed algorithm for patients scheduled for surgery during August 2014. Data related to these patients were extracted from the existing legacy Mayo Clinic Unified Data Platform that stores structured, unstructured, and other patient care–related data elements from various sources that support research and quality improvements.28

The ascertainment of true DM incidence as the reference standard was created by comparing results from the proposed method with those of the existing process, which is a manual review of the EHR for a DM diagnosis by bedside nurses. When the bedside nurse identifies DM, its documentation is noted in the preoperative evaluation patient flow sheet. This nurse preoperative evaluation patient flow sheet document serves as a surgery intake tool, nursing communication tool, and assessment tool. The nurse documents whether the patient has DM or a history of DM in the flow sheet. This is based on patient response, surgical listing information, the patient’s current medication regimen, and a clinical notes review. The nurse synthesizes this information and documents in the EHR if the nurse confirms that the patient has DM. To develop this reference standard, we first considered concordant cases with agreement between the results from the proposed method and the current manual process (eg, both suggest the presence of DM or neither suggests DM). Among such concordant cases, a randomized sample of 100 patients each (ie, a total of 200 concordant cases) was screened by an expert reviewer, who has 35 years of clinical nursing experience, including patient classification, and 16 years of experience in chart abstraction. This review was to confirm whether the patients indeed did or did not have DM (N=200). Second, we considered all discordant cases (ie, a total of 231 discordant cases) where the proposed method disagreed with the findings from the current manual process. All discordant pairs were screened manually by the independent reviewer to determine whether the patients had DM. Thus, a total of 431 cases were reviewed. Final determination of the presence or absence of DM involved manual review of a patient’s EHR by a research nurse specifically trained in the extraction of medical conditions of interest from the EHR. The decision to manually review a random sample of concordant cases was determined a priori and was arbitrarily based on the collective perception of the research team.