Crosscutting lessons learned

Although there were differences across the case studies that utilized Iterative RE-AIM, there were also crosscutting findings that emerged across projects. Table 4 and the text below summarize these findings.

Key crosscutting issues in applying Iterative RE-AIM.

The case studies employed different numbers and types of clinical and community partners, but most often centered on the team directly implementing the program. It is important to have team members share perceptions and agree on priorities, but it is unclear how many perspectives are needed and if these need to be the same persons across all Iterative RE-AIM assessments. Although most recommendations regarding team science (36) stress including a full array recipients (e.g., patients, employees, opinion leaders, organizational decision makers, community representatives) as part of the decision making team, the example cases did not involve all these categories of partners. Congruent with recent emphases and recommendations for complex interventions (20) and adaptations (7) we are finding the level of engagement of multiple implementation partners to be critical for success. However, including a larger number and different types of participants needs to be balanced against the logistics and costs of those members being able to meet regularly to continue the Iterative RE-AIM process over time. It will be informative to see if Iterative RE-AIM applications that involve more partners with more diverse perspectives produce better long-term results than those that do not.

Except for the lung ultrasound study, the current Iterative RE-AIM applications did not have real time, objective data on RE-AIM outcomes to evaluate progress. Sometimes project records provided information on Reach or Adoption rates, but many of the ratings of progress were made based on the subjective impressions of the team members. Design and proactive use of process data systems that can be queried to produce frequent updates on issues such as fidelity, adaptations, and representativeness (equity) of RE-AIM results would improve the quality of data available for decision making. Once data on progress on RE-AIM dimensions are available, they need to be summarized and communicated in a way that is readily understood and actionable. Current Iterative RE-AIM projects have used some form of a bar chart as shown in Figure 1, and most participants seem to understand and find these displays useful, with exception that information about variability across raters was unclear for some participants. Newer applications of Iterative RE-AIM are experimenting with different types of visual displays, including giving participants their choice of different data displays.

Example of Iterative RE-AIM gap analysis to study discrepancies between importance and progress on RE-AIM outcome dimensions.

These discussions can be rich and enlightening for participants but can also require experienced facilitation if there are large differences in perception, power, or information across team members.

The primary method to date has been estimating progress at the next meeting across RE-AIM dimensions, but this is non-specific and suffers from the same concerns about data quality noted above. Even with high quality data, without experimentally testing strategies, it is difficult to attribute improvement to use of a strategy separate from numerous other dynamic program and contextual factors (19, 37). This is a conceptual and methodological challenge for all approaches to adaptions, not just Iterative RE-AIM. Since it is impossible in many situations such as our Quadruple Aim QUERI project to separate and independently evaluate the impact of separate implementation strategies, this may never be knowable. It is likely best addressed through mixed methods approaches using proximal quantitative data (such as rapid EHR data) followed by qualitative probes to provide confirmation and contextual understanding.

The case studies vary from a single midpoint use of Iterative RE-AIM to numerous biweekly applications; the number of team members from one or two up to 14; and the work required to prepare data summaries from being very little when automated EHR reports are available to fairly time consuming if ratings from several persons need to be analyzed, integrated and feedback displays produced by hand.

We find useful the concepts of form and function (22) of adherence to core functions of Iterative RE-AIM as outlined in Table 1, while encouraging tailoring of the specific forms- e.g., data sources, data display choices, which staff to involve, number of iterations. We also experienced some challenges in making decisions about what constitutes an adaptation vs. just a small change that is not intended to improve fit to context.