Measurement and data collection

The two-part custom survey consisted of 78 questions adapted from a framework of 54 indicators to measure implementation success for Project ECHO^® [39]. These indictors sought to measure implementation success metrics across four key domains, outlined with definitions below:

Due to the implementation success indicators for Project ECHO^® being designed to measure the innovation’s organizational and individual ECHO^® Network specific outcomes, the survey was administered in two parts, with Part A and B incorporating indicators across each of the above domains as relevant, as illustrated in Fig. 3.

Survey design and data collection process

Part A collected organizational and demographic data for each Project ECHO^® hub that participated. This data was only required as a once-off, single point-in-time data submission per Project ECHO^® hub organization. There were 29 questions in Part A, which elicited organizational-specific demographics and indicators for measurement. The 49-question Part B component facilitated the collection of data from multiple individual ECHO^® Networks for each specific virtual community of practice that was operated by the respective Project ECHO^® hub organization. Responses to both Part A and Part B surveys consisted of free-text, multiple choice and 10-point Likert scale response fields. Having this two-part capability within the survey ensured a convenient and straightforward way for study participants to submit data where hub organizations operated more than one ECHO^® Network.

Data for this study was collected between June and September 2022. Data was collected in each participating Project ECHO^® hub organization by a nominated lead respondent using the fillable survey completion guide, then submitted via the online survey. The data collection phase lasted for three months to maximize the potential research participants to opt-in to the study, and for them to gather as much implementation data as possible. This acknowledged that the indicators used in this study extended beyond basic metrics captured by the current iECHO client relationship management (CRM) system used by all Project ECHO^® hub organizations around the world. Each part of the surveys took organizational leads approximately 20–30 min to complete by inputting the data collated in the survey completion guide.