Item Bank development

Six focus groups (~ 8 students/group) were conducted with 9th–12th grade high school students (Mean age = 16.49, Standard deviation [SD] = 1.35, 13–19-years, 86.5% girls, 35% non-Hispanic Black, 35% Hispanic/Latinx, 92% free/reduced lunch eligible) to better understand adolescents’ definition, operationalization, and use of HL (full results reported elsewhere [14, 15]). Focus groups were conducted after school and moderated by trained research assistants. During focus groups, participants also provided qualitative responses to scenarios related to IHL and CHL. Scenarios were developmentally appropriate in that it reflected experiences adolescents would have with health care, disease prevention, and disease promotion. The first author’s experience as a pediatric psychologist in hospital and community settings helped inform the scenarios. Scenarios were around obesity and obesity prevention, dental health, vaccinations, and understanding written health information. For example, participants were given a scenario about a low income family being given instructions by a doctor regarding changing behaviors to address a child’s obesity and was asked what would make it difficult for the family to follow the doctor’s recommendations and what can the community do to help families in similar situations. These focus groups were the first step to ensuring that the resulting assessments had content validity (measure adequately represents all aspects of the construct) and was developmentally appropriate [30]. The responses to the scenarios included in the focus groups were content-analyzed and used to inform response options to similar scenarios in the item bank and to develop other scenarios that would be familiar to adolescents. For example, based on the obesity scenario outlined above, adolescents’ responses were used to generate response options for a similar obesity scenario and attention was paid to the information in the scenarios that the participants were most responsive to so that short scenarios with only relevant information were created for the measure. Similarly, responses to questions about how adolescents use health literacy were used to develop relatable scenarios (e.g., interacting with family members about health choices, making decisions about healthy eating). Initial items were written by the first author and revised after feedback from the second and third authors and doctoral-level research assistants. Revisions included rewording items and adding and deleting information from scenarios. Items were cross-checked with focus groups data for content and consistency with adolescents’ responses and response styles. To establish face validity and further establish content validity, four graduate-level research assistants who completed substantial reading on HL and were involved in multiple HL projects but uninvolved in the item bank development engaged in a sorting activity to put items in three categories based on Nutbeam’s [9] definitions of FHL, IHL, and CHL. Two physicians who worked with adolescents were also provided with Nutbeam’s [9] definitions and the items and asked to indicate which type of HL the items belong to (if any) and provide feedback on current items and suggestions for additional items. Items were removed or revised if they were not unanimously sorted into the three categories. Items were also revised, removed, or added based on suggestions and feedback from the physicians. Next, adolescents 12–17-years-old were recruited from after-school settings to participate in cognitive interviews. Adolescents (n = 17, Mean age = 15.88 years, SD = 1.69, 47% girls, 41% non-Hispanic Black, 53% Hispanic/Latinx, 94% free/reduced lunch eligible) participated in the cognitive interviews while they completed the item bank questions. This process further established content validity as it provided data on how adolescents interpreted the questions and their thought processes as they responded. Cognitive interviews results were used to improve (e.g., rewording questions, calibrating the difficulties of the items) or remove problematic items. Figure 1 shows the iterative changes from initial item bank development to the revised item bank used for quantitative data collection. For both focus groups and cognitive interviews, data collection was discontinued once saturation was reached.

Illustration of iterative item bank revisions prior to large scale quantitative data collection