The following analyses were conducted using IBM SPSS version 25 (IBM Corp 2017) and Microsoft Excel (Microsoft Corporation 2018) after all data were collected. First, the frequency distribution for each item was generated to show how SMEs rated each item’s (a) domain relevance, (b) developmental age appropriateness, (c) method of measurement/prompt appropriateness, and (d) domain representation. Popham (1992) recommended that 70% of SMEs endorsing an item’s relevance to support content validity as sufficient, which is most closely approximated by nine out of 13 SMEs (69.2%). All percentages were compared against the threshold of 69.2%, which is informed by Popham (1992). The authors classified the strength of evidence based on percentages; that is, percentages 69% or greater were classified as strong, percentages between 50 and 60% as moderate, and percentages less than 50% as limited. Second, the content validity ratio (CVR, Lawshe 1975) was computed for each item as follows:

where ne is the number of SMEs rating the item as “Essential,” and N is the total number of SMEs who provided a rating. The CVR can range from − 1 to + 1, with higher scores indicating greater content validity evidence. A CVR of 0 indicates that 50% of the SMEs rated the item as “Essential.” Wilson et al. (2012) recalculated the critical values for Lawshe’s (1975) CVR based on differing levels of Type I error control and number of SMEs. The critical value for the CVR with 13 SMEs using a one-tailed test with Type I error rate of 5% is 0.456, which was used as a comparison for all CVRs in this study. The critical value of 0.456 corresponds to 10 or more SMEs out of 13 rating an item as “Essential” in order to be considered statistically significant (i.e., percentage of SME support exceeds 50%). The content validity index (CVI) was calculated as the average CVR across all items and can be interpreted as content validity evidence of the domain as a whole (Lynn 1986; Shultz et al. 2014). Additionally, the CVI was calculated for each level across domains and the test as a whole. Following Lawshe (1975), CVRs and CVIs apply only to domain relevance.

It should be noted that the critical value of CVR requires more SMEs to rate an item as “Essential” than Popham’s recommended criterion (i.e., 10 versus 9). Based on the size of the sample, each SME has considerable weight in the distribution of ratings. Ten out of 13 SMEs (76.9%) has a CVR of 0.54, and nine out of 13 SMEs (69.2%) has a CVR of 0.38. Thus, an SME endorsement rating of 69.2% is considered meaningful even though the hypothesis test of the CVR is more conservative.

The relationships between the method of measurement appropriateness ratings across different methods of measurement were estimated using Cramér’s V, which is a χ2-based measure of association between categorical or nominal variables (Cramér 1946; Liebetrau 1983; Rea and Parker 1992).

As noted previously, SMEs could provide recommendations for item revisions, additions, or deletions on the items within each domain. The responses provided were analyzed qualitatively using thematic analysis of the text. That is, the responses were reviewed and categorized based on thematic elements that emerged with respect to recommended changes to the instrument.

Note: The content above has been extracted from a research article, so it may not display correctly.



Q&A
Please log in to submit your questions online.
Your question will be posted on the Bio-101 website. We will send your questions to the authors of this protocol and Bio-protocol community members who are experienced with this method. you will be informed using the email address associated with your Bio-protocol account.



We use cookies on this site to enhance your user experience. By using our website, you are agreeing to allow the storage of cookies on your computer.