Reliability of the VITALS Instrument

Reliability was measured using statistical models based on generalizability theory (G-theory). We have previously described in detail the assessment of reliability using G-theory analysis.^11,12 The object of measurement in the current study was the tutor’s evaluation scores by students. The design of the G-theory analysis included two facets (raters and items) with raters nested within tutors, and both crossed with items. We selected a design where items were fixed at 10, while the raters were considered random, as we were interested in generalizing the findings beyond the context used in this study. We used the generalizability coefficient (Φ), as we were interested in the relative inferences for inter-individual comparison of tutors’ performance. A G-study was conducted to estimate sources of error in the tutors’ ratings and to determine reliability of students with a different set of raters. A D-study was used to determine the reproducibility of these ratings, and to identify the minimum number of raters and items required to achieve reliability of ≥0.80. We have also reported the standard error of measurement (SEM) which represents the standard deviation of all the errors of measurement in the study.

The statistical software package GENOVA was used for the G-theory analysis. Data from the questionnaires from tutors and students analyzed using SPSS- version 16. Descriptive statistics were calculated based on the percentages of those who agreed and those who disagreed with each item in the form and in the questionnaire. Strongly agree and agree anchors were considered as “agree”, and strongly disagree and disagree were considered as “disagree”