Analysis

The quality of the MCQ was assessed by internal consistency, items’ responsiveness, face, content and construct validity. The face and content validity of MCQ were developed with the support of the group in the Medical Education Department of the University of Porto, which was responsible for the evaluation of high-stakes examinations of the Faculty of Medicine to guarantee the quality of the items. Three experts in communication (one of them is an external member of the University) assessed and approved the assessment regarding its content. The internal consistency of the items was evaluated by Cronbach’s alpha. The responsiveness and construct validity were evaluated according to a published study, in which this MCQ test was applied.³⁴ The items’ responsiveness was considered adequate once the score before and after a course on communication improved significantly. The mean of improvement was 18.9% (95% CI, ranges from 15.8% to 22.1%) (p<0.001). The MCQ (pre and post-test) was applied to medical students who attained the same communication course at four universities. The improvement in the scores after the course did not show differences among universities (p=0.102). Thus, the results indicate an acceptable construct validity.

The psychometric quality of the OSCE was evaluated by validation of the content (applying the principal component analysis for dimensionality) and internal consistency. Dimensionality was assessed using a scree plot, and the number of components was assessed according to the ‘elbow rule’. An element or item was considered to contribute to a principal component when it had a correlation value higher than 0.30. Internal consistency was evaluated using Cronbach’s alpha (Cronbach 1951). Acceptable values for internal consistency were considered to be higher than 0.7. The linear associations between the assessment methods were assessed using the Pearson’s correlation considering missing complete at random to handle with missed correlations. It was also provided a 95% CI for the Pearson’s correlation to present the precision of the correlation.

To measure agreement between researchers, we used the intraclass single average value for absolute agreement. The inter-rater agreement rate was calculated for encoded fragments (TS) and for the RS. NVivo software (V.11.3.2 for Mac) was used for qualitative data analysis, while the SPSS, V.25.0, was used for quantitative data analysis.

Participant consent was requested in the form of an informed consent before the participation in the communication skills course. Signed written consent forms were completed by all participants.