Data analyses

IBM SPSS Statistics Version 23.0 (IBM Corp., Armonk, New York, USA) and GraphPad Prism Version 7.02 (GraphPad Software, La Jolla, California, USA) were used for all data analyses and visualizations. Descriptive statistics are provided as means and standard errors of the mean, unless otherwise mentioned. Only answers that consisted of a correct initial answer and a correct explanation were marked correct. Dependent samples t-tests were performed, for Refutation text and Standard text groups separately, to determine whether there was a difference in pre-test versus post-test scores.

An analysis of covariance (ANCOVA) was used to determine whether the post-test means, adjusted for pre-test scores, differed between groups. To determine the effects of response accuracy (incorrect or correct answer, i.e. cognitive effect), stage (pre- or post-intervention) and group (standard or refutation text), and their interactions with confidence (i.e. metacognitive effect), we used a multiple linear regression (MLR) model with dummy variables. We used effects coding to avoid multicollinearity. Consequently the coding for the dummy variables for response (R), stage (S) and group (G) was as follows: incorrect answer R = -1, correct answer R = + 1, pre-test S = -1, post-test S = + 1, standard text G = -1, refutation text G = + 1. The MLR model was: Y = B₀ + B_R.R + B_S.S + B_G.G + B_RS.R.S + B_RG.R.G + B_RSG.R.S.G.

This model was applied to the individually corrected confidence scores (Y): a student’s average confidence score was subtracted from their confidence scores on each question to remove the between-students variability in average confidence scores.

To test the hypercorrection hypothesis, we determined the fraction of initial misconceptions that were changed to a correct answer after intervention and the fraction of initial lack of knowledge that was changed to a correct answer. A hypercorrection effect is found if the fraction corrected misconceptions is higher than the fraction corrected lack of knowledge. Therefore, outcomes were made dichotomous: a confidence score below or equal to 3 was defined as low and a confidence score above 3 as high. This cut-off was chosen because students selecting “3” were still essentially unsure (‘in doubt’) about being correct. We used Hasan’s decision matrix to label the answers (see Fig. 1). According to this matrix incorrect answers given with high confidence are considered misconceptions, incorrect answers given with low confidence are considered a lack of knowledge [41]. Correct answers held with low confidence were labelled lucky guesses and correct answers with high confidence were labelled correct knowledge. In these terms, misconceptions and lucky guesses are considered low metacognition and correct knowledge and lack of knowledge high metacognition. Furthermore, a cognitive effect was labelled positive when one changed an incorrect answer to a correct answer. A metacognitive effect was labelled positive when one changed from low metacognition to high metacognition.

Hasan’s decision matrix (adjusted)