Multidimensional Rasch analysis

Based on results from IFA, the analysis was continued with multidimensional random coefficients of multinomial logit models (Adams et al. 1997), also known as the multidimensional Rasch model (Shih et al. 2013; Wang et al. 2006). This model is a generalisation of the simple Rasch model (Rasch 1960) and polytomous Rasch models (e.g. Andrich 1978).

If a test consists of several unidimensional tests (e.g. the five subscales of the OCLEI), it can be calibrated using standard Rasch analysis procedures. The test can be either analysed as a whole or, using the unidimensional Rasch model, it could be applied to each subscale separately, one test at a time (e.g. Pichardo et al. 2018). However, the unidimensional Rasch approach ignores the claims for the subscale structure of the test (Wang et al. 2004, 2006) because there are no estimated correlations between traits. In using that approach, one can only compute the correlation of person trait level between aspects.

To consider the correlations between latent traits, one needs a multidimensional model that simultaneously calibrates all the tests and utilises the correlations to increase measurement precision. In reality, because there are always non-zero correlations between latent traits, at least in theory, the multidimensional approach is more appropriate than the unidimensional one. In addition, the higher the correlations, the greater the measurement precision using the multidimensional approach (Wang et al. 2004, 2006).

In this study, the multidimensional version of the rating scale model (Andrich 1978) was chosen using the ACER Conquest 5.13 program with a marginal maximum likelihood estimation method for item parameters and a Monte Carlo-based approach with 2000 nodes for person parameter. The analysis was intended to obtain information about item fit, with Infit and Outfit MNSQ, ranging from 0.6 to 1.4, indicating that the item was fit for the model with Likert scales (Wright and Linacre 1994). Furthermore, the functioning of the 4-point Likert scale was also investigated with an Outfit MNSQ < 2.0, indicating that the response category was functioning well (Linacre 1999). Rasch analysis also yielded information about person separation reliability for each aspect of OCLEI in the form of plausible values (PV) reliability indices. This drew an estimate of how reliably items could be used to distinguish students’ underlying abilities (Fulmer et al. 2015), with values above 0.70 regarded as acceptable (Fauth et al. 2019).