A more comprehensive, more reliable multilevel approach for assessing and modeling teacher judgment accuracy using latent variables

We propose a novel approach for modeling judgment accuracy that, for the first time, allows for simultaneously considering the rank, level, and differentiation component, the predominantly applied operationalization of teacher judgment accuracy. These components are conceptualized as latent, unobserved individual abilities. The model is introduced technically and its functionality is illustrated. Next, several model extensions are described that enhance the model’s capabilities to address important research questions in teacher judgment accuracy research, such as concerning moderators of judgment accuracy or the effect of judgment accuracy on learning outcomes. We study the newly proposed model in three simulation studies demonstrating that our approach provides more accurate individual-level estimates of judgment accuracy components than the traditional, still widely applied person- and component-wise calculation (Simulation Study 1), that our approach yields more accurate standard errors of moderation effects of judgment accuracy components resulting in higher true positive rates across various typical sample size scenarios (Simulation Study 2), and that our approach yields lower parameter bias and higher coverage rates of predictive effects (Simulation Study 3). The findings underscore the model’s potential for improving the assessment and modeling of judgment accuracy although the improvement over the person-score-based two-step approach were rather small in some conditions. We present two real data examples, including a step-by-step tutorial how to apply the newly proposed approach, and conclude by discussing the implications of our results and suggesting directions for future research. Easy-to-use R code is provided to simplify the application of the new judgment accuracy model.