A general response process theory for situational judgment tests

Abstract

Situational judgment tests (SJTs) have emerged as a staple of assessment methodologies for organizational practitioners and researchers. Despite their prevalence, many questions regarding how to interpret respondent choices or how variations in item construction and instruction influence the nature of observed responses remain. Existing conceptual and empirical efforts to explore these questions have largely been rooted in reflexive psychometric measurement models that describe participant responses as indicative of (usually multiple) latent constructs. However, some have suggested that a key to better understanding SJT responses lies in unpacking the judgment and decision-making processes employed by respondents and the psychological and contextual factors that shape how those processes play out. To this end, the present paper advances an integrative and generalizable process-oriented theory of SJT responding. The framework, labeled Situated Reasoning and Judgment (SiRJ), proposes that respondents engage in a series of conditional reasoning, similarity, and preference accumulation judgments when deciding how to evaluate and respond to an SJT item. To evaluate the theory’s plausibility and utility, the SiRJ framework is translated into a formal computational model and results from a simulation study are analyzed using neural network and Bayesian survival analytic techniques that demonstrate its capability to replicate existing and new empirical effects, suggest insights into SJT interpretation and development, and stimulate new directions for future research. An interactive web application that allows users to explore the computational model developed for SiRJ https://grandjam.shinyapps.io/sirj as well as all reported data and the full model/simulation code https://osf.io/uwdfm/ are also provided.

James A. Grand

Associate Professor, Psychology

A scientist at heart, an organizational psychologist by training, and a lucky dad and husband all the time.