Joint modeling for learning decision-making dynamics in behavioral experiments

Abstract
Major depressive disorder (MDD), a leading cause of disability and mortality, is associated with reward-processing abnormalities and concentration issues. Motivated by the probabilistic reward task from the establishing moderators and biosignatures of antidepressant response in clinical care (EMBARC) study, we propose a novel framework that integrates the reinforcement learning (RL) model, hidden Markov model (HMM), and drift-diffusion model (DDM) to analyze reward-based decision-making alongside response times. To account for emerging evidence suggesting that decision-making may alternate between multiple interleaved strategies, we model latent state switching using an HMM. In the ‘engaged’ state, decisions follow an RL-DDM, simultaneously capturing reward processing, decision dynamics, and temporal structure. In contrast, in the ‘lapse’ state, decision-making is modeled using a simplified DDM, where specific parameters are fixed to approximate random guessing with equal probability. The proposed method is implemented using a computationally efficient generalized expectation-maximization (EM) algorithm with forward-backward procedures. Through extensive numerical studies, we demonstrate that our proposed method outperforms competing approaches under various reward-generating distributions, both with and without strategy switching. When applied to the EMBARC study, our framework reveals that MDD patients exhibit lower overall engagement than healthy controls and experience longer decision times when they do engage. Additionally, we examine the associations between measures of brain activities and decision-making characteristics.