Multimodal Emotion Recognition and Contextual Analysis in Therapy Sessions Using Video Large Language Models

Tracking patients’ emotions during therapy is crucial for accurate diagnosis and treatment planning, yet current automated emotion recognition methods are limited in that they either rely on a single modality or output only coarse emotion labels which are insufficient for therapeutic contexts. Recently, some multimodal techniques have employed video large language models (VLLMs) to produce emotional descriptions rather than mere labels; however, the potential of such approaches for therapeutic use remains underexplored. To address these gaps, we propose a VLLM-based, multimodal emotion detection system tailored for therapy dialogue. Our method isolates patient-only segments from session recordings, extracting both textual transcripts and audio features for each clip, which are processed using pretrained models to generate preliminary emotion labels. These, along with the segment transcript and preceding conversational context, are embedded in a structured prompt, which is passed to a VLLM with the corresponding video segment. The VLLM then produces an overall emotion classification, an explanation highlighting salient cues, a list of key observations, and a confidence estimate. Finally, the outputs across all patient segments are consolidated into a timeline reflecting the patient’s emotional progression throughout the session. 

Our key contribution is a narrative-based, interpretable emotion tracking system that offers therapists deeper insight into patient affect. By leveraging prompting for multimodal fusion, our approach avoids the need for domain-specific training and large datasets, which are particularly difficult to obtain in therapy contexts due to confidentiality. Preliminary results are promising and demonstrate the feasibility of applying VLLMs for enriched emotion analysis in therapeutic settings.