Representational interactions during audiovisual speech entrainment: redundancy in left posterior superior temporal gyrus and synergy in left motor cortex

Research output: Contribution to journalArticle

Authors

  • Hyojin Park
  • Robin A. A. Ince
  • Philippe G. Schyns
  • Gregor Thut
  • Joachim Gross

External organisations

  • University of Glasgow
  • University of Münster

Abstract

Integration of multimodal sensory information is fundamental to many aspects of human behavior, but the neural mechanism underlying these processes remain mysterious. For example, during face-to-face communication we know that the brain integrates the dynamic auditory and visual inputs but we do not yet understand where and how such integration mechanisms support speech comprehension. Here we quantify representational interactions between dynamic audio and visual speech signals and show that different brain regions exhibit different types of representational interaction. With a novel information theoretic measure, we found that theta (3-7 Hz) oscillations in the posterior superior temporal gyrus/sulcus (pSTG/S) represent auditory and visual inputs redundantly (i.e. represent common features of the two) whereas the same oscillations in left motor and inferior temporal cortex represent the inputs synergistically (i.e. the instantaneous relationship between audio and visual inputs is also represented). Importantly, redundant coding in the left pSTG/S and synergistic coding in the left motor cortex predict behavior – i.e. speech comprehension performance. Our findings therefore demonstrate that processes classically described as integration can have different statistical properties and may reflect distinct mechanisms that occur in different brain regions to support audiovisual speech comprehension.

Details

Original languageEnglish
Article numbere2006558
Number of pages26
JournalPLoS Biology
Volume16
Issue number8
Publication statusPublished - 6 Aug 2018

Keywords

  • Speech, Entrainment, Audiovisual integration, Information theory, Partial Information Decomposition (PID), Redundancy, Synergy