Co-activation of sign language phonological knowledge during spoken language lexical processing: Evidence from bimodal bilinguals of Cantonese & Hong Kong Sign Language

Bilingualism offers a unique window for researchers to explore fundamental questions about the relationship between language and cognition. In a bilingual mind, a single semantic concept is represented by two lexical words across languages. A key issue in bilingualism research is whether lexical processing is specific to just one language or involves interactions between both. Traditionally, it was believed that the two languages have independent lexicons, with only the lexicon of the active language being activated at any time, as if bilingual speakers can switch the languages on and off at will (i.e., ‘language switch hypothesis’, ‘input switch theory’). In the past two decades, evidence has emerged showing that cross-language interaction occurs at various grammatical levels and processing stages in bilingual minds. Most of these studies focus on bilinguals of two spoken languages (unimodal bilinguals) and suggest that when a speaker hears a word in one language, phonologically similar words in that language and the other are activated non-selectively in a bottom-up fashion. Several eye-tracking studies on hearing bimodal bilinguals (i.e., those knowing a spoken and a sign language) demonstrate that lexical co-activation occurs in a sign-spoken language pair without phonological overlap (Giezen, Blumenfeld, Shook, Marian, & Emmorey 2015; Shook & Marian 2010, 2012; Vilameriel, Costello, Giezen, Carreiras, 2022). These studies support a bilingual lexical processing model that includes top-down and lateral mechanisms for interaction between a sign and spoken language, alongside bottom-up, input-driven activation.

This study investigates whether the lexicon of Hong Kong Sign Language (HKSL) is activated when hearing bimodal bilinguals perform an auditory lexical identification task in Cantonese in a visual world paradigm eye-tracking experiment. It examines whether lexical co-activation exists and how factors like the onset age of sign language acquisition, signing proficiency, and length of sign language exposure influence the extent of co-activation. Following Shook and Marian (2010, 2012) and Giezen et al. (2015), we sampled participants’ eye movements while they heard a Cantonese target word in instructions to click on the correct picture. Each trial began with a four-picture display and an auditory stimulus in Cantonese, ‘Please click on the XXX’ with 'XXX' being the target word. The experiment included 4 practice trials, 24 critical trials, and 48 control trials. The critical trials featured 24 minimal pairs of HKSL signs, each consisting of a target and a competitor that shared three out of four phonological parameters (e.g., BOOK and CLAM differ only in handshape).

The participants included 20 hearing bimodal bilinguals (4 CODAs) who were native Cantonese speakers and frequent users of HKSL, along with 35 unimodal, non-signing Cantonese native speakers as the control group. We measured proportional fixation times (PFTs) as the dependent variable, which represent the ratio of time spent looking at a specific area of interest (AOI) to the total time fixated on all AOIs within a trial. Additionally, we employed growth curve analysis (GCA) to examine changes in PFTs across each trial.

The bimodal group exhibited greater fixation proportions for both target (b = 0.38, SE = 0.04, p < .001) and competitor conditions (b = 0.21, SE = 0.06, p < .001). Significant interactions between the quadratic terms and both target (b = –0.19, SE = 0.03, p < .001) and competitor conditions (b = –0.14, SE = 0.04, p < .001) indicated differing curved trajectories and rates of change in PFTs compared to the distractor condition. Additionally, a significant interaction between the competitor condition and group (b = –0.18, SE = 0.08, p = .02) suggested that the effects of the competitor condition on PFTs differed between unimodal and bimodal groups. A significant three-way interaction among the quadratic term, competitor condition, and group (b = 0.12, SE = 0.05, p = .02) indicated variation in the relationship between the quadratic term and fixation proportions for the competitor condition across groups.

The statistical results indicate that the bimodal group showed significantly greater fixation proportions for both target and competitor items, along with differing curved trajectories for each condition. This suggests that the bimodal group may have experienced simultaneous activation of HKSL while engaging in the Cantonese task. Further analysis revealed that the relationship between HKSL Sentence Repetition Test scores and fixation proportions varied across target and competitor conditions. The significant three-way interaction (b = –0.11, SE = 0.04, p = .008) indicates that attention patterns to target items are influenced by HKSL proficiency in bimodal bilinguals, with higher proficiency leading to stronger competition between target and competitor items. Additionally, a significant interaction was found between the quadratic term, years of HKSL exposure, and the competitor condition (b = 0.09, SE = 0.04, p = .01), while no significant interactions were found regarding the age of HKSL acquisition. This suggests that years of sign language exposure, rather than age at acquisition, influence bimodal participants’ performance in the visual world task.