Skip to main navigation Skip to search Skip to main content

Representations in vision and language converge in a shared, multidimensional space of perceived similarities

Research output: Contribution to journal › Article › peer-review

37 Downloads (Pure)

Abstract

Humans can effortlessly describe what they see, yet establishing a shared representational format between vision and language remains a significant challenge. Emerging evidence suggests that human brain representations in both vision and language are well predicted by semantic feature spaces obtained from large language models (LLMs). This raises the possibility that sensory systems converge in their inherent ability to transform their inputs onto shared, embedding-like representational space. However, it remains unclear how such a space manifests in human behavior. To investigate this, 63 participants performed behavioral similarity judgments separately on 100 natural scene images and 100 corresponding sentence captions from the Natural Scenes Dataset. We found that visual and linguistic similarity judgments not only converge at the behavioral level but also predict a remarkably similar network of functional magnetic resonance imaging brain responses evoked by viewing the natural scene images. Furthermore, computational models trained to map images onto LLM-embeddings outperformed both category-trained and AlexNet controls in predicting the behavioral similarity structure. These findings demonstrate that human visual and linguistic similarity judgments are grounded in a shared, modality-agnostic representational structure that mirrors how the visual system encodes experience. The convergence between sensory and artificial systems observed here suggests a common capacity of how conceptual representations are formed—not as arbitrary products of first order, modality-specific input, but as structured representations that reflect the stable, relational properties of the external world.

Original languageEnglish
Article number7
Number of pages14
JournalJournal of Vision
Volume26
Issue number5
DOIs
Publication statusPublished - 20 May 2026

Bibliographical note

Publisher Copyright:
Copyright 2026 The Authors. This work is licensed under a Creative Commons Attribution 4.0 International License https://creativecommons.org/licenses/by/4.0/.

Keywords

  • AlexNet
  • conceptual representation
  • language
  • large language models
  • representational similarity
  • scene perception
  • similarity judgments

ASJC Scopus subject areas

  • Ophthalmology
  • Sensory Systems

Fingerprint

Dive into the research topics of 'Representations in vision and language converge in a shared, multidimensional space of perceived similarities'. Together they form a unique fingerprint.

Cite this