Multimodality II: text and image

Research output: Chapter in Book/Report/Conference proceedingChapter

Colleges, School and Institutes

Abstract

This chapter navigates through the different approaches to multimodal analysis, involving the examination of communication where text and image interact, from the perspective of social semiotics multimodality, multimodal discourse analysis, and corpus-based methodology. It focuses mostly on latter, as it involves computer-assisted methods of analysis and has developed in recent years with the advent of digital humanities. It also examines the many challenges presented for the creation and analysis of multimodal corpora, from corpus design to theoretical generalisations.

One of the acutest problems in multimodal corpus analysis, especially when this involves non-linear data such as images, is how to make visual material searchable in such a way that specifically designed software could be used for their interrogation. Thus, this chapter will explore in depth issues of corpus annotation and will present a new methodological framework for the interrogation of large electronic collections of visual material, which relies on principles of visual content analysis. According to this framework, each relevant aspect of an image is converted into a variable, which can take certain values. Images can then be tagged with these variables and values, which can ultimately be converted into search parameters for corpus tools.

Finally, this framework is applied to the investigation of a multimodal corpus of popular science magazine covers in English and French. The focus is on the proportions of images of animate vs. inanimate participants occupying central position in the covers and on the proportions of verbs vs. nouns in the main headline, which accompanies the central image. Results reveal that there are both similarities and differences between the English and French magazine covers. Both publications are concerned with similar themes and topics, but these are communicated using different modes. Therefore, although these two publications might use visual elements differently, the message they convey is similar when the visual and textual modes are combined.

Details

Original languageEnglish
Title of host publicationThe Routledge handbook of English language and the digital humanities
EditorsSvenja Adolphs, Dawn Knight
Publication statusPublished - 5 May 2020

Publication series

NameRoutledge handbooks in English language studies
PublisherRoutledge