Multimodal affect recognition in an interactive gaming environment using eye tracking and speech signals

Ashwaq Alhargan; Neil Cooke; Tareq Binjammaz

doi:10.1145/3136755.3137016

Multimodal affect recognition in an interactive gaming environment using eye tracking and speech signals

Ashwaq Alhargan, Neil Cooke, Tareq Binjammaz

Electronic, Electrical and Systems Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

8 Citations (Scopus)

Abstract

This paper presents a multimodal affect recognition system for interactive virtual gaming environments using eye tracking and speech signals, captured in gameplay scenarios that are designed to elicit controlled affective states based on the arousal and valence dimensions. The Support Vector Machine is employed as a classifier to detect these affective states from both modalities. The recognition results reveal that eye tracking is superior to speech in affect detection and that the two modalities are complementary in the application of interactive gaming. This suggests that it is feasible to design an accurate multimodal recognition system to detect players' affects from the eye tracking and speech modalities in the interactive gaming environment. We emphasise the potential of integrating the proposed multimodal system into game interfaces to enhance interaction and provide an adaptive gaming experience.

Original language	English
Title of host publication	ICMI 2017 - Proceedings of the 19th ACM International Conference on Multimodal Interaction
Place of Publication	New York
Publisher	Association for Computing Machinery
Pages	479-486
Number of pages	8
Volume	2017-January
ISBN (Electronic)	9781450355438
DOIs	https://doi.org/10.1145/3136755.3137016
Publication status	Published - 3 Nov 2017
Event	19th ACM International Conference on Multimodal Interaction, ICMI 2017 - Glasgow, United Kingdom Duration: 13 Nov 2017 → 17 Nov 2017

Conference

Conference	19th ACM International Conference on Multimodal Interaction, ICMI 2017
Country/Territory	United Kingdom
City	Glasgow
Period	13/11/17 → 17/11/17

Keywords

Affects
Eye tracking
Games
Speech

ASJC Scopus subject areas

Human-Computer Interaction
Computer Science Applications
Computer Vision and Pattern Recognition
Hardware and Architecture

Access to Document

10.1145/3136755.3137016Licence: None: All rights reserved

Cite this

@inproceedings{2637b5e57eb34923a1b5acada120ccf5,

title = "Multimodal affect recognition in an interactive gaming environment using eye tracking and speech signals",

abstract = "This paper presents a multimodal affect recognition system for interactive virtual gaming environments using eye tracking and speech signals, captured in gameplay scenarios that are designed to elicit controlled affective states based on the arousal and valence dimensions. The Support Vector Machine is employed as a classifier to detect these affective states from both modalities. The recognition results reveal that eye tracking is superior to speech in affect detection and that the two modalities are complementary in the application of interactive gaming. This suggests that it is feasible to design an accurate multimodal recognition system to detect players' affects from the eye tracking and speech modalities in the interactive gaming environment. We emphasise the potential of integrating the proposed multimodal system into game interfaces to enhance interaction and provide an adaptive gaming experience.",

keywords = "Affects, Eye tracking, Games, Speech",

author = "Ashwaq Alhargan and Neil Cooke and Tareq Binjammaz",

year = "2017",

month = nov,

day = "3",

doi = "10.1145/3136755.3137016",

language = "English",

volume = "2017-January",

pages = "479--486",

booktitle = "ICMI 2017 - Proceedings of the 19th ACM International Conference on Multimodal Interaction",

publisher = "Association for Computing Machinery ",

note = "19th ACM International Conference on Multimodal Interaction, ICMI 2017 ; Conference date: 13-11-2017 Through 17-11-2017",

}

Alhargan, A, Cooke, N & Binjammaz, T 2017, Multimodal affect recognition in an interactive gaming environment using eye tracking and speech signals. in ICMI 2017 - Proceedings of the 19th ACM International Conference on Multimodal Interaction. vol. 2017-January, Association for Computing Machinery , New York, pp. 479-486, 19th ACM International Conference on Multimodal Interaction, ICMI 2017, Glasgow, United Kingdom, 13/11/17. https://doi.org/10.1145/3136755.3137016

Multimodal affect recognition in an interactive gaming environment using eye tracking and speech signals. / Alhargan, Ashwaq; Cooke, Neil; Binjammaz, Tareq.
ICMI 2017 - Proceedings of the 19th ACM International Conference on Multimodal Interaction. Vol. 2017-January New York: Association for Computing Machinery , 2017. p. 479-486.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Multimodal affect recognition in an interactive gaming environment using eye tracking and speech signals

AU - Alhargan, Ashwaq

AU - Cooke, Neil

AU - Binjammaz, Tareq

PY - 2017/11/3

Y1 - 2017/11/3

N2 - This paper presents a multimodal affect recognition system for interactive virtual gaming environments using eye tracking and speech signals, captured in gameplay scenarios that are designed to elicit controlled affective states based on the arousal and valence dimensions. The Support Vector Machine is employed as a classifier to detect these affective states from both modalities. The recognition results reveal that eye tracking is superior to speech in affect detection and that the two modalities are complementary in the application of interactive gaming. This suggests that it is feasible to design an accurate multimodal recognition system to detect players' affects from the eye tracking and speech modalities in the interactive gaming environment. We emphasise the potential of integrating the proposed multimodal system into game interfaces to enhance interaction and provide an adaptive gaming experience.

AB - This paper presents a multimodal affect recognition system for interactive virtual gaming environments using eye tracking and speech signals, captured in gameplay scenarios that are designed to elicit controlled affective states based on the arousal and valence dimensions. The Support Vector Machine is employed as a classifier to detect these affective states from both modalities. The recognition results reveal that eye tracking is superior to speech in affect detection and that the two modalities are complementary in the application of interactive gaming. This suggests that it is feasible to design an accurate multimodal recognition system to detect players' affects from the eye tracking and speech modalities in the interactive gaming environment. We emphasise the potential of integrating the proposed multimodal system into game interfaces to enhance interaction and provide an adaptive gaming experience.

KW - Affects

KW - Eye tracking

KW - Games

KW - Speech

UR - http://www.scopus.com/inward/record.url?scp=85046760293&partnerID=8YFLogxK

U2 - 10.1145/3136755.3137016

DO - 10.1145/3136755.3137016

M3 - Conference contribution

AN - SCOPUS:85046760293

VL - 2017-January

SP - 479

EP - 486

BT - ICMI 2017 - Proceedings of the 19th ACM International Conference on Multimodal Interaction

PB - Association for Computing Machinery

CY - New York

T2 - 19th ACM International Conference on Multimodal Interaction, ICMI 2017

Y2 - 13 November 2017 through 17 November 2017

ER -

Multimodal affect recognition in an interactive gaming environment using eye tracking and speech signals

Abstract

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Fingerprint

Cite this