Overview of the 2017 Spoken CALL Shared Task

Claudia Baur; Cathy Chua; Johanna Gerlach; Manny Rayner; Martin Russell; Helmer Strik; Xizi Wei

Overview of the 2017 Spoken CALL Shared Task

Claudia Baur, Cathy Chua, Johanna Gerlach, Manny Rayner, Martin Russell, Helmer Strik, Xizi Wei

Electronic, Electrical and Systems Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

We present an overview of the shared task for spoken CALL. Groups competed on a prompt-response task using English language data collected, through an online CALL game, from Swiss German teens in their second and third years of learning
English. Each item consists of a written German prompt and an audio file containing a spoken response. The task is to accept linguistically correct responses and reject linguistically incorrect ones, with “linguistically correct” being defined by a gold standard derived from human annotations; scoring was
performed using a metric defined as the ratio of the relative rejection rates on incorrect and correct responses. The task received twenty entries from nine different groups. We present the task itself, the results, a tentative analysis of what makes items challenging, a comparison between different metrics, and
suggestions for a continuation.

Original language	English
Title of host publication	Proceedings of the 7th ISCA Workshop on Speech and Language Technology in Education
Editors	O Engwall, J Lopes, I Leite
Place of Publication	Stockholm, Sweden
Number of pages	8
Publication status	Published - 25 Aug 2017

Keywords

CALL
Shared Task
Automatic speech recognition
Metrics

Access to Document

http://www.slate2017.org/proceedings.pdf

Cite this

@inproceedings{7b8da0907f814f828fb4c5be328d4e89,

title = "Overview of the 2017 Spoken CALL Shared Task",

abstract = "We present an overview of the shared task for spoken CALL. Groups competed on a prompt-response task using English language data collected, through an online CALL game, from Swiss German teens in their second and third years of learningEnglish. Each item consists of a written German prompt and an audio file containing a spoken response. The task is to accept linguistically correct responses and reject linguistically incorrect ones, with “linguistically correct” being defined by a gold standard derived from human annotations; scoring wasperformed using a metric defined as the ratio of the relative rejection rates on incorrect and correct responses. The task received twenty entries from nine different groups. We present the task itself, the results, a tentative analysis of what makes items challenging, a comparison between different metrics, andsuggestions for a continuation.",

keywords = "CALL, Shared Task, Automatic speech recognition, Metrics",

author = "Claudia Baur and Cathy Chua and Johanna Gerlach and Manny Rayner and Martin Russell and Helmer Strik and Xizi Wei",

year = "2017",

month = aug,

day = "25",

language = "English",

editor = "O Engwall and J Lopes and I Leite",

booktitle = "Proceedings of the 7th ISCA Workshop on Speech and Language Technology in Education",

}

TY - GEN

T1 - Overview of the 2017 Spoken CALL Shared Task

AU - Baur, Claudia

AU - Chua, Cathy

AU - Gerlach, Johanna

AU - Rayner, Manny

AU - Russell, Martin

AU - Strik, Helmer

AU - Wei, Xizi

PY - 2017/8/25

Y1 - 2017/8/25

N2 - We present an overview of the shared task for spoken CALL. Groups competed on a prompt-response task using English language data collected, through an online CALL game, from Swiss German teens in their second and third years of learningEnglish. Each item consists of a written German prompt and an audio file containing a spoken response. The task is to accept linguistically correct responses and reject linguistically incorrect ones, with “linguistically correct” being defined by a gold standard derived from human annotations; scoring wasperformed using a metric defined as the ratio of the relative rejection rates on incorrect and correct responses. The task received twenty entries from nine different groups. We present the task itself, the results, a tentative analysis of what makes items challenging, a comparison between different metrics, andsuggestions for a continuation.

AB - We present an overview of the shared task for spoken CALL. Groups competed on a prompt-response task using English language data collected, through an online CALL game, from Swiss German teens in their second and third years of learningEnglish. Each item consists of a written German prompt and an audio file containing a spoken response. The task is to accept linguistically correct responses and reject linguistically incorrect ones, with “linguistically correct” being defined by a gold standard derived from human annotations; scoring wasperformed using a metric defined as the ratio of the relative rejection rates on incorrect and correct responses. The task received twenty entries from nine different groups. We present the task itself, the results, a tentative analysis of what makes items challenging, a comparison between different metrics, andsuggestions for a continuation.

KW - CALL

KW - Shared Task

KW - Automatic speech recognition

KW - Metrics

M3 - Conference contribution

BT - Proceedings of the 7th ISCA Workshop on Speech and Language Technology in Education

A2 - Engwall, O

A2 - Lopes, J

A2 - Leite, I

CY - Stockholm, Sweden

ER -

Overview of the 2017 Spoken CALL Shared Task

Abstract

Keywords

Access to Document

Fingerprint

Cite this