Semantic vector evaluation and human performance on a new vocabulary MCQ test

Joseph P. Levy; John Bullinaria; Samantha  McCormick

Semantic vector evaluation and human performance on a new vocabulary MCQ test

Joseph P. Levy, John Bullinaria, Samantha McCormick

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Vectors derived from patterns of co-occurrence of words in large bodies of text have often been used as representations of some aspects of the meanings of different words. Generally, the distance between such vectors is used as a measure of the semantic similarity between the word meanings they represent. One important way of evaluating the performance of these vectors has been to use them to answer vocabulary multiple choice questions (MCQs) where the participant is asked to judge which of several choice words is closest in meaning to a stem word. The existing vocabulary MCQ tests used in this way have been very useful but there are some practical problems in their use as general evaluation measures. Here, we discuss why such tests remain useful
evaluation measures, introduce a new vocabulary test, evaluate several current sets of semantic vectors using the new test and compare their performance to human data.

Original language	English
Title of host publication	Proceedings of the 39th Annual Meeting of the Cognitive Science Society (CogSci 2017)
Editors	Glenn Gunzelmann, Andrew Howes, Thora Tenbrink, Eddy Davelaar
Place of Publication	Austin, TX
Publisher	Cognitive Science Society
Pages	2549-2554
ISBN (Print)	978-0-9911967-6-0
Publication status	Published - 26 Jul 2017
Event	39th Annual Meeting of the Cognitive Science Society: Computational Foundations of Cognition - London, United Kingdom Duration: 26 Jul 2017 → 29 Jul 2017

Conference

Conference	39th Annual Meeting of the Cognitive Science Society
Country/Territory	United Kingdom
City	London
Period	26/07/17 → 29/07/17

Keywords

Distributional semantics
vocabulary MCQ

Access to Document

https://mindmodeling.org/cogsci2017/cogsci17_proceedings.pdf

Cite this

Levy, J. P., Bullinaria, J., & McCormick, S. (2017). Semantic vector evaluation and human performance on a new vocabulary MCQ test. In G. Gunzelmann, A. Howes, T. Tenbrink, & E. Davelaar (Eds.), Proceedings of the 39th Annual Meeting of the Cognitive Science Society (CogSci 2017) (pp. 2549-2554). Cognitive Science Society. https://mindmodeling.org/cogsci2017/cogsci17_proceedings.pdf

@inproceedings{f1c4bb228e0748d9bcf789ad57bef652,

title = "Semantic vector evaluation and human performance on a new vocabulary MCQ test",

abstract = "Vectors derived from patterns of co-occurrence of words in large bodies of text have often been used as representations of some aspects of the meanings of different words. Generally, the distance between such vectors is used as a measure of the semantic similarity between the word meanings they represent. One important way of evaluating the performance of these vectors has been to use them to answer vocabulary multiple choice questions (MCQs) where the participant is asked to judge which of several choice words is closest in meaning to a stem word. The existing vocabulary MCQ tests used in this way have been very useful but there are some practical problems in their use as general evaluation measures. Here, we discuss why such tests remain usefulevaluation measures, introduce a new vocabulary test, evaluate several current sets of semantic vectors using the new test and compare their performance to human data.",

keywords = "Distributional semantics, vocabulary MCQ",

author = "Levy, {Joseph P.} and John Bullinaria and Samantha McCormick",

year = "2017",

month = jul,

day = "26",

language = "English",

isbn = "978-0-9911967-6-0",

pages = "2549--2554",

editor = "Glenn Gunzelmann and Andrew Howes and Thora Tenbrink and Eddy Davelaar",

booktitle = "Proceedings of the 39th Annual Meeting of the Cognitive Science Society (CogSci 2017)",

publisher = "Cognitive Science Society",

address = "United States",

note = "39th Annual Meeting of the Cognitive Science Society : Computational Foundations of Cognition ; Conference date: 26-07-2017 Through 29-07-2017",

}

Levy, JP, Bullinaria, J & McCormick, S 2017, Semantic vector evaluation and human performance on a new vocabulary MCQ test. in G Gunzelmann, A Howes, T Tenbrink & E Davelaar (eds), Proceedings of the 39th Annual Meeting of the Cognitive Science Society (CogSci 2017). Cognitive Science Society, Austin, TX, pp. 2549-2554, 39th Annual Meeting of the Cognitive Science Society, London, United Kingdom, 26/07/17. <https://mindmodeling.org/cogsci2017/cogsci17_proceedings.pdf>

Semantic vector evaluation and human performance on a new vocabulary MCQ test. / Levy, Joseph P.; Bullinaria, John; McCormick, Samantha .
Proceedings of the 39th Annual Meeting of the Cognitive Science Society (CogSci 2017). ed. / Glenn Gunzelmann; Andrew Howes; Thora Tenbrink; Eddy Davelaar. Austin, TX: Cognitive Science Society, 2017. p. 2549-2554.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Semantic vector evaluation and human performance on a new vocabulary MCQ test

AU - Levy, Joseph P.

AU - Bullinaria, John

AU - McCormick, Samantha

PY - 2017/7/26

Y1 - 2017/7/26

N2 - Vectors derived from patterns of co-occurrence of words in large bodies of text have often been used as representations of some aspects of the meanings of different words. Generally, the distance between such vectors is used as a measure of the semantic similarity between the word meanings they represent. One important way of evaluating the performance of these vectors has been to use them to answer vocabulary multiple choice questions (MCQs) where the participant is asked to judge which of several choice words is closest in meaning to a stem word. The existing vocabulary MCQ tests used in this way have been very useful but there are some practical problems in their use as general evaluation measures. Here, we discuss why such tests remain usefulevaluation measures, introduce a new vocabulary test, evaluate several current sets of semantic vectors using the new test and compare their performance to human data.

AB - Vectors derived from patterns of co-occurrence of words in large bodies of text have often been used as representations of some aspects of the meanings of different words. Generally, the distance between such vectors is used as a measure of the semantic similarity between the word meanings they represent. One important way of evaluating the performance of these vectors has been to use them to answer vocabulary multiple choice questions (MCQs) where the participant is asked to judge which of several choice words is closest in meaning to a stem word. The existing vocabulary MCQ tests used in this way have been very useful but there are some practical problems in their use as general evaluation measures. Here, we discuss why such tests remain usefulevaluation measures, introduce a new vocabulary test, evaluate several current sets of semantic vectors using the new test and compare their performance to human data.

KW - Distributional semantics

KW - vocabulary MCQ

M3 - Conference contribution

SN - 978-0-9911967-6-0

SP - 2549

EP - 2554

BT - Proceedings of the 39th Annual Meeting of the Cognitive Science Society (CogSci 2017)

A2 - Gunzelmann, Glenn

A2 - Howes, Andrew

A2 - Tenbrink, Thora

A2 - Davelaar, Eddy

PB - Cognitive Science Society

CY - Austin, TX

T2 - 39th Annual Meeting of the Cognitive Science Society

Y2 - 26 July 2017 through 29 July 2017

ER -

Semantic vector evaluation and human performance on a new vocabulary MCQ test

Abstract

Conference

Keywords

Access to Document

Fingerprint

Cite this