Risks of misinterpretation in the evaluation of distant supervision for relation extraction

Juan Luis Garcia-Mendoza, Luis Villasenor-Pineda, Felipe Orihuela-Espina

Research output: Contribution to journalArticlepeer-review

26 Downloads (Pure)


Distant Supervision is frequently used for addressing Relation Extraction. The evaluation of Distant Supervision in Relation Extraction has been attempted through Precision-Recall curves and/or calculation of Precision at N elements. However, such evaluation is challenging because the labeling of the instances results from an automatic process that can introduce noise into the labels. Consequently, the labels are not necessarily correct, affecting the learning process and the interpretation of the evaluation results. Therefore, this research aims to show that the performance of the methods measured with the mentioned evaluation strategies varies significantly if the correct labels are used during the evaluation. Besides, based on the preceding, the current interpretation of the results of these measures is questioned. To this end, we manually labeled a subset of a well-known data set and evaluated the performance of 6 traditional Distant Supervision approaches. We demonstrate quantitative differences in the evaluation scores when considering manually versus automatically labeled subsets. Consequently, the ranking of performance among distant supervision methods is different with both labeled.

Original languageEnglish
Pages (from-to)71-83
Number of pages13
JournalProcesamiento de Lenguaje Natural
Publication statusPublished - 30 Mar 2022

Bibliographical note

Funding Information:
The present work was supported by CONA- CyT/México (scholarship 937210 and grant CB-2015-01-257383). Additionally, the authors thank CONACYT for the computer resources provided through the INAOE Supercomputing Laboratory’s Deep Learning Platform for Language Technologies.

Publisher Copyright:
© 2022 Sociedad Espanola para el Procesamiento del Lenguaje Natural. All rights reserved.


  • Distant Supervision evaluation
  • Precision at N
  • Precision-Recall curves
  • Relation Extraction

ASJC Scopus subject areas

  • Language and Linguistics
  • Linguistics and Language
  • Computer Science Applications


Dive into the research topics of 'Risks of misinterpretation in the evaluation of distant supervision for relation extraction'. Together they form a unique fingerprint.

Cite this