Abstract
Identifying that the same meaning is expressed by, or can be inferred from, various language expressions is a major challenge for natural language understanding applications such as information extraction, question answering and automatic summarization. Dagan and Glickman [5] proposed Textual Entailment, the task of deciding whether a target text follows from a source text, as a unifying framework for modeling language variability, which has often been addressed in an application-specific manner. In this paper we describe the series of benchmarks developed for the textual entailment recognition task, known as the PASCAL RTE Challenges. As a concrete example, we describe in detail the second RTE challenge, in which our methodology was consolidated, and served as a basis for the subsequent RTE challenges. The impressive success of these challenges established textual entailment as an active research area in natural language processing, attracting a growing community of researchers.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Baker, C., Fillmore, C., Lowe, J.: The Berkeley Framenet project. In: Proceedings of the COLING-ACL, Montreal, Canada (1998)
Bar-Haim, R., Dagan, I., Dolan, B., Ferro, L., Giampiccolo, D., Magnini, B., Szpektor, I.: The Second PASCAL Recognising Textual Entailment Challenge. In: The Second PASCAL Challenges Workshop on Recognizing Textual Entailment (2006)
Bar-Haim, R., Szpecktor, I., Glickman, O.: Definition and analysis of intermediate entailment levels. In: Proceedings of the ACL Workshop on Empirical Modeling of Semantic Equivalence and Entailment, Ann Arbor, Michigan, pp. 55–60. Association for Computational Linguistics (June 2005)
Burger, J., Ferro, L.: Generating an entailment corpus from news headlines. In: Proceedings of the ACL Workshop on Empirical Modeling of Semantic Equivalence and Entailment, Ann Arbor, Michigan, pp. 49–54. Association for Computational Linguistics (June 2005)
Dagan, I., Glickman, O.: Probabilistic textual entailment: Generic applied modeling of language variability. In: PASCAL Workshop on Text Understanding and Mining (2004)
Dagan, I., Glickman, O., Magnini, B.: The PASCAL Recognising Textual Entailment Challenge. In: Quiñonero-Candela, J., Dagan, I., Magnini, B., d’Alché-Buc, F. (eds.) MLCW 2005. LNCS (LNAI), vol. 3944, pp. 177–190. Springer, Heidelberg (2006)
de Marneffe, M.C., MacCartney, B., Grenager, T., Cer, D., Rafferty, A., Manning, C.D.: Learning to distinguish valid textual entailments. In: The Second PASCAL Challenges Workshop on Recognizing Textual Entailment (2006)
Fellbaum, C. (ed.): WordNet: An Electronic Lexical Database. Language, Speech and Communication. MIT Press (1998)
Giampiccolo, D., Magnini, B., Dagan, I., Dolan, B.: The Third PASCAL Recognizing Textual Entailment Challenge. In: Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing (2007)
Giampiccolo, D., Trang Dang, H., Magnini, B., Dagan, I., Dolan, B.: The Fourth PASCAL Recognizing Textual Entailment Challenge. In: Proceedings of the TAC 2008 Workshop (2008)
Glickman, O., Dagan, I., Koppel, M.: A lexical alignment model for probabilistic textual entailment. In: Quiñonero-Candela, J., Dagan, I., Magnini, B., d’Alché-Buc, F. (eds.) MLCW 2005. LNCS (LNAI), vol. 3944, pp. 287–298. Springer, Heidelberg (2006)
Harabagiu, S., Hickl, A., Lacatusu, F.: Satisfying information needs with multi-document summaries. Inf. Process. Manage. 43(6), 1619–1642 (2007)
Hickl, A., Harabagiu, S.: Methods for using textual entailment in open-domain question answering. In: Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the ACL. Association for Computational Linguistics (2006)
Iftene, A., Balahur, A.: Answer validation on English and Romanian languages. In: Peters, C., Deselaers, T., Ferro, N., Gonzalo, J., Jones, G.J.F., Kurimo, M., Mandl, T., Peñas, A., Petras, V. (eds.) CLEF 2008. LNCS, vol. 5706, pp. 448–451. Springer, Heidelberg (2009)
Inkpen, D., Kipp, D., Nastase, V.: Machine learning experiments for textual entailment. In: The Second PASCAL Challenges Workshop on Recognizing Textual Entailment (2006)
Katrenko, S., Adriaans, P.: Using maximal embedded syntactic subtrees for textual entailment recognition. In: The Second PASCAL Challenges Workshop on Recognizing Textual Entailment (2006)
Landis, J.R., Koch, G.G.: The measurements of observer agreement for categorical data. Biometrics 33, 159–174 (1997)
Nicholson, J., Stokes, N., Baldwin, T.: Detecting entailment using an extended implementation of the basic elements overlap metric. In: The Second PASCAL Challenges Workshop on Recognizing Textual Entailment (2006)
Nielsen, R.D., Ward, W., Martin, J.H.: Classification errors in a domain-independent assessment system. In: Proceedings of the Third Workshop on Innovative Use of Natural Language Processing for Building Educational Applications, at the Forty-Sixth annual meeting of the Association for Computational Linguistics. ACL (2008)
Schilder, F., McInnes, B.T.: Word and tree-based similarities for textual entailment. In: The Second PASCAL Challenges Workshop on Recognizing Textual Entailment (2006)
Vanderwende, L., Dolan, W.B.: What syntax can contribute in the entailment task. In: Quiñonero-Candela, J., Dagan, I., Magnini, B., d’Alché-Buc, F. (eds.) MLCW 2005. LNCS (LNAI), vol. 3944, pp. 205–216. Springer, Heidelberg (2006)
Voorhees, E.M., Harman, D.: Overview of the seventh text retrieval conference. In: Proceedings of the Seventh Text REtrieval Conference (TREC-7). NIST Special Publication (1999)
Zanzotto, F.M., Moschitti, A., Pennacchiotti, M., Pazienza, M.T.: Learning textual entailment from examples. In: The Second PASCAL Challenges Workshop on Recognizing Textual Entailment (2006)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer-Verlag Berlin Heidelberg
About this chapter
Cite this chapter
Bar-Haim, R., Dagan, I., Szpektor, I. (2014). Benchmarking Applied Semantic Inference: The PASCAL Recognising Textual Entailment Challenges. In: Dershowitz, N., Nissan, E. (eds) Language, Culture, Computation. Computing - Theory and Technology. Lecture Notes in Computer Science, vol 8001. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-45321-2_19
Download citation
DOI: https://doi.org/10.1007/978-3-642-45321-2_19
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-45320-5
Online ISBN: 978-3-642-45321-2
eBook Packages: Computer ScienceComputer Science (R0)