Tsuneaki Kato
Abstract
There are strong expectations for the use of question answering technologies in information access dialogues, such as for information gathering and browsing. In this paper, we empirically examine what kinds of abilities are needed for question answering systems in such situations, and propose a challenge for evaluating those abilities objectively and quantitatively. We also show that existing technologies have the potential to address this challenge. From the empirical study, we found that questions that have values and names as answers account for a majority in realistic information-gathering situations and that those sequences of questions contain a wide range of reference expressions and are sometimes complicated by the inclusion of subdialogues and focus shifts. The challenge proposed is not only novel as an evaluation of the handling of information access dialogues, but also includes several valuable ideas such as categorization and characterization of information access dialogues, and introduces three measures to evaluate various aspects in addressing list-type questions and reference test sets for evaluating context-processing ability in isolation.
- Akiba, T., Itou, K., and Fujii, A. 2004. Question answering using “common sense” and utility maximization principle. Working Notes of the Fourth NTCIR Workshop Meeting. 297--303.Google Scholar
- AQUAINT Home Page: Advanced Question & Answering for Intelligence 2003. http://www.ic-arda.org/InfoExploit/aquaint/.Google Scholar
- Burger, J., Cardie, C., Chaudhri, V. 2001. Issues, Tasks and Program Structures to Roadmap Research in Question & Answering (Q&A) http://www-nlpir.nist.gov/projrcts/duc/roadmpping.html.Google Scholar
- Chai, J. Y. and Jin, R. 2004. Discource structure for context question answering. Proceedings of HLT-NAACL2004, Workshop on Pragmatics of Question Answering. 23--30.Google Scholar
- Fukumoto, J., Kato, T., and Masui, F. 2003. Question Answering Challenge (QAC-1) An Evaluation of question answering tasks at the NTCIR workshop 3. AAAI 2003 Spring Symposium New Directions in Question Answering. 122--133.Google Scholar
- Fukumoto, J., Niwa, T., Itoigawa, M., and Matuda, M. 2004. Rits-QA: List answer detection and context task with ellipses handling. Working Notes of the Fourth NTCIR Workshop Meeting. 310--314.Google Scholar
- Harabagiu, S., Moldovan, D., Paşca, M. 2001. Answering complex, list and context questions with LCC's Question-Answering Server. Proceedings of TREC 2001.Google Scholar
- Hickl, A., Lehmann, J., Williams, J., and Harabagiu, S. 2004. Experiments with interactive question answering in complex scenarios. Proceedings of HLT-NAACL2004 Workshop on Pragmatics of Question Answering. 60--69.Google Scholar
- Hidaka, N., Masui, F., and Tosaki, K. 2004. MAIQA: Mie Univ. Participated System at NTCIR4 QAC2. Working Notes of the Fourth NTCIR Workshop Meeting. 315--319.Google Scholar
- Hovy, E. 2001. http://www-nlpir.nist.gov/projects/duc/pubs/2001papers/isi_hovy_duc.pdf.Google Scholar
- Lehnert, W. G. 1977. A conceptual theory of question answering. Proceedings of the Fifth International Joint Conference on Artificial Intelligence. 158--164.Google Scholar
- Liddy, E. D. 2002. Why are people asking these question?: A call for bringing situation into question-answering system evaluation. LREC Workshop Proceedings on Question Answering · Strategy and Resources. 5--8.Google Scholar
- Liddy, E. D. 2003. Preparing to explore a new paradigm in information access: A scenario approach to question-answering. http://nrrc.mitre.org/NRRC/workshop03/Scenario_BaseQAWriteup.htm.Google Scholar
- Mani, I., House, D., Klein, G. 1998. The TIPSER SUMMAC text summarization evaluation final report. Technical Report MTR98W0000138, The MITRE Corporation.Google Scholar
- NTCIR (NII-NACSIS Test Collection for IR Systems) Project Home Page. 2003. http://research.nii.ac.jp/ntcir/index-en.html.Google Scholar
- Ram, A. 1999. A theory of questions and question asking. Understanding Language Understanding eds. Ashwin Ram and Kenneth Moorman, MIT Press, Cambridge, MA. 253--298. Google Scholar
- Small, S., Shimizu, N., Strzalkowski, T., and Ting, L. 2003. HITIQA: A data driven approach to interactive question answering: A preliminary report AAAI 2003 Spring Symposium New Directions in Question Answering. 94--104. Google Scholar
- Takaki, T. 2004. NTT DATA Question-answering experiment at the NTCIR-4 QAC2. Working Notes of the Fourth NTCIR Workshop Meeting. 402--405.Google Scholar
- TREC Home Page. 2003. http://trec.nist.gov/.Google Scholar
- Text Summarization Challenge Home Page. 2003. http://lr-www.pi.titech.ac.jp/tsc/index-en.html.Google Scholar
- Voorhees, E. M. and Tice, D. M. 2000. Building a question answering test collection. In Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 200--207. Google Scholar
- Voorhees, E. M. 2001. Overview of the TREC 2001 question answering track. Proceedings of TREC 2001. Google Scholar
Index Terms
- Are open-domain question answering technologies useful for information access dialogues?---an empirical study and a proposal of a novel challenge
Recommendations
Evaluating Temporal Information Understanding with Temporal Question Answering
ICSC '12: Proceedings of the 2012 IEEE Sixth International Conference on Semantic ComputingThe temporal annotation scheme Time ML was developed to support research in complex temporal question answering (QA). Given the complexity of temporal QA, most of the efforts have focused, so far, on extracting temporal information, which has been ...
Implementing clarification dialogues in open domain question answering
We examine the implementation of clarification dialogues, a mechanism for ensuring that question answering systems take into account user goals by allowing them to ask series of related questions either by refining or expanding on previous questions ...
Conversational question answering: a survey
AbstractQuestion answering (QA) systems provide a way of querying the information available in various formats including, but not limited to, unstructured and structured data in natural languages. It constitutes a considerable part of conversational ...
Comments