Abstract
Evaluation is central in research and development of information retrieval (IR). In addition to designing and implementing new retrieval mechanisms, one must also show through rigorous evaluation that they are effective. A major focus in IR is IR mechanisms’ capability of ranking relevant documents optimally for the users, given a query. Searching for information in practice involves searchers, however, and is highly interactive. When human searchers have been incorporated in evaluation studies, the results have often suggested that better ranking does not necessarily lead to better search task, or work task, performance. Therefore, it is not clear which system or interface features should be developed to improve the effectiveness of human task performance. In the present article, we focus on the evaluation of task-based information interaction (TBII). We give special emphasis to learning tasks to discuss TBII in more concrete terms. Information interaction is here understood as behavioral and cognitive activities related to task planning, searching information items, selecting between them, working with them, and synthesizing and reporting. These five generic activities contribute to task performance and outcome and can be supported by information systems. In an attempt toward task-based evaluation, we introduce program theory as the evaluation framework. Such evaluation can investigate whether a program consisting of TBII activities and tools works and how it works and, further, provides a causal description of program (in)effectiveness. Our goal in the present article is to structure TBII on the basis of the five generic activities and consider the evaluation of each activity using the program theory framework. Finally, we combine these activity-based program theories in an overall evaluation framework for TBII. Such an evaluation is complex due to the large number of factors affecting information interaction. Instead of presenting tested program theories, we illustrate how the evaluation of TBII should be accomplished using the program theory framework in the evaluation of systems and behaviors, and their interactions, comprehensively in context.
- A. Adler, A. Gujar, B. L. Harrison, K. O’Hara, and A. Sellen. 1998. A diary study of work-related reading: Design implications for digital reading devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’98). ACM/Addison-Wesley, New York, NY, 241--248. Google ScholarDigital Library
- T. D. Anderson. 2006. Uncertainty in action: Observing information seeking within the creative processes of scholarly research. Inf. Res. 12, 1 (2006), Article 283.Google Scholar
- P. Arvola and J. Kekäläinen. 2010. Simulating user interaction in result document browsing. In Workshop Proceedings of the 33rd Annual International ACM SIGIR Conference: Simulation of Interaction, Automated Evaluation of Interactive IR. ACM, New York, NY, 27--28.Google Scholar
- P. Arvola, J. Kekäläinen, and M. Junkkari. 2010. Expected reading effort in focused retrieval evaluation. Inf. Retrieval 13, 5 (2010), 460--484. Google ScholarDigital Library
- P. Arvola, J. Kekäläinen, and M. Junkkari. 2011. Contextualization models for XML retrieval. Inform. Process. Manage. 47, 5 (2011), 762--776. Google ScholarDigital Library
- S. Attfield and J. Dowell. 2003. Information seeking and use by newspaper journalists. J. Doc. 59, 2 (2003), 187--204.Google ScholarCross Ref
- C. L. Barry and L. Schamber. 1998. Users’ criteria for relevance evaluation: A cross-situational comparison. Inform. Process. Manage. 34, 2/3 (1998), 219--236. Google ScholarDigital Library
- M. J. Bates. 1989. The design of browsing and berrypicking techniques for the online search interface. Online Inform. Rev. 13, 5 (1989), 407--424.Google ScholarCross Ref
- M. J. Bates. 1990. Where should the person stop and the information search interface start? Inform. Process. Manage. 26, 5 (1990), 575--591. Google ScholarDigital Library
- M. J. Bates. 2007. What is browsing—really? A model drawing from behavioural science research. Inf. Res. 12, 4 (2007), Article 330.Google Scholar
- N. J. Belkin. 1980. Anomalous states of knowledge as a basis for information retrieval. Can. J. Inform. Lib. Sci. 5 (1980), 133--143.Google Scholar
- N. J. Belkin. 2010. On the evaluation of interactive information retrieval systems. In The Janus Faced Scholar. A Festschrift in Honor of Peter Ingwersen. Det Informationsvidenskabelige Akademi (Royal School of Library and Information Science, Copenhagen); ISSI (International Society for Scientometrics and Informetrics), Copenhagen, Denmark, 13--22.Google Scholar
- N. J. Belkin, R. Seeger, and G. Wersig. 1983. Distributed expert problem treatment as a model for information system analysis and design. J. Inform. Sci. 5 (1983), 153--167.Google ScholarCross Ref
- E. S. Berner. 2007. Clinical Decision Support Systems. Springer, New York, NY. Google ScholarDigital Library
- J. Biggs. 1988. The role of metacognition in enhancing learning. Aust. J. Educ. 32, 2 (1988), 127--138.Google ScholarCross Ref
- A. Blandford and S. Attfield. 2010. Interacting with Information. Morgan and Claypool, San Rafael, CA. Google ScholarDigital Library
- B. S. Bloom, M. D. Engelhart, E. J. Furst, W. H. Hill, and D. R. Krathwohl. 1956. Taxonomy of Educational Objectives: The Classification of Educational Goals. David McKay, New York, NY.Google Scholar
- A. Bookstein. 1983. Information retrieval: A sequential learning process. J. Am. Soc. Inf. Sci. 34, 5 (1983), 331--342.Google ScholarCross Ref
- C. L. Borgman. 1985. The user's mental model of an information retrieval system. In Proceedings of the 8th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’85). ACM, New York, NY, 268--273. Google ScholarDigital Library
- P. Borlund. 2000. Experimental components for the evaluation of interactive information retrieval systems. J. Doc. 50, 1 (2000), 71--90.Google ScholarCross Ref
- P. Borlund. 2003. The concept of relevance in IR. J. Am. Soc. Inf. Sci. Technol. 54, 10 (2003), 913--925. Google ScholarDigital Library
- L. Bowler. 2010. A taxonomy of adolescent metacognitive knowledge during the information search process. Libr. Inform. Sci. Res. 32, 1 (2010), 27--42.Google ScholarCross Ref
- M. Bron, J. van Gorp, F. Nack, M. de Rijke, A. Vishneuski, and S. de Leeuw. 2012. A subjunctive exploratory search interface to support media studies researchers. In Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’12). ACM, New York, NY, 425--434. doi:10.1145/2348283.2348342 Google ScholarDigital Library
- K. R. Butcher, S. Davies, A. Crockett, A. Dewald, and R. Zheng. 2011. Do graphical search interfaces support effective search for and evaluation of digital library resources? In Proceedings of the 11th Annual International ACM/IEEE Joint Conference on Digital Libraries (JCDL’11). ACM, New York, NY, 315--324. doi:10.1145/1998076 Google ScholarDigital Library
- K. Byström and P. Hansen. 2005. Conceptual framework for tasks in information studies. J. Am. Soc. Inf. Sci. Technol. 56, 10 (2005), 1050--1061. Google ScholarDigital Library
- K. Byström and K. Järvelin. 1995. Task complexity affects information seeking and use. Inform. Process. Manage. 31, 2 (1995), 191--213. Google ScholarDigital Library
- M. Carr, B. E. Kurtz, W. Schneider, L. A. Turner, and J. G. Borkowski. 1989. Strategy acquisition and transfer among German and American children: Environmental influences on metacognitive development. Dev. Psychol. 25, 5 (1989), 765--771.Google ScholarCross Ref
- P. Checkland and S. Holwell. 1998. Action research: Its nature and validity. Syst. Pract. Act. Res. 11, 1 (1998), 9--21.Google ScholarCross Ref
- C.-M. Chen and S.-T. Lin. 2014. Assessing effects of information architecture of digital libraries on supporting e-learning: A case study on the Digital Library of Nature & Culture. Comput. Educ. 75, 1 (2014), 92--102.Google ScholarCross Ref
- C. Cool and N. J. Belkin. 2002. A classification of interactions with information. In Emerging Frameworks and Methods, Proceedings of the 4th International Conference on Conceptions of Library and Information Science (CoLIS4). Greenwood Village, CO, Libraries Unlimited, 1--15.Google Scholar
- C. L. S. Coryn, L. A. Noakes, C. D. Westine, and D. C. Schröter. 2011. A systematic review of theory-driven evaluation practice from 1990 to 2009. Am. J. Eval. 32, 2 (2011), 199--226.Google ScholarCross Ref
- E. Cosijn and P. Ingwersen. 2000. Dimensions of relevance. Inform. Process. Manage. 36, 4 (2000), 533--550. Google ScholarDigital Library
- A. Dillon and D. Turnbull. 2005. Information architecture. In Encyclopedia of Library and Information Sciences (3rd ed.). 1, 1 (2005), 2361--2368.Google Scholar
- J. H. Flavell. 1979. Metacognition and cognitive monitoring: A new area of cognitive-developmental inquiry. Am. Psychol. 34, 10 (1979), 906--911.Google ScholarCross Ref
- L. Flower and J. R. Hayes. 1981. A cognitive process theory of writing. College Compos. Commun. 32, 4 (1981), 365--387.Google ScholarCross Ref
- L. Freund, E. G. Toms, and C. L. A. Clarke. 2005. Modeling task-genre relationships for IR in the workplace. In Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’05). ACM, New York, NY, 441--448. Google ScholarDigital Library
- S. Goldman, K. Lawless, and F. Manning. 2013. Research and development of multiple source comprehension assessment. In Reading - From Words to Multiple Texts, M. A. Britt, S. R. Goldman, and J.-F. Rouet (Eds.). Routledge, New York, NY, 180--199.Google Scholar
- K. Halttunen and K. Järvelin. 2005. Assessing learning outcomes in two information retrieval learning environments. Inform. Process. Manage. 41, 4 (2005), 949--972. Google ScholarDigital Library
- P. Hansen. 2011. Task-Based Information Seeking and Retrieval in the Patent Domain: Processes and Relationships. Acta Universitatis Tamperensis: 1631, Tampere University Press, Tampere, Finland.Google Scholar
- J. Heinström. 2005. Fast surfing, broad scanning and deep diving: The influence of personality and study approach on students’ information-seeking behavior. J. Doc. 61, 2 (2005), 228--247.Google ScholarCross Ref
- J. Heinström. 2006. Fast surfing for availability or deep diving into quality—motivation and information seeking among middle and high school students. Inform. Res. 11, 4 (2006), Article 433.Google Scholar
- W. Hersh. 1994. Relevance and retrieval evaluation: Perspectives from medicine. J. Am. Soc. Inf. Sci. 45, 1 (1994), 201--206. Google ScholarDigital Library
- W. Hersh, J. Pentecost, and D. Hickam. 1996. A task-oriented approach to information retrieval. J. Am. Soc. Inf. Sci. 47, 1 (1996), 50--56. Google ScholarDigital Library
- R. Howard, T. Serviss, and T. Rodrigue. 2010. Writing from sources, writing from sentences. Writing and Pedagogy 2, 2 (2010), 177--192.Google ScholarCross Ref
- S. Huuskonen and P. Vakkari. 2010. Client information system as an everyday information tool in child protection work. In Proceedings of the 3rd Symposium on Information Interaction in Context (IIiX’10). ACM, New York, NY, 3--12. Google ScholarDigital Library
- P. Ingwersen and K. Järvelin. 2005. The Turn: Integration of Information Seeking and Retrieval in Context. Springer, Dordrecht, The Netherlands. Google Scholar
- K. Järvelin. 2009. Explaining user performance in information retrieval: Challenges to IR evaluation. In Proceedings of the 2nd International Conference on the Theory of Information Retrieval (ICTIR’09). Springer, Berlin, Germany, 289--296. Google ScholarDigital Library
- K. Järvelin. 2011. Evaluation. In Interactive Information Seeking, Behaviour and Retrieval, I. Ruthven and D. Kelly (Eds.). Facet Publishing, London, UK, 113--138.Google Scholar
- M. G. Jones, J. D. Farquhar, and D. D. Surry. 1995. Using metacognitive theories to design user interfaces for computer-based learning. Educational Technol. 35, 4 (1995), 12--22.Google Scholar
- J. Kekäläinen, P. Arvola, and S. Kumpulainen. 2014. Browsing patterns in retrieved documents. In Proceedings of the 5th Information Interaction in Context Symposium (IIIX’14). ACM, New York, NY, 299--302. Google ScholarDigital Library
- J. Kekäläinen and K. Järvelin. 2002. Evaluating information retrieval systems under the challenges of interaction and multi-dimensional dynamic relevance. In Proceedings of the 4th CoLIS Conference. Libraries Unlimited, Greenwood Village, CO, 253--270.Google Scholar
- C. Kiili, L. Laurinen, M. Marttunen, and D. J. Leu. 2012. Working on understanding during collaborative online reading. J. Literacy Res. 44, 4 (2012), 448--483.Google ScholarCross Ref
- R. Kopak, L. Freund, and H. L. O’Brien. 2010. Supporting semantic navigation. In Proceedings of the 3rd Symposium on Information Interaction in Context (IIIX’10). ACM, New York, NY, 359--364. Google ScholarDigital Library
- D. R. Krathwohl. 2002. A revision of Bloom's taxonomy: An overview. Theory Pract. 41, 4 (2002), 212--264.Google ScholarCross Ref
- C. Kuhlthau. 1991. Inside the search process. J. Am. Soc. Inf. Sci. 42, 5 (1991), 361--371.Google ScholarCross Ref
- C. Kuhlthau. 1993. Seeking Meaning. Ablex, Norwood, NJ.Google Scholar
- C. Kuhlthau. 1999. The role of experience in the information search process of an early career information worker: Perceptions of uncertainty, complexity, construction, and sources. J. Am. Soc. Inf. Sci. 50, 5 (1999), 399--412. Google ScholarDigital Library
- C. Kuhlthau and S. L. Tama. 2001. Information search process of lawyers: A call for ‘just for me’ information services. J. Doc. 57, 1 (2001), 25--43.Google ScholarCross Ref
- S. Kumpulainen. 2013. Task-Based Information Access in Molecular Medicine: Task Performance, Barriers, and Searching Within a Heterogeneous Information Environment. Ph.D. Thesis, Tampere University Press, Acta Universitatis Tamperensis: 1879.Google Scholar
- S. Kumpulainen. 2014. Trails across the heterogeneous information environment: Manual integration patterns of search systems in molecular medicine. J. Doc. 70, 5 (2014), 856--877. doi: 10.1108/JD-06-2013-0082Google ScholarCross Ref
- S. Kumpulainen and K. Järvelin. 2010. Information interaction in molecular medicine: Integrated use of multiple channels. In Proceedings of the 3rd Symposium on Information Interaction in Context (IIiX’10). ACM, New York, NY, 95--104. Google ScholarDigital Library
- S. Kumpulainen and K. Järvelin. 2012. Barriers to task-based information access in molecular medicine. J. Am. Soc. Inf. Sci. Technol. 63, 1 (2012), 86--97. Google ScholarDigital Library
- Y. Li and N. Belkin. 2008. A faceted approach to conceptualizing tasks in information seeking. Inform. Process. Manage. 44, 6 (2008), 1822--1837. Google ScholarDigital Library
- Y. Li and N. Belkin. 2010. An exploration of the relationships between work task and interactive information search behavior. J. Am. Soc. Inf. Sci. Technol. 61, 9 (2010), 1771--1789. Google ScholarDigital Library
- Y. Li and C. P. Casanave. 2012. Two first-year students’ strategies for writing from sources: Patchwriting or plagiarism? J. Second Lang. Writing 21, 2 (2012), 165--180.Google ScholarCross Ref
- J. Liu and N. Belkin. 2012. Searching vs. writing: Factors affecting information use task performance. Proceedings of the American Society for Information Science and Technology 49, 1 (2012), 1--10.Google ScholarCross Ref
- C. Lucchese, S. Orlando, R. Perego, F. Silvestri, and G. Tolomei. 2011. Identifying task-based sessions in search engine query logs. In Proceedings of the 4th ACM International Conference on Web Search and Data Mining (WSDM’11). ACM, New York, NY, USA, 277--286. Google ScholarDigital Library
- G. Marchionini. 2006. Exploratory search: From finding to understanding. Commun. ACM 49, 4 (2006), 41--46. Google ScholarDigital Library
- M. Markkula and E. Sormunen. 2006. Video needs at the different stages of television program making process. In Proceedings of the 1st International Conference on Information Interaction in Context (IIiX’06). ACM, New York, NY, 111--118. Google ScholarDigital Library
- C. C. Marshall. 2009. Reading and Writing the Electronic Book. Morgan and Claypool, San Rafael, CA. Google ScholarDigital Library
- M. Mateos and I. Solé. 2009. Synthesising information from various texts: A study of procedures and products at different educational levels. Eur. J. Psychol. Educ. 24, 4 (2009), 435--451.Google ScholarCross Ref
- C. Monte-Sano and S. De La Paz. 2012. Using writing tasks to elicit adolescents’ historical reasoning. J. Lit. Res. 44, 3 (2012), 273--299.Google ScholarCross Ref
- A. de Moor and M. Aakhus. 2006. Argumentation support: From technologies to tools. Commun. ACM 49, 3 (2006), 93--98. Google ScholarDigital Library
- N. Moraveji, D. Russell, J. Bien, and D. Mease. 2011. Measuring improvement in user search performance resulting from optimal search tips. In Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’11). ACM, New York, NY, 355--364. Google ScholarDigital Library
- D. C. Neale, J. M. Carroll, and M. B. Rosson. 2004. Evaluating computer-supported cooperative work: Models and frameworks. In Proceedings of the 2004 ACM Conference on Computer Supported Cooperative Work (CSCW’04). ACM, New York, NY, 112--121. Google ScholarDigital Library
- D. W. Oard and J. Kim. 2001. Modeling information content using observable behavior. In Proceedings of the 64th Annual Conference of the American Society for Information Science and Technology (ASIST’01). 481--488.Google Scholar
- J. Pearson, G. Buchanan, and H. Thimbleby. 2013. Designing for Digital Reading. Morgan and Claypool, San Rafael, CA.Google Scholar
- Y. Qu and G. Furnas. 2008. Model-driven evaluation of exploratory search: A study under sensemaking framework. Inf. Process. Manage. 44, 2 (2008), 534--555. Google ScholarDigital Library
- P. J. Rogers, A. Petrosino, T. A. Huebner, and T. A. Hacsi. 2000. Program theory evaluation: Practice, promise, and problems. New Directions Eval. 87 (2000), 5--13.Google ScholarCross Ref
- P. H. Rossi, M. W. Lipsey, and H. Freeman. 2004. Evaluation: A Systematic Approach. Sage, Thousand Oaks, CA.Google Scholar
- J.-F. Rouet. 2006. The Skills of Document Use: From Text Comprehension to Web-Based Learning. Erlbaum, Mahwah, NJ.Google Scholar
- I. Ruthven. 2008. Interactive information retrieval. Annu. Rev. Inform. Sci. 42, 1 (2008), 43--91. Google ScholarDigital Library
- I. Ruthven. 2012. Grieving online: The use of search engines in times of grief and bereavement. In Proceedings of the 4th Information Interaction in Context Symposium (IIIX’12). ACM, New York, NY, 120--129. Google ScholarDigital Library
- M. Saastamoinen, S. Kumpulainen, and K. Järvelin. 2012. Task complexity and information searching in administrative tasks revisited. In Proceedings of the 4th Information Interaction in Context Symposium (IIIX’12). ACM, New York, NY, 204--213. Google ScholarDigital Library
- T. Sakai and Z. Dou. 2013. Summaries, ranked retrieval and sessions: A unified framework for information access evaluation. In Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’13). ACM, New York, NY, 473--482. Google ScholarDigital Library
- M. Sanderson. 2010. Test collection based evaluation of information retrieval systems. Foundations and Trends Inform. Retrieval 4, 4 (2010), 247--375.Google ScholarCross Ref
- T. Saracevic. 1975. Relevance: A review of and a framework for thinking on the notion in information science. J. Am. Soc. Inf. Sci. 26, 6 (1975), 321--343.Google ScholarCross Ref
- T. Saracevic. 1996a. Modeling interaction in information retrieval (IR): A review and proposal. Proceedings of the American Society for Information Science 33 (1996), 3--9.Google Scholar
- T. Saracevic. 1996b. Relevance reconsidered ’96. In Information science: Integration in Perspectives. Proceedings of the 2nd Conference on Conceptions of Library and Information Science. The Royal School of Librarianship, Copenhagen, DK, 201--218.Google Scholar
- G. Schraw. 1998. Promoting general metacognitive awareness. Instruct. Sci. 26, 1--2 (1998), 113--125.Google ScholarCross Ref
- M. Scriven. 1991. Evaluation Thesaurus (4th ed.). Sage, Newbury Park, CA.Google Scholar
- R. Segev-Miller. 2004. Writing from sources: The effect of explicit instruction on college students’ processes and products. Educ. Stud. Lang. Lit. 4, 1 (2004), 5--33.Google ScholarCross Ref
- C. Shah. 2012. Collaborative Information Seeking: The Art and Science of Making the Whole Greater Than the Sum of All. Springer, Heidelberg, Germany. Google ScholarDigital Library
- C. L. Smith and P. B. Kantor. 2008. User adaptation: Good results from poor systems. In Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’08). ACM, New York, NY, 147--154. Google ScholarDigital Library
- E. Sormunen and L. Lehtiö. 2011. Authoring Wikipedia articles as an information literacy assignment -- copy-pasting or expressing new understanding in one's own words? Inf. Res. 16, 4 (2011), Article 503.Google Scholar
- E. Sormunen, J. Heinström, L. Romu, and R. Turunen. 2012. A method for the analysis of information use in source-based writing. Inf. Res. 17, 4 (2012), Article 535.Google Scholar
- E. Sormunen, M. Tanni, T. Alamettälä, and J. Heinström. 2014. Students’ group work strategies in source-based writing assignments. J. Assoc. Inf. Sci. Technol. 65, 6 (2014), 1217--1231.Google ScholarDigital Library
- N. N. Spivey 1997. The Constructivist Metaphor. Reading, Writing, and the Making of Meaning. Academic Press, San Diego, CA.Google Scholar
- L. Suchman. 1987. Plans and Situated Actions: The Problem of Human-Machine Communication. Cambridge University Press, New York. Google ScholarDigital Library
- D. Swanson. 1977. Information retrieval as a trial-and-error process. Libr. Quart. 47, 2 (1977), 128--148.Google ScholarCross Ref
- J. Tague and R. Schultz. 1988. Some measures and procedures for evaluation of the user interface in an information retrieval system. In Proceedings of the 11th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’88). ACM, New York, NY, 371--385. Google ScholarDigital Library
- M. Tanni and E. Sormunen. 2008. A critical review of research on information behavior in assigned learning tasks. J. Doc. 64, 6 (2008), 893--914.Google ScholarCross Ref
- R. Todd. 2006. From information to knowledge: Charting and measuring changes in students’ knowledge of a curriculum topic. Inf. Res. 11, 4 (2006), Article 264.Google Scholar
- A. Turpin and F. Scholer. 2006. User performance versus precision measures for simple search tasks. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’06). ACM, New York, NY, 11--18. Google ScholarDigital Library
- P. Vakkari. 1999. Task complexity, problem structure and information actions: Integrating studies on information seeking and retrieval. Inform. Process. Manage. 35, 6 (1999), 819--837. Google ScholarDigital Library
- P. Vakkari. 2000. Relevance and contributing information types of searched documents in task performance. In Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’00). ACM, New York, NY, 2--9. Google ScholarDigital Library
- P. Vakkari. 2001a. A theory of the task-based information retrieval process: A summary and generalization of a longitudinal study. J. Doc. 57, 1 (2001), 44--60.Google ScholarCross Ref
- P. Vakkari. 2001b. Changes in search tactics and relevance judgments in preparing a research proposal: A summary of findings of a longitudinal study. Inf. Retrieval 4, 3--4 (2001), 295--310. Google ScholarDigital Library
- P. Vakkari. 2003. Task based information searching. Annu. Rev. Inform. Sci. 37, 1 (2003). Information Today, Medford, NJ.Google Scholar
- P. Vakkari. 2010. Exploratory searching as conceptual exploration. In Proceedings of 4th Workshop on Human-Computer Interaction and Information Retrieval. Rutgers University, New Brunswick, NJ, USA, 24--27.Google Scholar
- P. Vakkari and N. Hakala. 2000. Changes in relevance criteria and problem stages in task performance. J. Doc. 56, 5 (2000), 540--562.Google ScholarCross Ref
- P. Vakkari and S. Huuskonen. 2012. Search effort degrades search output but improves task outcome. J. Am. Soc. Inf. Sci. Technol. 63, 4 (2012), 657--670.Google ScholarCross Ref
- P. Vakkari, M. Pennanen, and S. Serola. 2003. Changes of search terms and tactics while writing a research proposal. Inform. Process. Manage. 39, 3 (2003), 445--463. Google ScholarDigital Library
- P. Wang and D. Soergel. 1998. A cognitive model of document use during a research project: Study I: Document selection. J. Am. Soc. Inf. Sci. 49, 2 (1998), 115--133. Google ScholarDigital Library
- B. Wildemuth. 2004. The effects of domain knowledge on search tactic formulation. J. Am. Soc. Inf. Sci. Technol. 55, 3 (2004), 246--258. Google ScholarDigital Library
- B. M. Wildemuth, R. de Bliek, C. P. Friedman, and D. D. File. 1995. Medical students’ personal knowledge, searching proficiency, and database use in problem solving. J. Am. Soc. Inf. Sci. Technol. 46, 8 (1995), 590--607. Google ScholarDigital Library
- I. C. Wu, D. R. Lu, and P. C. Chang. 2008. Towards incorporating a task-stage identification technique into long-term document support process. Inform. Process. Manage. 44, 5 (2008), 1649--1672. Google ScholarDigital Library
- I. Xie. 2008. Interactive Information Retrieval in Digital Environments. IGI, New York, NY. Google ScholarDigital Library
- H. Zaugg, R. E. West, I. Tateishi, and D. L. Randall. 2011. Mendeley: Creating communities of scholarly inquiry through research collaboration. TechTrends 55, 1 (2011), 32--36.Google ScholarCross Ref
Index Terms
- Task-Based Information Interaction Evaluation: The Viewpoint of Program Theory
Recommendations
Human-Computer interaction view on information retrieval evaluation
PROMISE'12: Proceedings of the 2012 international conference on Information Retrieval Meets Information VisualizationThe field of information retrieval (IR) has experienced tremendous growth over the years. Researchers have however identified Human-Computer Interaction (HCI) aspects as important concerns in IR research. Incorporation of HCI techniques in IR can ensure ...
Current Status of the Evaluation of Information Retrieval
This is the second in the series of the articles on an application of the systems analytic approach to evaluation of information retrieval (IR). In the previous article a historical overview of IR was presented and existing terminological problems ...
Studies on Evaluation Index System for Business Information System Based on Evaluation Subject
ICEE '12: Proceedings of the 2012 3rd International Conference on E-Business and E-Government - Volume 05Evaluation of an business information system should be conducted periodically in the whole developing process. At different stages of the evaluation process, evaluation goals and objectives are different. As a result, evaluation subjects should be ...
Comments