Skip to main content
SpringerLink
Log in

Evaluating recommender systems from the user’s perspective: survey of the state of the art

  • Original Paper
  • Published:
User Modeling and User-Adapted Interaction Aims and scope Submit manuscript

Abstract

A recommender system is a Web technology that proactively suggests items of interest to users based on their objective behavior or explicitly stated preferences. Evaluations of recommender systems (RS) have traditionally focused on the performance of algorithms. However, many researchers have recently started investigating system effectiveness and evaluation criteria from users’ perspectives. In this paper, we survey the state of the art of user experience research in RS by examining how researchers have evaluated design methods that augment RS’s ability to help users find the information or product that they truly prefer, interact with ease with the system, and form trust with RS through system transparency, control and privacy preserving mechanisms finally, we examine how these system design features influence users’ adoption of the technology. We summarize existing work concerning three crucial interaction activities between the user and the system: the initial preference elicitation process, the preference refinement process, and the presentation of the system’s recommendation results. Additionally, we will also cover recent evaluation frameworks that measure a recommender system’s overall perceptive qualities and how these qualities influence users’ behavioral intentions. The key results are summarized in a set of design guidelines that can provide useful suggestions to scholars and practitioners concerning the design and development of effective recommender systems. The survey also lays groundwork for researchers to pursue future topics that have not been covered by existing methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Adomavicius G., Tuzhilin A.: Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions. IEEE Trans. Knowl. Data Eng. 17(6), 734–749 (2005)

    Article  Google Scholar 

  • Adomavicius, G., Tuzhilin, A.: Context-aware recommender systems. In: The 2nd ACM Conference on Recommender Systems (RecSys ’08), pp. 335–336. ACM, New York (2008)

  • Basartan, Y.: Amazon versus the shopbot: an experiment about how to improve the shopbots. Unpublished Ph.D. Summer Paper, Carnegie Mellon University, Pittsburgh, PA (2001)

  • Beenen, G., Ling, K., Wang, X., Chang, K., Frankowski, D., Resnick, P., Kraut, R.E.: Using social psychology to motivate contributions to online communities. In: 2004 ACM Conference on Computer Supported Cooperative Work (CSCW ’04), pp. 212–221. ACM, New York (2004)

  • Berger H., Denk M., Dittenbach M., Merkl D., Pesenhofer A.: Quo Vadis Homo Turisticus? Towards a picture-based tourist profiler. Inf. Commun. Technol. Tour. 2, 87–96 (2007)

    Google Scholar 

  • Bollen, D.G.F.M., Knijnenburg, B.P., Willemsen, M.C., Graus, M.P.: Understanding choice overload in recommender systems. In: The 4th ACM Conference on Recommender Systems (RecSys’10), pp. 63–70. ACM, New York (2010)

  • Brodie C., Karat C.M., Karat J.: Creating an E-commerce environment where consumers are willing to share personal information. In: Karat, C., Blom, J.O., Karat, J. (eds) Designing Personalized User Experiences in eCommerce, pp. 185–206. Springer, Netherlands (2004)

    Chapter  Google Scholar 

  • Burke R.: Hybrid recommender systems: survey and experiments. User Model. User Adapt. Interact. 12(4), 331–370 (2002)

    Article  MATH  Google Scholar 

  • Burke R., Hammond K., Young B.: The FindMe approach to assisted browsing. IEEE Expert Intell. Syst. Appl. 12(4), 32–40 (1997)

    Google Scholar 

  • Chen, L., Pu, P.: Preference-based organization interface: aiding user critiques in recommender systems. In: International Conference on User Modeling (UM’07), Corfu, Greece, 25–29 June, pp. 77–86 (2007)

  • Chen L., Pu P.: Interaction design guidelines on critiquing-based recommender systems. User Model. User Adapt. Interact. 19(3), 167–206 (2009)

    Article  Google Scholar 

  • Chen L., Pu P.: Experiments on the preference-based organization interface in recommender systems. ACM Trans. Comput. Hum. Interact. 17(1), 1–33 (2010)

    Google Scholar 

  • Cosley, D., Lam, S.K., Albert, I., Konstan, J.A., Riedl, J.: Is seeing believing? How recommender system interfaces affect users’ opinions. In: SIGCHI Conference on Human Factors in Computing Systems (CHI ’03), pp. 585–592. ACM, New York (2003)

  • Davis F.D.: Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q. 13, 319–339 (1989)

    Article  Google Scholar 

  • Drenner, S., Sen, S., Terveen, L.: Crafting the initial user experience to achieve community goals. In: 2008 ACM Conference on Recommender Systems (RecSys ’08), pp. 187–194. ACM, New York (2008)

  • Dunn, G., Wiersema, J., Ham, J., Aroyo, L.: Evaluating interface variants on personality acquisition for recommender systems. In: Houben, G.-J., McCalla, G., Pianesi, F., Zancanaro, M. (eds.) User Modeling, Adaptation, and Personalization, vol. 5535, pp. 259–270. Springer-Verlag, Berlin (2009)

  • Einhorn H., Hogarth R.: Confidence in judgment: persistence of the illusion of validity. Psychol. Rev. 85, 395–416 (1978)

    Article  Google Scholar 

  • Guttman, R.H.: Merchant differentiation through integrative negotiation in agent-mediated electronic commerce. Master’s Thesis, School of Architecture and Planning, Program in Media Arts and Sciences, Massachusetts Institute of Technology (1998)

  • Haubl G., Trifts V.: Consumer decision making in online shopping environments: the effects of interactive decision aids. Mark. Sci. 19, 4–21 (2000)

    Article  Google Scholar 

  • Herlocker, J.L., Konstan, J.A., Riedl, J.: Explaining collaborative filtering recommendations. In: 2000 ACM Conference on Computer Supported Cooperative Work (CSCW ’00), pp. 241–250. ACM, New York (2000)

  • Herlocker J.L., Konstan J.A., Terveen L.G., Riedl J.: Evaluating collaborative filtering recommender systems. ACM Trans. Inf. Syst. 22(1), 5–53 (2004)

    Article  Google Scholar 

  • Hu, R., Pu, P.: A comparative user study on rating vs. personality quiz based preference elicitation methods. In: The 14th International Conference on Intelligent User Interfaces (IUI ’09), 8–11 February, pp. 367–372. ACM, Sanibel Island (2009a)

  • Hu, R., Pu, P.: Acceptance issues of personality-based recommender systems. In: The 3rd ACM Conference on Recommender Systems (RecSys 2009), 22–25 October, pp. 221–224. ACM, New York (2009b)

  • Hu, R., Pu, P.: A study on user perception of personality-based recommender systems. In: De Bra, P., Kobsa, A., Chin, D. (eds.) User Modeling, Adaptation, and Personalization, LNCS 6075, pp. 291–302. Springer, Heidelberg (2010)

  • Hu, R., Pu, P.: Enhancing recommendation diversity with organization interfaces. In: the 16th International Conference on Intelligent user Interfaces (IUI ’11), pp. 347–350. ACM, New York (2011)

  • Jones, N., Pu, P.: User technology adoption issues in recommender systems. In: Networking and Electronic Commerce Research Conference (NAEC ’07), pp. 379–394 (2007)

  • Karau, S.J., Williams, K.D.: Understanding individual motivation in groups: the collective effort model. In: Turner, M.E. (ed.) Groups at Work: Theory and Research, pp. 113–141. LEA, Mahwah (2001)

  • Kirakowski J.: SUMI: the software usability measurement inventory. Br. J. Educ. Technol. 24(3), 210–214 (1993)

    Article  Google Scholar 

  • Kleinmuntz D.N., Schkade D.A.: Information displays and decision processes. Psychol. Sci. 4, 221–227 (1993)

    Article  Google Scholar 

  • Knijnenburg, B.P., Willemsen, M.C., Hirtbach, S.: Receiving recommendations and providing feedback: the user-experience of a recommender system. In: The 11th International Conference on Electronic Commerce and Web Technologies, pp. 207–216 (2010)

  • Knijnenburg, B.P., Willemsen, M.C., Gantner, Z., Soncu, H., Newell, C.: Explaining the user experience of recommender systems. User Model. User Adapt. Interact. 22 (2012) doi:10.1007/s11257-011-9118-4

  • Kobsa A., Schreck J.: Privacy through pseudonymity in user-adaptive systems. ACM Trans. Internet Technol. 3(2), 149–183 (2003)

    Article  Google Scholar 

  • Lam, S.K., Frankowski, D., Riedl, J.: Do you trust your recommendations? An exploration of security and privacy issues in recommender systems. In: 2006 International Conference on Emerging Trends in Information and Communication Security (ETRICS), Freiburg, Germany, pp. 14–29 (2006)

  • Linden, G., Hanks, S., Lesh, N.: Interactive assessment of user preference models: the automated travel assistant. In: The Sixth International Conference on User Modeling, pp. 67–78. Springer, Chia Laguna (1997)

  • Locke E.A., Latham G.P.: Building a practically useful theory of goal setting and task motivation: a 35 year odyssey. Am. Psychol. 57(9), 705–717 (2002)

    Article  Google Scholar 

  • Mahmood T., Ricci F., Venturini A.: Improving recommendation effectiveness by adapting the dialogue strategy in online travel planning. Int. J. Inf. Technol. Tour. 11(4), 285–302 (2010)

    Google Scholar 

  • McCarthy, K., Reilly, J., McGinty, L., Smyth, B.: Thinking positively—explanatory feedback for conversational recommender systems. In: European Conference on Case-Based Reasoning (ECCBR-04) Explanation Workshop, Madrid, Spain, pp. 115–124 (2004)

  • McCarthy, K., Reilly, J., McGinty, L., Smyth, B.: Experiments in dynamic critiquing. In: The 10th International Conference on Intelligent User Interfaces (IUI ’05), pp. 175–182. ACM, New York (2005)

  • McGinty, L., Smyth, B.: On the role of diversity in conversational recommender systems. In: The Fifth International Conference on Case-Based Reasoning, pp. 276–290. Springer, Berlin (2003)

  • McNee, M.S., Albert, I., Cosley, D., Gopalkrishnan, P., Lam, S.K., Rashid, A.M., Konstan, J.A., Riedl, J.: On the recommending of citations for research papers. In: 2002 ACM Conference on Computer Supported Cooperative Work (CSCW ’02), pp. 116–125. ACM, New York (2002)

  • McNee, S.M., Lam, S.K., Konstan, J.A., Riedal, J.: 2003. Interfaces for eliciting new user preferences in recommender systems. In: User Modeling 2003, pp. 178–187 (2003)

  • McNee, S.M., Riedl, J., Konstan, J.A.: Being accurate is not enough: how accuracy metrics have hurt recommender systems. In: CHI Extended Abstracts, pp. 1097–1101 (2006a)

  • McNee, S.M., Riedl, J., Konstan, J.A.: Making recommendations better: an analytic model for human-recommender interaction. In: CHI Extended Abstracts, pp. 1103–1108 (2006b)

  • McSherry D.: Explanation in recommender systems. Artif. Intell. Rev. 24(2), 179–197 (2005)

    Article  MATH  Google Scholar 

  • Ochi P., Rao S., Takayama L., Nass C.: Predictors of user perceptions of web recommender systems: How the basis for generating experience and search product recommendations affects user responses. Int. J. Hum. Comput. Stud. 68(8), 472–482 (2010)

    Article  Google Scholar 

  • Paramythis A., Weibelzahl S., Masthoff J.: Layered evaluation of interactive adaptive systems: framework and formative methods. User Model. User Adapt. Interact. 20(5), 383–453 (2010)

    Article  Google Scholar 

  • Payne J.W., Bettman J.R., Schkade D.A.: Measuring constructed preference: towards a building code. J. Risk Uncertain. 19(1–3), 243–270 (1999)

    Article  MATH  Google Scholar 

  • Pu, P., Chen, L.: Integrating tradeoff support in product search tools for e-commerce sites. In: The 6th ACM Conference on Electronic Commerce (EC ’05), pp. 269–278. ACM, New York (2005)

  • Pu, P., Chen, L.: Trust building with explanation interfaces. In: The 11th International Conference on Intelligent User Interfaces (IUI ’06), pp. 93–100. ACM, New York (2006)

  • Pu P., Chen L.: Trust-inspiring explanation interfaces for recommender systems. Knowl-Based. Syst. 20(6), 542–556 (2007)

    Article  Google Scholar 

  • Pu, P., Chen, L.: A user-centric evaluation framework of recommender systems. In: Workshop on User-Centric Evaluation of Recommender Systems and Their Interfaces (UCERSTI’10), ACM Conference on Recommender Systems (RecSys’10), Barcelona, Spain, pp. 14–21 (2010)

  • Pu P., Faltings B.: Decision tradeoff using example-critiquing and constraint programming. Constraints Int. J. 9(4), 289–310 (2004)

    Article  Google Scholar 

  • Pu, P., Kumar, P.: Evaluating example-based search tools. In: The 5th ACM Conference on Electronic Commerce (EC ’04), pp. 208–217. ACM, New York (2004)

  • Pu, P., Viappiani, P., Faltings, B.: Increasing user decision accuracy using suggestions. In: Grinter, R., Rodden, T., Aoki, P., Cutrell, E., Jeffries, R., Olson, G. (eds.) The SIGCHI Conference on Human Factors in Computing Systems (CHI ’06), ACM, New York (2006)

  • Pu, P., Zhou, M., Castagnos, S.: Critiquing recommenders for public taste products. In: The Third ACM Conference on Recommender Systems (RecSys ’09), pp. 249–252. ACM, New York (2009)

  • Pu, P., Faltings, B., Chen, L., Zhang, J.Y., Viappiani, P.: Usability guidelines for product recommenders based on example critiquing research. In: Ricci, F., Rokach, L., Shapira, B., Kantor, P.B. (eds.) Recommender Systems Handbook, Chapter 16, pp. 511–546. Springer (2010)

  • Pu, P., Chen, L., Hu, R.: A user-centric evaluation framework for recommender systems. In: The 5th ACM Conference on Recommender Systems (RecSys’11), Chicago, IL, USA, 23–27 October (2011)

  • Rashid, A.M., Albert, I., Cosley, D., Lam, S.K., McNee, S.M., Konstan, J.A., Riedl, J.: Getting to know you: learning new user preferences in recommender systems. In: The 7th International Conference on Intelligent User Interfaces (IUI ’02), pp. 127–134. ACM, New York (2002)

  • Reilly, J., McCarthy, K., McGinty, L., Smyth, B.: Dynamic critiquing. In: Funk, P., González Calero, P.A. (eds.) Advances in Case-Based Reasoning (ECCBR 2004), LNAI 3155, pp. 763–777. Springer, Heidelberg (2004)

  • Resnick P., Varian H.R.: Recommender systems. Commun. ACM 40, 56–58 (1997)

    Article  Google Scholar 

  • Riedl J.: Personalization and privacy. IEEE Internet Comput. 5(6), 29–31 (2001)

    Article  Google Scholar 

  • Sarwar, B., Karypis, G., Konstan, J., Riedl, J.: Analysis of recommendation algorithms for e commerce. In: The 2nd ACM Conference on Electronic Commerce (EC ’00), pp. 158–167. ACM, New York (2000)

  • Sarwar, B., Karypis, G., Konstan, J., Riedl, J. Item-based collaborative filtering recommendation algorithms. In: WWW’01, pp. 285–295 (2001)

  • Shearin, S., Lieberman, H.: Intelligent profiling by example. In: The 6th International Conference on Intelligent User Interfaces (IUI ’01), pp. 145–151. ACM, New York (2001)

  • Simonson I.: Determinants of customers’ responses to customized offers: conceptual framework and research propositions. J. Mark. 69, 32–45 (2005)

    Article  Google Scholar 

  • Sinha, R., Swearingen, K.: Comparing recommendations made by online systems and friends. In: DELOS-NSF Workshop on Personalization and Recommender Systems in Digital Libraries (2001)

  • Sinha, R., Swearingen, K.: The role of transparency in recommender systems. In: CHI Extended Abstracts, pp. 830–831(2002)

  • Smyth, B.: Case-based recommendation. In: Brusilovsky, P., Kobsa, A., Nejdl, W. (eds.) The Adaptive Web: Methods and Strategies of Web Personalization, Lecture Notes in Computer Science, vol. 4321. Springer-Verlag, Berlin (2007)

  • Smyth, B., McClave, P.: Similarity vs. diversity. In: Aha, D.W., Watson, I. (eds.) The 4th International Conference on Case-Based Reasoning: Case-Based Reasoning Research and Development (ICCBR ’01), pp. 347–361. Springer-Verlag, London (2001)

  • Spiekermann S., Parachiv C.: Motivating human-agent interaction: transferring insights from behavioral marketing to interface design. J. Electron. Commer. Res. 2(3), 255–285 (2002)

    Article  MATH  Google Scholar 

  • Spiekermann, S., Grossklags, J., Berendt, B.: E-privacy in 2nd generation E-commerce: privacy preferences versus actual behavior. In: The 3rd ACM Conference on Electronic Commerce, pp. 38–47. ACM, New York (2001)

  • Swearingen, K., Sinha, R.: Interaction design for recommender systems. In: Designing Interactive Systems (DIS’02), London, 25–28 June (2002)

  • Tintarev, N., Masthoff, J.: Effective explanations of recommendations: user-centered design. In: 2007 ACM Conference on Recommender Systems (RecSys ’07), pp. 153–156. ACM, New York (2007a)

  • Tintarev, N., Masthoff, J.: A survey of explanations in recommender systems. In: The 23rd IEEE International Conference on Data Engineering Workshop, pp. 801–810 (2007b)

  • Viappiani, P., Faltings, B., Pu, P.: Evaluating preference-based search tools: a tale of two approaches. In: The Twenty-First National Conference on Artificial Intelligence (AAAI-06), Boston, USA, 16–20 July, pp. 205–210 (2006)

  • Viappiani P., Faltings B., Pu P.: Preference-based search using example-critiquing with suggestions. J. Artif. Intell. Res. 27, 465–503 (2007)

    Google Scholar 

  • Vig, J., Sen, S., Riedl, J.: Tagsplanations: explaining recommendations using tags. In: The 13th International Conference on Intelligent User Interfaces (IUI’09), pp. 47–56. ACM, New York (2009)

  • Williams, M.D., Tou, F.N.: RABBIT: an interface for database access. In: ACM ’82 Conference (ACM ’82), pp. 83–87. ACM, New York (1982)

  • Xiao B., Benbasat I.: Ecommerce product recommendation agents: use, characteristics, and impact. MIS Q. 31(1), 137–209 (2007)

    Google Scholar 

  • Zanker M., Jessenitschnig M.: Case-studies on exploiting explicit customer requirements in recommender systems. User Model. User Adapt. Interact. 19(1–2), 133–166 (2009)

    Article  Google Scholar 

  • Ziegler, C., McNee, S.M., Konstan, J.A., Lausen, G.: Improving recommendation lists through topic diversification. In: The 14th International Conference on World Wide Web, pp. 22–32 (2005)

  • Zukerman I., Albrecht D.W.: Predictive statistical models for user modeling. User Model. User Adapt. Interact. 11(1–2), 5–18 (2001)

    Article  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Human Computer Interaction Group, School of Computer and Communication Sciences, Swiss Federal Institute of Technology (EPFL), 1015, Lausanne, Switzerland

    Pearl Pu & Rong Hu

  2. Department of Computer Science, Hong Kong Baptist University, Kowloon, Hong Kong

    Li Chen

Authors
  1. Pearl Pu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Li Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rong Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Pearl Pu.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Pu, P., Chen, L. & Hu, R. Evaluating recommender systems from the user’s perspective: survey of the state of the art. User Model User-Adap Inter 22, 317–355 (2012). https://doi.org/10.1007/s11257-011-9115-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11257-011-9115-7

Keywords

Search

Navigation