Abstract
Rankings or ratings are popular methods for structuring large information sets in search engines, e-Commerce, e-Learning, etc. But do they produce the right rankings for their users? In this paper, we give an overview of major evaluation approaches for rankings as well as major challenges facing the use and usability of rankings. We point out the importance of an interdisciplinary perspective for a truly user-centric evaluation of rankings. We then focus on two central problems: the multidimensionality of the criteria that influence both users’ and systems’ rankings, and the randomness inherent in users’ preferences. We propose multicriteria decision analysis and the integration of randomness into rankings as solution approaches to these problems. We close with an outlook on new challenges arising for ranking when systems address not only individuals, but also groups.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Altman, A., Tennenholtz, M.: Axiomatic foundations for ranking systems. Journal of Artificial Intelligence Research 31, 473–495 (2008), http://www.jair.org/media/2306/live-2306-3748-jair.pdf (retrieved 2009-06-15)
Bamberg, G., Coenenberg, A.G.: Betriebswirtschaftliche Entscheidungslehre, 11th edn. Vahlen, Munich (2002)
Berendt, B.: Ranking – use and usability. To appear in Bulletin of the Belgian Mathematical Society – Simon Stevin (2009)
Berendt, B., Günther, O., Spiekermann, S.: Privacy in e-commerce: Stated preferences vs. actual behavior. Communications of the ACM 48(4), 101–106 (2005)
Berendt, B., Kralisch, A.: A user-centric approach to identifying best deployment strategies for language tools: The impact of content and access language on web user behaviour and attitudes. Journal of Information Retrieval 12(3), 380–399 (2009)
Berendt, B., Trümper, D.: Semantics-based analysis and navigation of heterogeneous text corpora: The porpoise news and blogs engine. In: Ting, I.-H., Wu, H.-J. (eds.) Web Mining Applications in E-commerce and E-services, pp. 45–64. Springer, Berlin (2009)
Chakrabarti, S.: Mining the Web. Morgan Kaufmann, San Francisco (2003)
Dix, A., Finlay, J., Abowd, G., Beale, R.: Human Computer Interaction. Prentice Hall Europe, Englewood Cliffs (1998)
d’Ocagne, M.: Coordonnées Parallèles et Axiales: Méthode de transformation géométrique et procédé nouveau de calcul graphique déduits de la considération des coordonnées parallèlles. Gauthier-Villars, Paris (1885)
Eyetools, Eyetools research and reports: Eyetools, enquiro, and did-it uncover search’s golden triangle (2008), http://www.eyetools.com/inpage/research_google_eyetracking_heatmap.htm (retrieved 2009-06-15)
Figueira, J., Greco, S., Erhgott, M. (eds.): Multiple Criteria Decision Analysis: State of the art surveys. Springer Science and Business Media, Boston (2005)
Glänzel, W.: Seven myths in bibliometrics about facts and fiction in quantitative science studies. Collnet Journal of Scientometrics and Information Management 2(1), 9–17 (2008), http://www.collnet.de/Berlin-2008/GlanzelWIS2008smb.pdf (retrieved 2009-06-15)
Hearst, M.A.: Design recommendations for hierarchical faceted search interfaces. In: SIGIR 2006 Faceted Search Workshop (2006), http://flamenco.berkeley.edu/papers/faceted-workshop06.pdf (retrieved 2009-06-15)
Herlocker, J.L., Konstan, J.A., Terveen, L.G., Riedl, J.T.: Evaluating collaborative filtering recommender systems. ACM Trans. Inf. Syst. 22(1), 5–53 (2004)
Hu, J., Zeng, H.-J., Li, H., Niu, C., Chen, Z.: Demographic prediction based on user’s browsing behavior. In: WWW 2007: Proceedings of the 16th international conference on World Wide Web, pp. 151–160. ACM, New York (2007)
Inselberg, A.: The plane with parallel coordinates. Visual Computer 1(4), 69–91 (1985)
International Organization for Standardization, ISO 9241-400:2007. ergonomics of human–system interaction – part 400: Principles and requirements for physical input devices (2007), http://www.iso.org/iso/iso_catalogue/catalogue_tc/catalogue_detail.htm?csnumber=38896 (retrieved 2009-06-15)
International Organization for Standardization, ISO 9241-151:2008. ergonomics of human-system interaction – part 151: Guidance on world wide web user interfaces (2008), http://www.iso.org/iso/iso_catalogue/catalogue_tc/catalogue_detail.htm?csnumber=37031 (retrieved 2009-06-15)
Jameson, A.: Adaptive interfaces and agents. In: Jacko, J.A., Sears, A. (eds.) Human-Computer Interaction Handbook, pp. 305–330. Erlbaum, Mahwah (2003), http://dfki.de/~jameson/abs/Jameson03Handbook.html (retrieved 2009-06-15)
Jameson, A., Smyth, B.: Recommendation to groups. In: Brusilovsky, P., Kobsa, A., Nejdl, W. (eds.) Adaptive Web 2007. LNCS, vol. 4321, pp. 596–627. Springer, Heidelberg (2007)
Jansen, B.J., Zhang, M., Zhang, Y.: Brand awareness and the evaluation of search results. In: WWW 2007: Proceedings of the 16th international conference on World Wide Web, pp. 1139–1140. ACM, New York (2007)
Kaplan, R.S., Norton, D.P.: The Balanced Scorecard. Translating Strategy Into Action. Harvard Business School Press (1996)
Kaplan, R.S., Norton, D.P.: Strategy Maps: Converting Intangible Assets Into Tangible Outcomes. Harvard Business School Press (2004)
Keeney, R., Raiffa, H.: Decisions with Multiple Objectives; Preferences and Value Tradeoffs. John Wiley & Sons, Chichester (1976)
Kobsa, A., Teltzrow, M.: Impacts of contextualized communication of privacy practices and personalization benefits on purchase behavior and perceived quality of recommendation. In: Beyond Personalization 2005: A Workshop on the Next Stage of Recommender Systems Research (IUI 2005), San Diego, CA, pp. 48–53 (2005)
Kralisch, A., Berendt, B.: Language-sensitive search behaviour and the role of domain knowledge. The New Review of Hypermedia and Multimedia 11(2), 221–246 (2005)
Kralisch, A., Köppen, V.: The impact of language on website use and user satisfaction: Project description. In: Proceedings of the 13th European Conference on Information Systems, Information Systems in a Rapidly Changing Economy, ECIS 2005 (2005)
Langville, A.N., Meyer, C.D.: Google’s PageRank and Beyond: The Science of Search Engine Rankings. Princeton University Press, Princeton (2006)
Lewandowski, D., Höchstötter, N.: Web searching: A quality measurement perspective. In: Spink, A., Zimmer, M. (eds.) Web Searching: Interdisciplinary Perspectives. Springer, Dordrecht (2007)
Liu, H., Mihalcea, R.: Of men, women, and computers: Data-driven gender modeling for improved user interfaces. In: Proceedings of the International Conference on Weblogs Social Media (ICWSM), pp. 121–128 (2007)
Marchant, T.: An axiomatic characterization of the ranking based on the h-index and some other bibliometric rankings of authors. Scientometrics (2009), http://www.springerlink.com/content/e71h95u774701j1k (retrieved 2009-06-15)
Mozilla Labs, Introducing geode (2008), http://labs.mozilla.com/2008/10/introducing-geode/ (retrieved 2009-06-15)
Murakami, T., Mori, K., Orihara, R.: Metrics for evaluating the serendipity of recommendation lists. In: Satoh, K., Inokuchi, A., Nagao, K., Kawamura, T. (eds.) JSAI 2007. LNCS (LNAI), vol. 4914, pp. 40–46. Springer, Heidelberg (2008)
Nielsen, J.: Eyetracking research (2006), http://www.useit.com/eyetracking (retrieved 2009-06-15)
Pan, B., Hembrooke, H., Joachims, T., Lorigo, L., Gay, G., Granka, L.: In: google we trust: Users’ decisions on rank, position, and relevance. Journal of Computer-Mediated Communication 12(3), 801–823 (2007)
Radlinski, F., Joachims, T.: Query chains: learning to rank from implicit feedback. In: Grossman, R., Bayardo, R.J., Bennett, K.P. (eds.) Proceedings of the Eleventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 239–248. ACM, New York (2005)
Robertson, S., Zaragoza, H.: On rank-based effectiveness measures and optimization. Inf. Retr. 10(3), 321–339 (2007)
Rousseau, R.: Woeginger’s axiomatisation of the h-index and its relation to the g-index, the h(2)-index and the R2-index. Journal of Informetrics 2(4), 335–340 (2008)
Ruthven, I., Lalmas, M.: A survey on the use of relevance feedback for information access systems. Knowl. Eng. Rev. 18(2), 95–145 (2003)
Saaty, T.: The Analytic Hierarchy Process for Decisions in a Complex World. McGraw-Hill, New York (1980)
Schneeweiß, C.: Planung 1, Systemanalytische und entscheidungstheoretische Grundlagen. Springer, Berlin (1991)
Sørensen, P., Lerche, D., Thomsen, M.: Developing decision support based on field data and partial order theory, pp. 259–283. Springer, Berlin (2006)
Tullis, T., Albert, W.: Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics. Morgan Kaufmann, San Francisco (2008)
Tversky, A., Kahneman, D.: The framing of decisions and the psychology of choice. Science 211, 453–458 (1981)
Varian, H.R.: Intermediate Microeconomics: A Modern Approach, 7th edn. W W Norton & Co. (2007)
Vaughan, L.: New measurements for search engine evaluation proposed and tested. Information Processing & Management 40(4), 677–691 (2004)
Zhang, M., Hurley, N.: Statistical modeling of diversity in top-n recommender systems. In: To appear in Proc. of the ACM Web Intelligence Conference WI 2009 (2009)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer-Verlag Berlin Heidelberg
About this chapter
Cite this chapter
Berendt, B., Köppen, V. (2010). Improving Ranking by Respecting the Multidimensionality and Uncertainty of User Preferences. In: Armano, G., de Gemmis, M., Semeraro, G., Vargiu, E. (eds) Intelligent Information Access. Studies in Computational Intelligence, vol 301. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-14000-6_3
Download citation
DOI: https://doi.org/10.1007/978-3-642-14000-6_3
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-13999-4
Online ISBN: 978-3-642-14000-6
eBook Packages: EngineeringEngineering (R0)