Skip to main content
SpringerLink
Log in

An Overview of Evaluation Campaigns in Multimedia Retrieval

  • Chapter
ImageCLEF

Part of the book series: The Information Retrieval Series ((INRE,volume 32))

Abstract

This chapter presents an academic and research perspective on the impact and importance of ImageCLEF and similar evaluation workshops in multimedia information retrieval (MIR). Three main themes are examined: the position of ImageCLEF compared with other evaluation conferences; general views on the usefulness of evaluation conferences and possible alternatives, and the impact and real–world meaning of evaluation metrics used within ImageCLEF. We examine the value of ImageCLEF, and related evaluation conferences, for the multimedia IR researcher as providing not only a forum for assessing and comparing outcomes but also serving to promote research aims, provide practical guidance (e.g. standard data sets) and inspire research directions.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  • Borlund P (2003) The concept of relevance in IR. Journal of the American Society for information Science and Technology 54(10):913–925

    Article  Google Scholar 

  • Bozzon A, Brambilla M, Fraternali P, Nucci F, Debald S, Moore E, Neidl W, Plu M, Aichroth P, Pihlajamaa O, Laurier C, Zagorac S, Backfried G, Weinland D, Croce V (2009) PHAROS: an audiovisual search platform. In: ACM International conference on research and development in information retrieval. ACM press, p 841

    Google Scholar 

  • Brookes B (1981) Information technology and the science of information. Information retrieval research. London: Butterworths 1–8

    Google Scholar 

  • Buckley C, Voorhees E (2000) Evaluating evaluation measure stability. In: Proceedings of the 23rd annual international ACM SIGIR conference on research and development in information retrieval, p 40. ACM press

    Google Scholar 

  • Clarke C, Kolla M, Cormack G, Vechtomova O, Ashkan A, Büttcher S, MacKinnon I (2008) Novelty and diversity in information retrieval evaluation. In: Proceedings of the 31st annual international ACM SIGIR conference on research and development in information retrieval. ACM press, pp 659–666

    Google Scholar 

  • Cleverdon C, Mills J, Keen E (1966) Factors determining the performance of indexing systems,(Volume 1: Design). Cranfield: College of Aeronautics

    Google Scholar 

  • Cooniss L, Ashford A, Graham M (2000) Information seeking behaviour in image retrieval. VISOR 1 final report. Technical report, Library and Information Commission Research Report, British Library

    Google Scholar 

  • Cooniss L, Davis J, Graham M (2003) A user–oriented evaluation framework for the development of electronic image retrieval systems in the workplace: VISOR 2 final report. Technical report, Library and Information Commission Research Report, British Library

    Google Scholar 

  • Cosijn E, Ingwersen P (2000) Dimensions of relevance. Information Processing & Management 36(4):533–550

    Article  Google Scholar 

  • Downie JS (2008) The music information retrieval evaluation exchange (2005–2007): a window into music information retrieval research. Acoustical Science and Technology 29(4):247–255

    Article  Google Scholar 

  • Ellis D (1996) The dilemma of measurement in information retrieval research. Journal of the American Society for Information Science 47(1):23–36

    Article  Google Scholar 

  • Harman D (2005) The importance of focused evaluations: A Case Study of TREC and DUC. Charting a New Course: Natural Language Processing and Information Retrieval 16:175–194

    Article  Google Scholar 

  • Heesch D, Pickering M, Rüger S, Yavlinsky A (2003) Video retrieval using search and browsing with key frames. In: TREC Video Retrieval Evaluation

    Google Scholar 

  • Heesch D, Howarth P, Magalhães J, May A, Pickering M, Yavlinsky A, Rüger S (2004) Video retrieval using search and browsing. In: TREC Video Retrieval Evaluation

    Google Scholar 

  • Hersh W, Turpin A, Price S, Chan B, Kramer D, Sacherek L, Olson D (2000) Do batch and user evaluations give the same results? In: Proceedings of the 23rd annual international ACM SIGIR conference on research and development in information retrieval, p 24. ACM press

    Google Scholar 

  • Howarth P, Yavlinsky A, Heesch D, Rüger S (2005) Medical image retrieval using texture, locality and colour. In: Proceedings of the Cross Language Evaluation Forum 2004. Lecture Notes in Computer Science (LNCS), vol 3491. Springer, pp 740–749

    Google Scholar 

  • Järvelin K (2009) Explaining user performance in information retrieval: Challenges to IR evaluation. Advances in Information Retrieval Theory 289–296

    Google Scholar 

  • Järvelin K, Kekäläinen J (2002) Cumulated gain–based evaluation of ir techniques. ACM Transactions on Information Systems 20(4):422–446

    Article  Google Scholar 

  • Jesus R, Magalhães J, Yavlinsky A, Rüger S (2005) Imperial college at trecvid. In: TREC Video Retrieval Evaluation, Gaithersburg, MD

    Google Scholar 

  • Llorente A, Zagorac S, Little S, Hu R, Kumar A, Shaik S, Ma X, Rüger S (2008) Semantic video annotation using background knowledge and similarity–based video retrieval. In: TREC Video Retrieval Evaluation (TRECVid, Gaithersburg, MD)

    Google Scholar 

  • Llorente A, Little S, Rüger S (2009) MMIS at ImageCLEF 2009: Non–parametric density estimation algorithms. In: Working notes of CLEF 2009, Corfu, Greece

    Google Scholar 

  • Magalhães J, Overell S, Yavlinsky A, Rüger S (2006) Imperial college at TRECVID. In: TREC Video Retrieval Evaluation, Gaithersburg, MD

    Google Scholar 

  • Mandl T, Gey F, Di Nunzio G, Ferro N, Larson R, Sanderson M, Santos D, Womser-Hacker C, Xie X (2008) GeoCLEF 2007: the CLEF 2007 cross–language geographic information retrieval track overview. In: Advances in Multilingual and Multimodal Information Retrieval. Lecture Notes in Computer Science (LNCS), vol 5152. Springer, pp 745–772

    Google Scholar 

  • Moffat A, Zobel J (2008) Rank–biased precision for measurement of retrieval effectiveness. ACM Transactions on Information Systems 27(1):1–27

    Article  Google Scholar 

  • Nowak S, Lukashevich H, Dunker P, Rüger S (2010) Performance measures for multilabel classification — a case study in the area of image classification. In: ACM SIGMM International Conference on Multimedia Information Retrieval, Philadelphia, Pennsylvania

    Google Scholar 

  • Over P, Smeaton A (eds) (2007) TVS 2007: proceedings of the international workshop on TRECVid video summarization. ACM press

    Google Scholar 

  • Overell S, Magalhães J, Rüger S (2006) Place disambiguation with co–occurrence models. In: Working notes CLEF 2006, Alicante, Spain

    Google Scholar 

  • Overell S, Llorente A, Liu HM, Hu R, Rae A, Zhu J, Song D, Rüger S (2008) MMIS at ImageCLEF 2008: Experiments combining different evidence sources. In: Working notes of CLEF 2008, Aarhus, Denmark

    Google Scholar 

  • Pickering M, Rüger S (2002) Multi–timescale video shot–change detection. In: Text Retrieval Conf, NIST (Trec, Gaithersburg, MD, Nov 2001), NIST Special Publication 500–250, pp 275–278

    Google Scholar 

  • van Rijsbergen CJ (1989) Towards an information logic. In: Proceedings of the 12th annual international ACM SIGIR conference on research and development in information retrieval. ACM press, pp 77–86

    Google Scholar 

  • Rüger S (2010) Multimedia information retrieval. Lecture notes in the series Synthesis Lectures on Information Concepts, Retrieval, and Services. Morgan–Claypool

    Google Scholar 

  • Sakai T (2007) On the reliability of information retrieval metrics based on graded relevance. Information Processing & Management 43(2):531–548

    Article  Google Scholar 

  • Salton G (1992) The state of retrieval system evaluation. Information Processing and Management 28(4):441–449. Special Issue: Evaluation Issues in Information Retrieval

    Article  Google Scholar 

  • Saracevic T (1995) Evaluation of evaluation in information retrieval. In: Proceedings of the 18th annual international ACM SIGIR conference on research and development in information retrieval, pp 138–146. ACM press

    Google Scholar 

  • Saracevic T (1997) Users lost: Reflections on the past, future, and limits of information science. ACM SIGIR Forum 31(2):16–27

    Article  Google Scholar 

  • Smeaton A, Over P, Kraaij W (2006) Evaluation campaigns and TRECVid. In: ACM International Workshop on Multimedia Information Retrieval. ACM press

    Google Scholar 

  • Smeaton A, Over P, Kraaij W (2009) High–level feature detection from video in TRECVid: a 5–year retrospective of achievements. In: Divakaran A (ed) Multimedia Content Analysis: Theory and Applications. Springer, pp 151–174

    Google Scholar 

  • Smeulders A, Worring M, Santini S, Gupta A, Jain R (2000) Content–based image retrieval at the end of the early years. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(12):1349–1380

    Article  Google Scholar 

  • Snoek CGM, Worring M, de Rooij O, van de Sande KEA, Yan R, Hauptmann AG (2008) VideOlympics: Real–time evaluation of multimedia retrieval systems. IEEE MultiMedia 15(1):86–91

    Article  Google Scholar 

  • Soboroff I, Nicholas C, Cahan P (2001) Ranking retrieval systems without relevance judgments. In: Proceedings of the 24th annual international ACM SIGIR conference on research and development in information retrieval, pp 66–73. ACM press

    Google Scholar 

  • Turpin A, Hersh W (2001) Why batch and user evaluations do not give the same results. In: Proceedings of the 24th annual international ACM SIGIR conference on research and development in information retrieval, pp 225–231. ACM press

    Google Scholar 

  • Voorhees EM (2000) Variations in relevance judgments and the measurement of retrieval effectiveness. Information Processing & Management 36(5):697–716

    Article  Google Scholar 

  • Voorhees EM (2002) The philosophy of information retrieval evaluation. In: Revised Papers from the Second Workshop of the Cross–Language Evaluation Forum on Evaluation of Cross–Language Information Retrieval Systems — CLEF 2001. Lecture Notes in Computer Science (LNCS). Springer, London, UK, pp 355–370

    Google Scholar 

  • Westerveld T, van Zwol R (2006) The INEX 2006 multimedia track. In: Comparative Evaluation of XML Information Retrieval Systems, International Workshop of the Initiative for the Evaluation of XML Retrieval. Lecture Notes in Computer Science (LNCS), vol 4518. Springer, pp 331–344

    Google Scholar 

  • Zagorac S, Llorente A, Little S, Liu HM, Rüger S (2009) Automated content based video retrieval. In: TREC Video Retrieval Evaluation (TRECVid, Gaithersburg, MD)

    Google Scholar 

  • Zobel J (1998) How reliable are the results of large–scale information retrieval experiments? In: Proceedings of the 21st annual international ACM SIGIR conference on research and development in information retrieval. ACM press, pp 307–314

    Google Scholar 

  • van Zwol R, Kazai G, Lalmas M (2005) INEX 2005 multimedia track. In: Advances in XML Information Retrieval and Evaluation, International Workshop of the Initiative for the Evaluation of XML Retrieval. Lecture Notes in Computer Science (LNCS), vol 3977. Springer, pp 497–510

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. KMi, The Open University, Walton Hall, MK7 6AA, UK

    Suzanne Little, Ainhoa Llorente & Stefan Rüger

Authors
  1. Suzanne Little
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ainhoa Llorente
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Stefan Rüger
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Suzanne Little .

Editor information

Editors and Affiliations

  1. HES-SO Business Information Systems, TechnoArk 3, Sierre, 3960, Switzerland

    Henning Müller

  2. Dept. Information Studies, University of Sheffield, Portobello Street 211, Sheffield, S1 4DP, United Kingdom

    Paul Clough

  3. , Computer Vision Lab/ETF-C 113.2, ETH Zürich, Zürich, 8092, Switzerland

    Thomas Deselaers

  4. Idiap Research Institute, rue Marconi 19, Martigny, 1920, Switzerland

    Barbara Caputo

Rights and permissions

Reprints and permissions

Copyright information

© 2010 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Little, S., Llorente, A., Rüger, S. (2010). An Overview of Evaluation Campaigns in Multimedia Retrieval. In: Müller, H., Clough, P., Deselaers, T., Caputo, B. (eds) ImageCLEF. The Information Retrieval Series, vol 32. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-15181-1_27

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-15181-1_27

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-15180-4

  • Online ISBN: 978-3-642-15181-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation