Abstract
The Microsoft SenseCam is a small lightweight wearable camera used to passively capture photos and other sensor readings from a user’s day-to-day activities. It can capture up to 3,000 images per day, equating to almost 1 million images per year. It is used to aid memory by creating a personal multimedia lifelog, or visual recording of the wearer’s life. However the sheer volume of image data captured within a visual lifelog creates a number of challenges, particularly for locating relevant content. Within this work, we explore the applicability of semantic concept detection, a method often used within video retrieval, on the novel domain of visual lifelogs. A concept detector models the correspondence between low-level visual features and high-level semantic concepts (such as indoors, outdoors, people, buildings, etc.) using supervised machine learning. By doing so it determines the probability of a concept’s presence. We apply detection of 27 everyday semantic concepts on a lifelog collection composed of 257,518 SenseCam images from 5 users. The results were then evaluated on a subset of 95,907 images, to determine the precision for detection of each semantic concept and to draw some interesting inferences on the lifestyles of those 5 users. We additionally present future applications of concept detection within the domain of lifelogging.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Bell, G., Gemmell, J.: A Digital Life. Scientific American (2007)
Bovik, A.C., Clark, M., Geisler, W.S.: Multichannel texture analysis using localized spatial filters. IEEE Transactions on Pattern Analysis and Machine Intelligence 12(1), 55–73 (1990)
Bush, V.: As We May Think. Atlantic Monthly 176(1), 101–108 (1945)
Byrne, D., Lavelle, B., Doherty, A.R., Jones, G.J.F., Smeaton, A.F.: Using Bluetooth and GPS Metadata to Measure Event Similarity in SenseCam Images. In: Proc. of IMAI 2007 (July 2007)
Chang, C.C., Lin, C.J.: LIBSVM: a library for support vector machines (2001), http://www.csie.ntu.edu.tw/~cjlin/libsvm/
DeVaul, R.W., Dunn, S.: Real-Time Motion Classification for Wearable Computing Applications, Project paper (2001), http://www.media.mit.edu/wearables/mithril/realtime.pdf
Doherty, A.R., Byrne, D., Smeaton, A.F., Jones, G.J.F., Hughes, M.: Investigating Keyframe Selection Methods in the Novel Domain of Passively Captured Visual Lifelogs. In: Proc. of the ACM CIVR 2008, Niagara Falls, Canada (2008)
Doherty, A.R., Smeaton, A.F.: Automatically Segmenting Lifelog Data Into Events. In: Proc. 9th International Workshop on Image Analysis for Multimedia Interactive Services (2008)
Fleiss, J.L.: Measuring nominal scale agreement among many raters. Psychological Bulletin 76(5), 378–382 (1971)
Fuller, M., Kelly, L., Jones, G.J.F.: Applying Contextual Memory Cues for Retrieval from Personal Information Archives. In: Proceedings of PIM 2008 Workshop Florence, Italy (2008)
Geusebroek, J.M.: Compact object descriptors from local colour invariant histograms. In: British Machine Vision Conference, Edinburgh, UK (2006)
Geusebroek, J., Smeulders, A.W.M.: A six-stimulus theory for stochastic texture. International Journal of Computer Vision 62(1/2), 7–16 (2005)
Gurrin, C., Smeaton, A.F., Byrne, D., O’Hare, N., Jones, G.J., O’Connor, N.: An examination of a large visual lifelog. In: Li, H., Liu, T., Ma, W.-Y., Sakai, T., Wong, K.-F., Zhou, G. (eds.) AIRS 2008. LNCS, vol. 4993, pp. 537–542. Springer, Heidelberg (2008)
Hoang, M., Geusebroek, J., Smeulders, A.W.M.: Color texture measurement and segmentation. Signal Processing 85(2), 265–275 (2005)
Hodges, S., Williams, L., Berry, E., Izadi, S., Srinivasan, J., Butler, A., Smyth, G., Kapur, N., Wood, K.: SenseCam: A Retrospective Memory Aid. In: 8th International Conference on Ubiquitous Computing, Orange County, USA, pp. 177–193 (2006)
Jiang, Y.G., Ngo, C.W., Yang, J.: Towards Optimal bag-of-features for Object Categorization and Semantic Video Retrieval. In: Proceedings of the ACM International Conference on Image and Video Retrieval, Amsterdam, The Netherlands, pp. 494–501 (2007)
Jurie, F., Triggs, B.: Creating efficient codebooks for visual recognition. In: IEEE International Conference on Computer Vision, Beijing, China, pp. 604–610 (2005)
Kapur, J.N., Sahoo, P.K., Wong, A.K.C.: A New Method for Graylevel Picture Thresholding using the Entropy of the Histogram. Comp. Vis., Grap., & Image Proc. (1985)
Landis, J.R., Koch, G.G.: The measurement of observer agreement for categorical data. Biometrics 33, 159–174 (1977)
Lee, H., Smeaton, A.F., O’Connor, N.E., Jones, G.F.J.: Adaptive Visual Summary of LifeLog Photos for Personal Information Management. In: Proc. 1st Intnl. Workshop on Adaptive Infor. Retrieval, pp. 22–23 (2006)
Lin, H.T., Lin, C.J., Weng, R.: A note on Platt’s probabilistic outputs for support vector machines. Machine Learning 68(3), 267–276 (2007)
Naphade, M.R., Kennedy, L., Kender, J.R., Chang, S., Smith, J.R., Over, P., Hauptmann, A.: A Light Scale Concept Ontology for Multimedia Understanding for TRECVID 2005. Technical Report RC23612, IBM T.J. Watson Research Center (2005)
Snoek, C.G.M., Worring, M., van Gemert, J.C., Geusebroek, J.M., Smeulders, A.W.M.: The Challenge Problem for Automated Detection of 101 Semantic Concepts in Multimedia. In: ACM Multimedia 2006, Santa Barbara, USA, pp. 421–430 (2006)
Snoek, C.G.M., van Gemert, J.C., Gevers, T., Huurnink, B., Koelma, D.C., van Liempt, M., de Rooij, O., van de Sande, K.E.A., Seinstra, F.J., Smeulders, A.W.M., Thean, A.H.C., Veenman, C.J., Worring, M.: The MediaMill TRECVID 2006 semantic video search engine. In: Proceedings of the 4th TRECVIDWorkshop, Gaithersburg, USA (2006)
van Gemert, J.C., Snoek, C.G.M., Veenman, C.J., Smeulders, A.W.M., Geusebroek, J.M.: Comparing compact codebooks for visual categorization. Computer Vision and Image Understanding (submitted, 2008)
Vapnik, V.: The Nature of Statistical Learning Theory, 2nd edn. Springer, New York (2000)
Wang, D., Liu, X., Luo, L., Li, J., Zhang, B.: Video Diver: generic video indexing with diverse features. In: Proceedings of the ACM SIGMM International Workshop on Multimedia Information Retrieval, Augsburg, Germany, pp. 61–70 (2007)
Yanagawa, A., Chang, S.F., Kennedy, L., Hsu, W.: Columbia university’s baseline detectors for 374 LSCOM semantic visual concepts. Technical Report 222-2006-8, Columbia University ADVENT Technical Report (2007)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2008 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Byrne, D., Doherty, A.R., Snoek, C.G.M., Jones, G.G.F., Smeaton, A.F. (2008). Validating the Detection of Everyday Concepts in Visual Lifelogs. In: Duke, D., Hardman, L., Hauptmann, A., Paulus, D., Staab, S. (eds) Semantic Multimedia. SAMT 2008. Lecture Notes in Computer Science, vol 5392. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-92235-3_4
Download citation
DOI: https://doi.org/10.1007/978-3-540-92235-3_4
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-92234-6
Online ISBN: 978-3-540-92235-3
eBook Packages: Computer ScienceComputer Science (R0)