Improving Image Retrieval Using Semantic Resources

Summary

Many people use the Internet to find pictures of things. When extraneous images appear in response to simple queries on a search engine, the user has a hard time understanding why his seemingly clear request was not properly satisfied. If the computer could only understand what he wanted better, then maybe the results would be more precise. The introduction of an ontology, though hidden from the user, into current image retrieval engines may provide more accurate image responses to his query. The improvement of the results translates into the possibility of offering structured results, to disambiguate queries and to provide more interactivity options to the user, transforming the current string of character based retrieval into a concept based process. Each one of these aspects is presented and examples are used to support our proposals. We equally discuss the notion of picturability and justify our choice to work exclusively with entities that can be directly represented in a picture. Coordinating the use of a lexical ontology (an OWL representation of WordNet) with image processing techniques, we have developed a system that, given an initial query, automatically returns images associated with the query using automatic reformulation (each concepts is represented by its deepest hyponyms from the ontology). We show that picking randomly from this new set of pictures provides an improved representation for the initial, more general query. We also treat the visual aspects of the images for these deepest hyponyms (the leaves of WordNet). The depictions associated to leaf categories are clustered into coherent sets using low-level image features like color and texture. Some limitations (e.g. the quality and coverage of the semantic structure, the impossibility to answer complex queries) of the ontology based retrieval are equally discussed.