Hanwang Zhang
Shuicheng Yan
ABSTRACT
This paper presents a novel Attribute-augmented Semantic Hierarchy (A2 SH) and demonstrates its effectiveness in bridging both the semantic and intention gaps in Content-based Image Retrieval (CBIR). A2 SH organizes the semantic concepts into multiple semantic levels and augments each concept with a set of related attributes, which describe the multiple facets of the concept and act as the intermediate bridge connecting the concept and low-level visual content. A hierarchical semantic similarity function is learnt to characterize the semantic similarities among images for retrieval. To better capture user search intent, a hybrid feedback mechanism is developed, which collects hybrid feedbacks on attributes and images. These feedbacks are then used to refine the search results based on A2 SH. We develop a content-based image retrieval system based on the proposed A2 SH. We conduct extensive experiments on a large-scale data set of over one million Web images. Experimental results show that the proposed A2 SH can characterize the semantic affinities among images accurately and can shape user search intent precisely and quickly, leading to more accurate search results as compared to state-of-the-art CBIR solutions.
- M. Crucianu, M. Ferecatu, and N. Boujemaa. Relevance feedback for image retrieval: a short survey. DELOS2 Report, 2004.Google Scholar
- R. Datta, D. Joshi, J. Li, and J. Wang. Image retrieval: Ideas, influences, and trends of the new age. ACM Computing Surveys, 2008. Google ScholarDigital Library
- J. Deng, A. C. Berg, and L. Fei-Fei. Hierarchical semantic indexing for large scale image retrieval. In CVPR, 2011.Google ScholarDigital Library
- J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.Google ScholarCross Ref
- T. Deselaers and V. Ferrari. Visual and semantic similarity in imagenet. In CVPR, 2011. Google ScholarDigital Library
- M. Douze, A. Ramisa, and C. Schmid. Combining attributes and fisher vectors for efficient image retrieval. In CVPR, 2011. Google ScholarDigital Library
- J. Fan, Y. Gao, and H. Luo. Integrating concept ontology and multitask learning to achieve more effective classifier training for multilevel image annotation. TIP, 2008. Google ScholarDigital Library
- A. Farhadi, I. Endres, D. Hoiem, and D. Forsyth. Describing objects by their attributes. In CVPR, 2009.Google ScholarCross Ref
- X. Felix, R. Ji, M. Tsai, G. Ye, and S. Chang. Weak attributes for large-scale image retrieval. In CVPR, 2012.Google Scholar
- C. Fellbaum. Wordnet. Theory and Applications of Ontology: Computer Applications, 2010. Google ScholarDigital Library
- A. Hanjalic, C. Kofler, and M. Larson. Intent and its discontents: the user at the wheel of the online video search engine. In MM, 2012. Google ScholarDigital Library
- A. Jaimes and S. fu Chang. A conceptual framework for indexing visual information at multiple levels. In SPIE Internet Imaging, 2000.Google Scholar
- A. Kovashka, D. Parikh, and K. Grauman. Whittlesearch: Image search with relative attribute feedback. In CVPR, 2012. Google ScholarDigital Library
- T. Leung and J. Malik. Representing and recognizing the visual appearance of materials using three-dimensional textons. IJCV, 2001. Google ScholarDigital Library
- M. Lew, N. Sebe, C. Djeraba, and R. Jain. Content-based multimedia information retrieval: State of the art and challenges. TOMCCAP, 2006. Google ScholarDigital Library
- D. G. Lowe. Distinctive image features from scale-invariant keypoints. IJCV, 2004. Google ScholarDigital Library
- Z. Ma, Y. Yang, Z. Xu, S. Yan, N. Sebe, and A. G. Hauptmann. Complex event detection via multi-source video attributes. In CVPR, 2012. Google ScholarDigital Library
- M. Marszalek and C. Schmid. Semantic hierarchies for visual object recognition. In CVPR, 2007.Google ScholarCross Ref
- F. Monay and D. Gatica-Perez. On image auto-annotation with latent space models. In MM, 2003. Google ScholarDigital Library
- M. Naphade, J. R. Smith, J. Tesic, S.-F. Chang, W. Hsu, L. Kennedy, A. Hauptmann, and J. Curtis. Large-scale concept ontology for multimedia. Multimedia, IEEE, 2006. Google ScholarDigital Library
- P. Over, G. Awad, M. Michel, J. Fiscus, G. Sanders, B. Shaw, W. Kraaij, A. F. Smeaton, and G. Quéenot. Trecvid 2012 -- an overview of the goals, tasks, data, evaluation mechanisms and metrics. In TRECVID, 2012.Google Scholar
- D. Parikh and K. Grauman. Interactively building a discriminative vocabulary of nameable attributes. In CVPR, 2011. Google ScholarDigital Library
- Y. Rui, T. S. Huang, and S.-F. Chang. Image retrieval: Current techniques, promising directions, and open issues. JVCIR, 1999.Google ScholarDigital Library
- Y. Rui, T. S. Huang, M. Ortega, and S. Mehrotra. Relevance feedback: a power tool for interactive content-based image retrieval. TCSVT, 1998. Google ScholarDigital Library
- O. Russakovsky and L. Fei-Fei. Attribute learning in large-scale datasets. In ECCV, 2010. Google ScholarDigital Library
- W. J. Scheirer, N. Kumar, P. N. Belhumeur, and T. E. Boult. Multi-attribute spaces: Calibration for attribute fusion and similarity search. In CVPR, 2012.Google ScholarCross Ref
- N. Sebe, M. S. Lew, X. Zhou, T. S. Huang, and E. M. Bakker. The state of the art in image and video retrieval. In Image and Video Retrieval. 2003. Google ScholarDigital Library
- A. Smeulders, M. Worring, S. Santini, A. Gupta, and R. Jain. Content-based image retrieval at the end of the early years. TPAMI, 2000. Google ScholarDigital Library
- J. R. Smith and S.-F. Chang. Visualseek: a fully automated content-based image query system. In MM, 1997. Google ScholarDigital Library
- C. G. Snoek, B. Huurnink, L. Hollink, M. De Rijke, G. Schreiber, and M. Worring. Adding semantics to detectors for video retrieval. TMM, 2007. Google ScholarDigital Library
- C. G. Snoek and M. Worring. Concept-based video retrieval. FTIR, 2008. Google ScholarDigital Library
- Y. Song, M. Zhao, J. Yagnik, and X. Wu. Taxonomic classification for web-based videos. In CVPR, 2010.Google ScholarCross Ref
- D. Tao, X. Tang, X. Li, and X. Wu. Asymmetric bagging and random subspace for support vector machines-based relevance feedback in image retrieval. TPAMI, 2006. Google ScholarDigital Library
- S. Tong and E. Chang. Support vector machine active learning for image retrieval. In MM, 2001. Google ScholarDigital Library
- N. Verma, D. Mahajan, S. Sellamanickam, and V. Nair. Learning hierarchical similarity metrics. In CVPR, 2012.Google ScholarCross Ref
- J. Wang, J. Yang, K. Yu, F. Lv, T. Huang, and Y. Gong. Locality-constrained linear coding for image classification. In CVPR, 2010.Google ScholarCross Ref
- K. Q. Weinberger, J. Blitzer, and L. K. Saul. Distance metric learning for large margin nearest neighbor classification. In NIPS, 2006.Google ScholarDigital Library
- Z.-J. Zha, L. Yang, T. Mei, M. Wang, and Z. Wang. Visual query suggestion. In MM, 2009. Google ScholarDigital Library
- H. Zhang, Z.-J. Zha, S. Yan, J. Bian, and T.-S. Chua. Attribute feedback. In MM, 2012. Google ScholarDigital Library
- K. Zhang, I. W. Tsang, and J. T. Kwok. Maximum margin clustering made practical. TNN, 2009. Google ScholarDigital Library
Index Terms
- Attribute-augmented semantic hierarchy: towards bridging semantic gap and intention gap in image retrieval
Recommendations
Attribute-Augmented Semantic Hierarchy: Towards a Unified Framework for Content-Based Image Retrieval
Special Issue on Multiple Sensorial (MulSeMedia) Multimodal Media : Advances and ApplicationsThis article presents a novel attribute-augmented semantic hierarchy (A2SH) and demonstrates its effectiveness in bridging both the semantic and intention gaps in content-based image retrieval (CBIR). A2SH organizes semantic concepts into multiple ...
Semantic feedback for interactive image retrieval
MULTIMEDIA '05: Proceedings of the 13th annual ACM international conference on MultimediaIn this paper we present a semantic image retrieval system with integrated feedback mechanism. In our system, we propose a novel feedback solution for semantic retrieval: semantic feedback, which allows our system to interact with users directly at the ...
Augmented Image Retrieval using Multi-order Object Layout with Attributes
MM '14: Proceedings of the 22nd ACM international conference on MultimediaIn image retrieval, users' search intention is usually specified by textual queries, exemplar images, concept maps, and even sketches, which can only express the search intention partially. These query strategies lack the abilities to indicate the ...
Comments