Chih-Fong Tsai
Skip Abstract Section
Abstract
Many users of image retrieval systems would prefer to express initial queries using keywords. However, manual keyword indexing is very time-consuming. Therefore, a content-based image retrieval system which can automatically assign keywords to images would be very attractive. Unfortunately, it has proved very challenging to build such systems, except where either the image domain is restricted or the keywords relate only to low-level concepts such as color. This article presents a novel image indexing and classification system, called CLAIRE (CLAssifying Images for REtrieval), composed of one image processing module and three modules of support vector machines for color, texture, and high-level concept classification for keyword assignment. The experimental prototype system described here assigns up to five keywords selected from a controlled vocabulary of 60 terms to each image. The system is trained offline by 1639 examples from the Corel stock photo library. For evaluation, five judges reviewed a sample of 800 unknown images to identify which automatically assigned keywords were actually relevant to the image. The system proved to have an 80% probability to assign at least one relevant keyword to an image.
- Aksoy, S. and Haralick, R. M. 2000. Using texture in image similarity and retrieval. In Texture Analysis in Machine Vision, M. Pietikainen and H. Bunke, eds. World Scientific, River Edge, NJ 129--149.Google Scholar
- Antani, S., Kasturi, R., and Jain, R. 2002. A survey on the use of pattern recognition methods for abstraction, indexing and retrieval of images and video. Pattern Recogn. 35, 945--965.Google Scholar
- Armitage, L. and Enser, P. G. B. 1997. Analysis of user need in image archives. J. Inf. Sci. 23, 287--299.Google Scholar
- Barnard, K., Duygulu, P., Forsyth, D., de Freitas, N., Blei, D. M., and Jordan, M. I. 2003. Matching words and pictures. J. Mach. Learn. Res. 3, 1107--1135. Google Scholar
- Benitez, A. B. and Smith, J. R. 2001. New frontiers for intelligent content-based retrieval. In Proceedings of the SPIE Conference on Storage and Retrieval for Media Databases, vol. 4315. San Jose, CA, Jan.Google Scholar
- Black, J. A., jr., Fahmy, G., and Panchanathan, S. 2002. A method for evaluating the performance of content-based image retrieval system based on subjectivity determined similarity between images. In Proceedings of the International Conference on Image and Video Retrieval. London, UK, 356--366. Google Scholar
- Boutell, M., Luo, J., Shen, X., and Brown, C. M. 2004. Learning multi-label scene classification. Pattern Recogn. 37, 1757--1771.Google Scholar
- Burges, C. J. C. 1998. A tutorial on support vector machines for pattern recognition. Data Mining Knowl. Discovery 2, 121--167. Google Scholar
- Byun, H. and Lee, S.-W. 2002. Applications of support vector machines for pattern recognition. In Proceedings of the 1st International Workshop on Pattern Recognition with Support Vector Machines. Niagra Falls, Canada, Aug. 213--236. Google Scholar
- Campbell, N. W. and Thomas, B. T. 1997. Automatic segmentation and classification of outdoor images using neural networks. Int. J. Neural Syst. 8, 137--144.Google Scholar
- Chan, P. and Stolfo, S. J. 1997. On the accuracy of metalearning for scalable data mining. J. Intel. Inf. Syst. 8, 5--28. Google Scholar
- Chang, S.-F., Smith, j.r., Beigi, M., and Benitez, A. 1997. Visual information retrieval from large distributed on-line repositories. Commun. ACM 40, 63--71. Google Scholar
- Ciocca, G., Cusano, C., Schettini, R., and Brambilla, C. 2003. Semantic labeling of digital photos by classification. In Proceedings of the Conference Internet Imaging IV, SPIE 5018, Santa Clara, CA, Jan.Google Scholar
- Clark, A. A., Troscianko, T., Campbell, N. W., and Thomas, B. T. 2000. A comparison between human and machine labelling of image regions. Perception 29, 1127--1138.Google Scholar
- Cortes, C. and Vapnik, V. 1995. Support vector networks. J. Mach. Learn. 20, 273--297. Google Scholar
- Cristianini, N. and Shawe-Taylor, J. 2000. An Introduction to Support Vector Machines and Other Kernel-Based Learning Methods. Cambridge University Press, New York. Google Scholar
- Das, M., Manmatha, R., and Riseman, E. M. 1999. Indexing flower patent images using domain knowledge. IEEE Intell. Syst. 15, 24--33. Google Scholar
- Dietterich, T. G. 1997. Machine learning research: Four current directions. Artif. Intell. 18, 97--136.Google Scholar
- Dumais, S., Platt, J., Heckerman, D., and Sahami, M. 1998. Inductive learning algorithms and representations for text categorization. In Proceedings of the International Conference on Information and Knowledge Management. Bethesda, MD, Nov. 148--155. Google Scholar
- Eakins, J. P. 2002. Towards intelligent image retrieval. Pattern Recogn. 35, 3--14.Google Scholar
- Fidel, R. 1997. The image retrieval task: Implications for the design and evaluation of image databases. New Rev. Hypermedia Multimedia 3, 181--199.Google Scholar
- Flickner, M., Sawhney, H., Niblack, W., Ashley, J., Huang, Q., Dom, B., Gorkani, M., Hafner, J., Lee, D., Petkovic, D., Steele, D., and Yanker, P. 1995. Query by image and video content: The QBIC system. IEEE Comput. 28, 23--32. Google Scholar
- Forsyth, D. A. and Fleck, M. 1997. Body plans. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition. San Juan, PR, June, 678--683. Google Scholar
- Gupta, A. and Jain, R. 1997. Visual information retrieval. Commun. ACM 40, 71--79. Google Scholar
- Hsu, C.-W. and Lin, C.-J. 2002. A comparison of methods for multi-class support vector machines. IEEE Trans. Neural Netw. 12, 1288--1298. Google Scholar
- Iyatomi, H. and Hagiwara, M. 2002. Scenery image recognition and interpretation using fuzzy inference neural networks. Pattern Recogn. 35, 1793--1806.Google Scholar
- Jaimes, A. and Chang, S.-F. 2001. Learning structured visual detectors from user input at multiple levels. Int. J. Images Graphics 1, 415--444.Google Scholar
- Jain, A. K., Duin, R. P. W., and Mao, J. 2000. Statistical pattern recognition: A review. IEEE Trans. Pattern Anal. Mach. Intell. 22, 4--37. Google Scholar
- Joachims, T. 2001. Learning to Classify Text Using Support Vector Machines: Methods, Theory and Algorithms. Kluwer Academic, Hingham, MA. Google Scholar
- Kittler, J., Hatef, M., Duin, R. P. W., and Matas, J. 1998. On combining classifiers. IEEE Trans. Pattern Recog. Mach. Intell. 20, 226--239. Google Scholar
- Kohavi, R. and Foster, P. 1998. Glossary of terms. J. Mach. Learn. 30, 271--274. Google Scholar
- Kuroda, K. and Hagiwara, M. 2002. An image retrieval system by impression words and specific object names---IRIS. Neurocomput. 43, 259--276.Google Scholar
- Laaksonen, J. T., Koskela, J. M., Laakso, S. P., and Oja, E. 2000. PicSOM---Content-based image retrieval with self-organizing maps. Pattern Recog. Lett. 21, 1197--1207. Google Scholar
- Lai, T.-S. 2000. CHROMA: A photographic image retrieval system. PhD. Thesis, University of Sunderland, UK.Google Scholar
- Li, S., Kwok, J. T., Zhu, H., and Wang, Y. 2003. Texture classification using support vector machines. Pattern Recogn. 36, 2883--2893.Google Scholar
- McDonald, S., Lai, T.-S., and Tait, T. 2001. Evaluating a content based image retrieval system. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. New Orleans, LA, Sept., 232--240. Google Scholar
- McDonald, S. and Tait, T. 2003. Search strategies in content-based image retrieval. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. Toronto, Canada, July 28--Aug. 1, 80--87. Google Scholar
- Ojala, T., Pietikanen, M., and Harwood, D. 1996. A comparative study of texture measures with classification based on feature distributions. Pattern Recogn. 29, 51--59.Google Scholar
- Paek, S., Sable, C. L., Hatzivassiloglou, V., Jaimes, A., Schiffman, B. H., Chang, S.-F., and McKeown, K. R. 1999. Integration of visual and text-based approaches for the content labeling and classification of photographs. In Proceedings of the ACM SIGIR Workshop on Multimedia Indexing and Retrieval. Berkeley, CA, Aug.Google Scholar
- Pagano, R. R. 2001. Understanding Statistics in the Behavioral Sciences, 6th ed. Wadsworth/Thomson Learning, CA.Google Scholar
- Park, S. B., Lee, J. W., and Kim, S. K. 2004. Content-based image classification using a neural network. Pattern Recogn. Lett. 25, 287--300. Google Scholar
- Pentland, A., Picard, R. W., and Sclaroff, S. 1996. Photobook: Content-based manipulation of image databases. Int. J. Comput. Vision 18, 233--254. Google Scholar
- Pichler, O., Teuner, A., and Hosticka, B. J. 1996. A comparison of texture feature extraction using adaptive gabor filtering, pyramidal and tree structured wavelet transforms. Pattern Recogn. 29, 733--742. Google Scholar
- Randen, T. and Husoy, J. H. 1999. Filtering for texture classification: A comparative study. IEEE Trans. Pattern Anal. Mach. Intell. 21, 291--310. Google Scholar
- Rasmussen, E. M. 1997. Indexing images. Annual Rev. Inf. Sci. Technol. 32, 169--196.Google Scholar
- Roli, F. and Giacinto, G. 2002. Design of multiple classifier systems. In Hybrid Methods in Pattern Recognition, H. Bunke, and A. Kandel, eds. World Scientific, River Edge, NJ.Google Scholar
- Rui, Y., Huang, T. S., and Chang, S.-F. 1999. Image retrieval current techniques, promising directions and open issues. J. Visual Commun. Image Representation 10, 39--62.Google Scholar
- Schapire, R. E., Freund, Y., Bartlett, P., and Lee, W. S. 1997. Boosting the margin: A new explanation for the effectiveness of voting methods. In Proceedings of the International Conference on Machine Learning. Nashville, TN, July, 322--330. Google Scholar
- Sebe, N. and Lew, M. S. 2000. Wavelet based texture classification. In Proceedings of the IEEE International Conference on Pattern Recognition (ICPR). Barcelona, Sept., 3959--3962. Google Scholar
- Serrano, N., Savakis, A., and Luo, J. 2002. A computationally efficient approach to indoor/outdoor scene classification. In Proceedings of the IEEE International Conference on Pattern Recognition. Quebec, Canada, Aug., 146--149. Google Scholar
- Sharkey, A. J. C. 1997. Combing Artificial Neural Nets: Ensemble and Modular Multi-Net Systems. Springer Verlag, New York. Google Scholar
- Sheikholeslami, G., Chang, W., and Zhang, A. 1998. SemQuery: Semantic clustering and querying on heterogeneous features for visual data. IEEE Trans. Knowl. Data Eng. 14, 988--1002. Google Scholar
- Smith, J. R. and Chang, S.-F. 1996. VisualSEEk: A fully automated content-based image query system. In Proceedings of the ACM International Conference on Multimedia. Boston, Nov., 87--98. Google Scholar
- Smeulders, A. W. M., Worring, M., Santini, S., Gupta, A., and Jain, R. 2000. Content-based image retrieval at the end of the early years. IEEE Trans. Pattern Anal. Mach. Intell. 22, 1349--1380. Google Scholar
- Song, Y. and Zhang, A. 2003. Analyzing scenery images by monotonic tree. ACM Multimedia Syst. 8, 495--511.Google Scholar
- Spink, A. and Jansen, J. 2004. Web Search: Public Searching of the Web. Springer Verlag, New York. Google Scholar
- Szummer, M. and Picard, R. W. 1998. Indoor-outdoor image classification. In Proceedings of the IEEE International Workshop on Content-Based Access of Image and Video Databases. Bombay, Jan., 42--51. Google Scholar
- Tait, J. 2006. Exploratory search: Image retrieval without deep semantics. To appear in Proceedings of the Artificial Intelligence Innovations and Applications. Athens, June.Google Scholar
- Tsai, C.-F. 2003. Stacked generalization: A novel solution to bridge the semantic gap for content-based image retrieval. Online Inf. Rev. 27, 442--445.Google Scholar
- Tsai, C.-F., McGarry, K., and Tait, J. 2003. Image classification using hybrid neural networks. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. Toronto, Canada, July 28--Aug. 1, 431--432. Google Scholar
- Tsai, C.-F., McGarry, K., and Tait, J. 2004. Automatic metadata annotation of images via a two-level learning framework. In Proceedings of the 2nd International Workshop on Semantic Web Conference, in conjunction with the ACM SIGIR '04. Sheffield, UK, July, 32--42.Google Scholar
- Tsai, C.-F., McGarry, K., and Tait, J. 2006. Qualitative evaluation of automatic assignment of keywords to images. Inf. Process. Manage. 42, 136--154. Google Scholar
- Tumer, K. and Ghosh, J. 1996. Error correlation and error reduction in ensemble classifiers. Connection Sci. 8, 385--404.Google Scholar
- Vailaya, A., Figueiredo, M. A. T., Jain, A. K., and Zhang, H.-J. 2001. Image classification for content-based indexing. IEEE Trans. Image Process. 10, 117--130. Google Scholar
- Veltkamp, R. C. and Tanase, M. 2000. Content-based image retrieval systems: A survey. Tech. Rep. UU-CS-2000-34, Utrecht University. Available online from: http://give-lab.cs.uu.nl/cbirsurvey/Google Scholar
- Vogel, J. and Schiele, B. 2001. Performance prediction for vocabulary-supported image retrieval. In Proceedings of the IEEE International Conference on Image Processing. Thessaloniki, Greece, Oct., 753--756.Google Scholar
- Westerveld, T. and de Vries, A.P. 2003. Experimental evaluation of a generative probabilistic image retrieval model on ‘easy’ data. In Proceedings of the Multimedia Information Retrieval Workshop, in conjunction with the 26th Annual ACM SIGIR Conference on Information Retrieval. Toronto, Canada, July 28--Aug. 1. Google Scholar
- Wu, J. K., Kankanhalli, M. S., Lim, H.-H., and Hong, D. 2000. Perspectives on Content-Based Multimedia Systems. Kluwer Academic, Hingham, MA. Google Scholar
- Zhou, X. S. and Huang, T. S. 2002. Unifying keywords and visual contents in image retrieval. IEEE Multimedia 9, 23--33. Google Scholar
- Zhu, X., Liu, W., Zhang, H., and Wu, L. 2001. An image retrieval and semi-automatic annotation scheme for large image databases on the web. In Proceedings of the 13th SPIE Symposium on Electronic Imaging-EI24 Internet Imaging II. 168--177.Google Scholar
Index Terms
- CLAIRE: A modular support vector image indexing and classification system
Recommendations
On Hierarchical Content-Based Image Retrieval by Dynamic Indexing and Guided Search
This paper presents a new approach to content-based image retrieval by using dynamic indexing and guided search in a hierarchical structure, and extending data mining and data warehousing techniques. The proposed algorithms include a wavelet-based ...
Content-based image retrieval by clustering
MIR '03: Proceedings of the 5th ACM SIGMM international workshop on Multimedia information retrievalIn a typical content-based image retrieval (CBIR) system, query results are a set of images sorted by feature similarities with respect to the query. However, images with high feature similarities to the query may be very different from the query in ...
Mammogram retrieval through machine learning within BI-RADS standards
A content-based mammogram retrieval system can support usual comparisons made on images by physicians, answering similarity queries over images stored in the database. The importance of searching for similar mammograms lies in the fact that physicians ...
Comments