Hangyu Lin
Yanwei Fu
ABSTRACT
Sketch has been employed as an effective communication tool to express the abstract and intuitive meaning of object. While content-based sketch recognition has been studied for several decades, the instance-level Sketch Based Image Retrieval (iSBIR) task has attracted significant research attention recently. In many previous iSBIR works -- TripletSN, and DSSA, edge maps were employed as intermediate representations in bridging the cross-domain discrepancy between photos and sketches. However, it is nontrivial to efficiently train and effectively use the edge maps in an iSBIR system. Particularly, we find that such an edge map based iSBIR system has several major limitations. First, the system has to be pre-trained on a significant amount of edge maps, either from large-scale sketch datasets, e.g., TU-Berlin~\citeeitz2012hdhso, or converted from other large-scale image datasets, e.g., ImageNet-1K\citedeng2009imagenet dataset. Second, the performance of such an iSBIR system is very sensitive to the quality of edge maps. Third and empirically, the multi-cropping strategy is essentially very important in improving the performance of previous iSBIR systems. To address these limitations, this paper advocates an end-to-end iSBIR system without using the edge maps. Specifically, we present a Triplet Classification Network (TC-Net) for iSBIR which is composed of two major components: triplet Siamese network, and auxiliary classification loss. Our TC-Net can break the limitations existed in previous works. Extensive experiments on several datasets validate the efficacy of the proposed network and system.
- John Canny. 1986. A computational approach to edge detection. IEEE TPAMI 6 (1986), 679--698.Google Scholar
Digital Library
- Xiaochun Cao, Hua Zhang, Si Liu, Xiaojie Guo, and Liang Lin. 2013. Sym-fish: A symmetry-aware flip invariant sketch histogram shape descriptor. In ICCV .Google Scholar
- Yang Cao, Changhu Wang, Liqing Zhang, and Lei Zhang. 2011. Edgel index for large-scale sketch-based image search. In CVPR .Google Scholar
- De Cheng, Yihong Gong, Sanping Zhou, JinjunWang, and Nanning Zheng. 2016a. Person Re-Identification by Multi-Channel Parts-Based CNN with Improved Triplet Loss Function. In CVPR .Google Scholar
- De Cheng, Yihong Gong, Sanping Zhou, Jinjun Wang, and Nanning Zheng. 2016b. Person re-identification by multi-channel parts-based cnn with improved triplet loss function. In CVPR .Google Scholar
- Jia Deng, Wei Dong, R. Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A large-scale hierarchical image database. In CVPR . https://doi.org/10.1109/CVPR.2009.5206848Google Scholar
- Jiankang Deng, Yuxiang Zhou, and Stefanos Zafeiriou. 2017. Marginal loss for deep face recognition. In CVPR, Faces in-the-wild Workshop/Challenge .Google Scholar
- Mathias Eitz, James Hays, and Marc Alexa. 2012. How Do Humans Sketch Objects? ACM SIGGRAPH , Vol. 31, 4 (2012), 44:1--44:10.Google ScholarDigital Library
- Mathias Eitz, Kristian Hildebrand, Tamy Boubekeur, and Marc Alexa. 2010. An evaluation of descriptors for large-scale image retrieval from sketched feature lines. Computers & Graphics , Vol. 34, 5 (2010), 482--498.Google ScholarDigital Library
- Mathias Eitz, Kristian Hildebrand, Tamy Boubekeur, and Marc Alexa. 2011. Sketch-based image retrieval: Benchmark and bag-of-features descriptors. TVCG .Google Scholar
- Wang F., Kang L., and Li Y. 2015. Sketch-based 3d shape retrieval using convolutional neural networks. In CVPR .Google Scholar
- Yunchao Gong, Qifa Ke, Michael Isard, and Svetlana Lazebnik. 2013. A Multi-View Embedding Space for Modeling Internet Images, Tags, and their Semantics. IJCV (2013).Google Scholar
- Alexander Hermans, Lucas Beyer, and Bastian Leibe. 2017. In defense of the triplet loss for person re-identification. arXiv preprint arXiv:1703.07737 (2017).Google Scholar
- R. Hu and J. Collomosse. 2013. A performance evaluation of gradient field HOG descriptor for sketch based image retrieval. CVIU (2013).Google Scholar
- R. Hu, T. Wang, and J. Collomosse. 2011. A bag-of-regions approach to sketch based image retrieval. In ICIP .Google Scholar
- Gao Huang, Zhuang Liu, Kilian Q Weinberger, and Laurens van der Maaten. 2017. Densely connected convolutional networks. In CVPR .Google Scholar
- J. Huang, R. S. Feris, Q. Chen, and S. Yan. 2015. Cross-domain image retrieval with a dual attribute-aware ranking network. In ICCV .Google Scholar
- Yu-Gang Jiang, Minjun Li, Xi Wang, Wei Liu, and Xian-Sheng Hua. 2018. DeepProduct: Mobile product search with portable deep features. ACM TOMM , Vol. 14, 2 (2018), 50.Google Scholar
- T. Kato, T. Kurita, N. Otsu, and K. Hirata. 1992. A sketch retrieval method for full color image database-query by visual example. In IAPR .Google Scholar
- Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. In NIPS .Google ScholarDigital Library
- Ke Li, Kaiyue Pang, Jifei Song, Yi-Zhe Song, Tao Xiang, Timothy M., Hospedales, and Honggang Zhang. 2018. Universal Sketch Perceptual Grouping. In arxiv .Google Scholar
- Jiawei Liu, Zheng-Jun Zha, QI Tian, Dong Liu, Ting Yao, Qiang Ling, and Tao Mei. 2016. Multi-Scale Triplet CNN for Person Re-Identification. In ACM Multimedia .Google Scholar
- Li Liu, Fumin Shen, Yuming Shen, Xianglong Liu, and Ling Shao. 2017a. Deep sketch hashing: Fast free-hand sketch-based image retrieval. In CVPR .Google Scholar
- Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. 2017b. Sphereface: Deep hypersphere embedding for face recognition. In CVPR .Google Scholar
- E. Mathias, H. Kristian, B. Tamy, and A. Marc. 2011. Sketch-based image retrieval: Benchmark and bag-of-features descriptors. TVCG (2011).Google Scholar
- Umar Riaz Muhammad, Yongxin Yang, Yi-Zhe Song, Tao Xiang, Timothy M Hospedales, et almbox. 2018. Learning Deep Sketch Abstraction. arXiv preprint arXiv:1804.04804 (2018).Google Scholar
- Filip Radenovi´c, Giorgos Tolias, and Ondrej Chum. 2018. Deep Shape Matching. In arxiv .Google Scholar
- Jose M Saavedra, Juan Manuel Barrios, and S Orand. 2015. Sketch based Image Retrieval using Learned KeyShapes (LKS).. In BMVC .Google Scholar
- Patsorn Sangkloy, Nathan Burnell, Cusuh Ham, and James Hays. 2016. The Sketchy Database: Learning to Retrieve Badly Drawn Bunnies. ACM SIGGRAPH (2016).Google ScholarDigital Library
- Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embedding for face recognition and clustering. In CVPR .Google Scholar
- Karen Simonyan and Andrew Zisserman. 2015. VERY DEEP CONVOLUTIONAL NETWORKS FOR LARGE-SCALE IMAGE RECOGNITION. In ICLR .Google Scholar
- Jifei Song, Yu Qian, Yi-Zhe Song, Tao Xiang, and Timothy Hospedales. 2017. Deep spatial-semantic attention for fine-grained sketch-based image retrieval. In ICCV .Google Scholar
- Yaniv Taigman, Ming Yang, Marc'Aurelio Ranzato, and Lior Wolf. 2014. Deepface: Closing the gap to human-level performance in face verification. In CVPR .Google Scholar
- Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Zhifeng Li, Dihong Gong, Jingchao Zhou, and Wei Liu. 2018. CosFace: Large margin cosine loss for deep face recognition. arXiv preprint arXiv:1801.09414 (2018).Google Scholar
- Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. 2016. A discriminative feature learning approach for deep face recognition. In ECCV .Google Scholar
- Holger Winnemöller, Jan Eric Kyprianidis, and Sven C Olsen. 2012. XDoG: an extended difference-of-Gaussians compendium including advanced image stylization. Computers & Graphics , Vol. 36, 6 (2012), 740--753.Google ScholarDigital Library
- Peng Xu, Yongye Huang, Tongtong Yuan, Kaiyue Pang, Yi-Zhe Song, Tao Xiang, Timothy M Hospedales, Zhanyu Ma, and Jun Guo. 2018. Sketchmate: Deep hashing for million-scale human sketch retrieval. In CVPR .Google Scholar
- Peng Xu, Qiyue Yin, Yonggang Qi, Yi-Zhe Song, Zhanyu Ma, Liang Wang, and Jun Guo. 2016. Instance-level coupled subspace learning for fine-grained sketch-based image retrieval. In European Conference on Computer Vision. Springer, 19--34.Google ScholarCross Ref
- Weidong Yin, Yanwei Fu, Yiqiang Ma, Yu-Gang Jiang, Tao Xiang, and Xiangyang Xue. 2017. Learning to Generate and Edit Hairstyles. In ACM Multimedia .Google Scholar
- Qian Yu, Feng Liu, Yi-Zhe Song, Tao Xiang, Timothy M. Hospedales, and Chen Change Loy. 2016. Sketch Me That Shoe. In CVPR .Google Scholar
- Qian Yu, Yongxin Yang, Feng Liu, Yi-Zhe Song, Tao Xiang, and Timothy M. Hospedales. 2017. Sketch-a-Net: a Deep Neural Network that Beats Humans. IJCV (2017).Google Scholar
- Xiao Zhang, Zhiyuan Fang, Yandong Wen, Zhifeng Li, and Yu Qiao. 2017. Range loss for deep face recognition with long-tailed training data. In CVPR .Google Scholar
- Han Zhu, Mingsheng Long, Jianmin Wang, and Yue Cao. 2016. Deep Hashing Network for Efficient Similarity Retrieval.. In AAAI .Google Scholar
Index Terms
- TC-Net for iSBIR: Triplet Classification Network for Instance-level Sketch Based Image Retrieval
Recommendations
A survey on freehand sketch recognition and retrieval
AbstractWith the development of digital devices and pressure sensing equipment, research into freehand sketches from touch-screen interfaces has increased significantly in recent years. As such, we provide the first comprehensive survey of ...
Highlights- Provide a thorough review of recent developments in sketch recognition and retrieval
Multi-Level Region Matching for Fine-Grained Sketch-Based Image Retrieval
MM '22: Proceedings of the 30th ACM International Conference on MultimediaFine-Grained Sketch-Based Image Retrieval (FG-SBIR) is to use free-hand sketches as queries to perform instance-level retrieval in an image gallery. Existing works usually leverage only high-level information and perform matching in a single region. ...
Free2CAD: parsing freehand drawings into CAD commands
CAD modeling, despite being the industry-standard, remains restricted to usage by skilled practitioners due to two key barriers. First, the user must be able to mentally parse a final shape into a valid sequence of supported CAD commands; and second, ...
Comments