Lyndon Kennedy
Mor Naaman
ABSTRACT
The advent of media-sharing sites like Flickr and YouTube has drastically increased the volume of community-contributed multimedia resources available on the web. These collections have a previously unimagined depth and breadth, and have generated new opportunities - and new challenges - to multimedia research. How do we analyze, understand and extract patterns from these new collections? How can we use these unstructured, unrestricted community contributions of media (and annotation) to generate "knowledge".
As a test case, we study Flickr - a popular photo sharing website. Flickr supports photo, time and location metadata, as well as a light-weight annotation model. We extract information from this dataset using two different approaches. First, we employ a location-driven approach to generate aggregate knowledge in the form of "representative tags" for arbitrary areas in the world. Second, we use a tag-driven approach to automatically extract place and event semantics for Flickr tags, based on each tag's metadata patterns.
With the patterns we extract from tags and metadata, vision algorithms can be employed with greater precision. In particular, we demonstrate a location-tag-vision-based approach to retrieving images of geography-related landmarks and features from the Flickr dataset. The results suggest that community-contributed media and annotation can enhance and improve our access to multimedia resources - and our understanding of the world.
- S. Ahern, M. Naaman, R. Nair, and J. Yang. World explorer: Visualizing aggregate data from unstructured text in geo-referenced collections. In Proceedings of the Seventh ACM/IEEE-CS Joint Conference on Digital Libraries. ACM Press, June 2007. Google Scholar
Digital Library
- M. Ames and M. Naaman. Why we tag: Motivations for annotation in mobile and online media. In CHI '07: Proceedings of the SIGCHI conference on Human Factors in computing systems, New York, NY, USA, 2007. ACM Press. Google ScholarDigital Library
- T. L. Berg and D. A. Forsyth. Automatic ranking of iconic images. Technical report, U. C. Berkeley, January 2007.Google Scholar
- C.-C. Chang and C.-J. Lin. LIBSVM: a library for support vector machines, 2001. Software available at http://www.csie.ntu.edu.tw/öcjlin/libsvm. Google ScholarDigital Library
- M. Davis, M. Smith, F. Stentiford, A. Bambidele, J. Canny, N. Good, S. King, and R. Janakiraman. Using context and similarity for face and location identification. In Proceedings of the IS&T/SPIE 18th Annual Symposium on Electronic Imaging Science and Technology, 2006.Google ScholarCross Ref
- M. Dubinko, R. Kumar, J. Magnani, J. Novak, P. Raghavan, and A. Tomkins. Visualizing tags over time. In WWW '06: Proceedings of the 15th international conference on World Wide Web, pages 193--202, New York, NY, USA, 2006. ACM Press. Google ScholarDigital Library
- R. Fergus, P. Perona, and A. Zisserman. A visual category filter for Google images. Proc. ECCV, pages 242--256, 2004.Google ScholarCross Ref
- W. Hsu, L. Kennedy, and S.-F. Chang. Video search reranking via information bottleneck principle. In ACM Multimedia, Santa Babara, CA, USA, 2006. Google ScholarDigital Library
- A. Jaffe, M. Naaman, T. Tassa, and M. Davis. Generating summaries and visualization for large collections of geo-referenced photographs. In MIR '06: Proceedings of the 8th ACM international workshop on Multimedia information retrieval, pages 89--98, New York, NY, USA, 2006. ACM Press. Google ScholarDigital Library
- L. Kennedy and S.-F. Chang. A reranking approach for context-based concept fusion in video indexing and retrieval. In Conference on Image and Video Retrieval, Amsterdam, 2007. Google ScholarDigital Library
- L. Kennedy, S.-F. Chang, and I. Kozintsev. To search or to label?: predicting the performance of search-based automatic image classifiers. Proceedings of the 8th ACM international workshop on Multimedia information retrieval, pages 249--258, 2006. Google ScholarDigital Library
- D. Lowe. Distinctive Image Features from Scale-Invariant Keypoints. International Journal of Computer Vision, 60(2):91--110, 2004. Google ScholarDigital Library
- B. Manjunath and W. Ma. Texture features for browsing and retrieval of image data. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 18(8):837--842, 1996. Google ScholarDigital Library
- M. Naaman, A. Paepcke, and H. Garcia-Molina. From where to what: Metadata sharing for digital photographs with geographic coordinates. In 10th International Conference on Cooperative Information Systems (CoopIS), 2003.Google ScholarCross Ref
- M. Naaman, Y. J. Song, A. Paepcke, and H. Garcia-Molina. Automatic organization for digital photographs with geographic coordinates. In Proceedings of the Fourth ACM/IEEE-CS Joint Conference on Digital Libraries, 2004. Google ScholarDigital Library
- A. Natsev, M. Naphade, and J. Tešić. Learning the semantics of multimedia queries and concepts from a small number of examples. Proceedings of the 13th annual ACM international conference on Multimedia, pages 598--607, 2005. Google ScholarDigital Library
- N. O'Hare, C. Gurrin, G. J. Jones, and A. F. Smeaton. Combination of content analysis and context features for digital photograph retrieval. In 2nd IEE European Workshop on the Integration of Knowledge, Semantic and Digital Media Technologies, 2005.Google ScholarCross Ref
- A. Pigeau and M. Gelgon. Organizing a personal image collection with statistical model-based ICL clustering on spatio-temporal camera phone meta-data. Journal of Visual Communication and Image Representation, 15(3):425--445, September 2004. Google ScholarDigital Library
- T. Rattenbury, N. Good, and M. Naaman. Towards automatic extraction of event and place semantics from flickr tags. In Proceedings of the Thirtieth International ACM SIGIR Conference. ACM Press, July 2007. Google ScholarDigital Library
- R. Sarvas, E. Herrarte, A. Wilhelm, and M. Davis. Metadata creation system for mobile images. In Proceedings of the 2nd international conference on Mobile systems, applications, and services, pages 36--48. ACM Press, 2004. Google ScholarDigital Library
- A. W. M. Smeulders, M. Worring, S. Santini, A. Gupta, and R. Jain. Content-based image retrieval at the end of the early years. IEEE Trans. Pattern Anal. Mach. Intell., 22(12):1349--1380, 2000. Google ScholarDigital Library
- N. Snavely, S. Seitz, and R. Szeliski. Photo tourism: exploring photo collections in 3D. ACM Transactions on Graphics (TOG), 25(3):835--846, 2006. Google ScholarDigital Library
- M. Stricker and M. Orengo. Similarity of color images. Proc. SPIE Storage and Retrieval for Image and Video Databases, 2420:381--392, 1995.Google Scholar
- K. Toyama, R. Logan, and A. Roseway. Geographic location tags on digital images. In Proceedings of the 11th International Conference on Multimedia (MM2003), pages 156--166. ACM Press, 2003. Google ScholarDigital Library
- C.-M. Tsai, A. Qamra, and E. Chang. Extent: Inferring image metadata from context and content. In IEEE International Conference on Multimedia and Expo. IEEE, 2005.Google Scholar
- V. Vapnik. The Nature of Statistical Learning Theory. Springer, 2000. Google ScholarDigital Library
- Y. Wu, E. Y. Chang, and B. L. Tseng. Multimodal metadata fusion using causal strength. In Proceedings of the 13th International Conference on Multimedia (MM2005), pages 872--881, New York, NY, USA, 2005. ACM Press. Google ScholarDigital Library
Index Terms
- How flickr helps us make sense of the world: context and content in community-contributed media collections
Recommendations
Social distance, mobility and place: global and intimate genres in geo-tagged photographs of Guguletu, South Africa
DIS '10: Proceedings of the 8th ACM Conference on Designing Interactive SystemsThis paper documents locative photographic practices on photo-sharing sites Flickr and The Grid and analyses how geo-tagged photographs of the Guguletu in South Africa represent interpersonal meanings and social distance. Distinct communicative genres ...
Generating summaries for large collections of geo-referenced photographs
WWW '06: Proceedings of the 15th international conference on World Wide WebWe describe a framework for automatically selecting a summary set of photographs from a large collection of geo-referenced photos. The summary algorithm is based on spatial patterns in photo sets, but can be expanded to support social, temporal, as well ...
Generating diverse and representative image search results for landmarks
WWW '08: Proceedings of the 17th international conference on World Wide WebCan we leverage the community-contributed collections of rich media on the web to automatically generate representative and diverse views of the world's landmarks? We use a combination of context- and content-based tools to generate representative sets ...
Comments