Abstract
Temporal coherence is crucial in content-aware video retargeting. To date, this problem has been addressed by constraining temporally adjacent pixels to be transformed coherently. However, due to the motion-oblivious nature of this simple constraint, the retargeted videos often exhibit flickering or waving artifacts, especially when significant camera or object motions are involved. Since the feature correspondence across frames varies spatially with both camera and object motion, motion-aware treatment of features is required for video resizing. This motivated us to align consecutive frames by estimating interframe camera motion and to constrain relative positions in the aligned frames. To preserve object motion, we detect distinct moving areas of objects across multiple frames and constrain each of them to be resized consistently. We build a complete video resizing framework by incorporating our motion-aware constraints with an adaptation of the scale-and-stretch optimization recently proposed by Wang and colleagues. Our streaming implementation of the framework allows efficient resizing of long video sequences with low memory cost. Experiments demonstrate that our method produces spatiotemporally coherent retargeting results even for challenging examples with complex camera and object motion, which are difficult to handle with previous techniques.
Supplemental Material
Available for Download
- Avidan, S., and Shamir, A. 2007. Seam carving for content-aware image resizing. ACM Trans. Graph. 26, 3, 10. Google ScholarDigital Library
- Chen, L. Q., Xie, X., Fan, X., Ma, W. Y., Zhang, H. J., and Zhou, H. Q. 2003. A visual attention model for adapting images on small displays. ACM Multimedia Systems Journal 9, 4, 353--364.Google ScholarDigital Library
- Chen, B.-Y., Lee, K.-Y., Huang, W.-T., and Lin, J.-S. 2008. Capturing intention-based full-frame video stabilization. Computer Graphics Forum 27, 7, 1805--1814.Google ScholarCross Ref
- Cho, T. S., Butman, M., Avidan, S., and Freeman, W. T. 2008. The patch transform and its applications to image editing. In CVPR '08.Google Scholar
- Deselaers, T., Dreuw, P., and Ney, H. 2008. Pan, zoom, scan - time-coherent, trained automatic video cropping. In CVPR '08.Google Scholar
- Fischler, M. A., and Bolles, R. C. 1981. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24, 6, 381--395. Google ScholarDigital Library
- Gal, R., Sorkine, O., and Cohen-Or, D. 2006. Feature-aware texturing. In EGSR '06, 297--303. Google ScholarDigital Library
- Gleicher, M. L., and Liu, F. 2008. Re-cinematography: Improving the camerawork of casual video. ACM Trans. Multimedia Comput. Commun. Appl. 5, 1, 1--28. Google ScholarDigital Library
- Itti, L., Koch, C., and Niebur, E. 1998. A model of saliency-based visual attention for rapid scene analysis. IEEE Trans. Pattern Anal. Mach. Intell. 20, 11, 1254--1259. Google ScholarDigital Library
- Kang, H.-W., Matsushita, Y., Tang, X., and Chen, X.-Q. 2006. Space-time video montage. In CVPR '06. Google ScholarDigital Library
- Kraevoy, V., Sheffer, A., Cohen-Or, D., and Shamir, A. 2008. Non-homogeneous resizing of complex models. ACM Trans. Graph. 27, 5, 111. Google ScholarDigital Library
- Krähenbühl, P., Lang, M., Hornung, A., and Gross, M. 2009. A system for retargeting of streaming video. ACM Trans. Graph. 28, 5. Google ScholarDigital Library
- Liu, F., and Gleicher, M. 2006. Video retargeting: automating pan and scan. In Multimedia '06, 241--250. Google ScholarDigital Library
- Liu, H., Xie, X., Ma, W.-Y., and Zhang, H.-J. 2003. Automatic browsing of large pictures on mobile devices. In Proceedings of ACM International Conference on Multimedia, 148--155. Google ScholarDigital Library
- Lowe, D. G. 2004. Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vision 60, 2, 91--110. Google ScholarDigital Library
- Rasheed, Z., and Shah, M. 2003. Scene detection in hollywood movies and tv shows. In CVPR '03, vol. 2, II-343--8.Google Scholar
- Rubinstein, M., Shamir, A., and Avidan, S. 2008. Improved seam carving for video retargeting. ACM Trans. Graph. 27, 3, 16. Google ScholarDigital Library
- Rubinstein, M., Shamir, A., and Avidan, S. 2009. Multioperator media retargeting. ACM Trans. Graph. 28, 3, 23. Google ScholarDigital Library
- Santella, A., Agrawala, M., DeCarlo, D., Salesin, D., and Cohen, M. 2006. Gaze-based interaction for semiautomatic photo cropping. In Proceedings of CHI, 771--780. Google ScholarDigital Library
- Setlur, V., Takagi, S., Raskar, R., Gleicher, M., and Gooch, B. 2005. Automatic image retargeting. In MUM '05, 59--68. Google ScholarDigital Library
- Simakov, D., Caspi, Y., Shechtman, E., and Irani, M. 2008. Summarizing visual data using bidirectional similarity. In CVPR '08.Google Scholar
- Sorkine, O., Lipman, Y., Cohen-Or, D., Alexa, M., Rössl, C., and Seidel, H.-P. 2004. Laplacian surface editing. In SGP '04, 179--188. Google ScholarDigital Library
- Suh, B., Ling, H., Bederson, B. B., and Jacobs, D. W. 2003. Automatic thumbnail cropping and its effectiveness. In Proceedings of UIST, 95--104. Google ScholarDigital Library
- Szeliski, R. 2006. Image alignment and stitching: a tutorial. Foundations and Trends in Computer Graphics and Vision 2, 1, 1--104. Google ScholarDigital Library
- Tao, C., Jia, J., and Sun, H. 2007. Active window oriented dynamic video retargeting. In Workshop on Dynamical Vision, ICCV '07.Google Scholar
- Wang, Y.-S., Lee, T.-Y., and Tai, C.-L. 2008. Focus+context visualization with distortion minimization. IEEE Trans. Visualization and Computer Graphics 14, 6. Google ScholarDigital Library
- Wang, Y.-S., Tai, C.-L., Sorkine, O., and Lee, T.-Y. 2008. Optimized scale-and-stretch for image resizing. ACM Trans. Graph. 27, 5, 118. Google ScholarDigital Library
- Wolf, L., Guttmann, M., and Cohen-Or, D. 2007. Non-homogeneous content-driven video-retargeting. In ICCV '07.Google Scholar
- Zhang, Y.-F., Hu, S.-M., and Martin, R. R. 2008. Shrinkability maps for content-aware video resizing. In PG '08.Google Scholar
Index Terms
- Motion-aware temporal coherence for video resizing
Recommendations
Motion-aware temporal coherence for video resizing
SIGGRAPH Asia '09: ACM SIGGRAPH Asia 2009 papersTemporal coherence is crucial in content-aware video retargeting. To date, this problem has been addressed by constraining temporally adjacent pixels to be transformed coherently. However, due to the motion-oblivious nature of this simple constraint, ...
Motion-based video retargeting with optimized crop-and-warp
We introduce a video retargeting method that achieves high-quality resizing to arbitrary aspect ratios for complex videos containing diverse camera and dynamic motions. Previous content-aware retargeting methods mostly concentrated on spatial ...
Motion-based video retargeting with optimized crop-and-warp
SIGGRAPH '10: ACM SIGGRAPH 2010 papersWe introduce a video retargeting method that achieves high-quality resizing to arbitrary aspect ratios for complex videos containing diverse camera and dynamic motions. Previous content-aware retargeting methods mostly concentrated on spatial ...
Comments