Abstract
We present a novel hyperspectral image reconstruction algorithm, which overcomes the long-standing tradeoff between spectral accuracy and spatial resolution in existing compressive imaging approaches. Our method consists of two steps: First, we learn nonlinear spectral representations from real-world hyperspectral datasets; for this, we build a convolutional autoencoder which allows reconstructing its own input through its encoder and decoder networks. Second, we introduce a novel optimization method, which jointly regularizes the fidelity of the learned nonlinear spectral representations and the sparsity of gradients in the spatial domain, by means of our new fidelity prior. Our technique can be applied to any existing compressive imaging architecture, and has been thoroughly tested both in simulation, and by building a prototype hyperspectral imaging system. It outperforms the state-of-the-art methods from each architecture, both in terms of spectral accuracy and spatial resolution, while its computational complexity is reduced by two orders of magnitude with respect to sparse coding techniques. Moreover, we present two additional applications of our method: hyperspectral interpolation and demosaicing. Last, we have created a new high-resolution hyperspectral dataset containing sharper images of more spectral variety than existing ones, available through our project website.
Supplemental Material
Available for Download
Supplemental material.
- M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat. G. Irving, M. Isard, M. Kudlur, J. Levenberg, R. Monga, S. Moore, D. G. Murray, B. Steiner, P. Tucker, V. Vasudevan, P. Warden, M. Wicke, Y. Yu, and X. Zheng. 2016. TensorFlow: A System for Large-scale Machine Learning. In Proc. USENIX Conf. Operating Systems Design and Implementation (OSDI'16). 265--283. Google ScholarDigital Library
- Manya V. Afonso, José M. Bioucas-Dias, and Mário A. T. Figueiredo. 2011. An Augmented Lagrangian Approach to the Constrained Optimization Formulation of Imaging Inverse Problems. IEEE Trans. Image Processing (TIP) 20, 3 (2011), 681--695. Google ScholarDigital Library
- Sara Alvarez, Timo Kunkel, and Belen Masia. 2016. Practical Low-Cost Recovery of Spectral Power Distributions. Computer Graphics Forum 35, 1 (2016), 166--178. Google ScholarDigital Library
- Michael Attas, Edward Cloutis, Catherine Collins, Douglas Goltz, Claudine Majzels. James R Mansfield, and Henry H Mantsch. 2003. Near-infrared spectroscopic imaging in art conservation: investigation of drawing constituents. J. Cultural Heritage 4, 2 (2003), 127--136.Google ScholarCross Ref
- Seung-Hwan Baek, Incheol Kim, Diego Gutierrez, and Min H. Kim. 2017. Compact Single-Shot Hyperspectral Imaging Using a Prism. ACM Transactions on Graphics (Proc. SIGGRAPH Asia 2017) 36, 6 (2017). Google ScholarDigital Library
- Jose M. Bioucas-Dias and Mario A. T. Figueiredo. 2007. A new TwIST: two-step iterative shrinkage/thresholding for image restoration. IEEE Trans. Image Processing (TIP) 16, 12 (2007), 2992--3004. Google ScholarDigital Library
- N. Brusco, S. Capeleto, M. Fedel, A. Paviotti, L. Poletto, G. M. Cortelazzo, and G. Tondello. 2006. A system for 3D modeling frescoed historical buildings with multispectral texture information. Machine Vision and Applications 17, 6 (2006), 373--393. Google ScholarDigital Library
- Xun Cao, Hao Du, Xin Tong, Qionghai Dai, and Stephen Lin. 2011. A prism-mask system for multispectral video acquisition. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 33, 12 (2011), 2423--2435. Google ScholarDigital Library
- A. Chakrabarti and T. Zickler. 2011. Statistics of Real-World Hyperspectral Images. In Proc. IEEE Conf. Comput. Vis. Pattern. Recognit. (CVPR). 193--200. Google ScholarDigital Library
- Yushi Chen, Zhouhan Lin, Xing Zhao, Gang Wang, and Yanfeng Gu. 2014. Deep Learning-Based Classification of Hyperspectral Data. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 7, 6 (2014), 2094--2107.Google ScholarCross Ref
- B. Du, W. Xiong, J. Wu, L. Zhang, L. Zhang, and D. Tao. 2016. Stacked Convolutional Denoising Auto-Encoders for Feature Representation. IEEE Trans. Cybernetics PP. 99 (2016), 1--11.Google Scholar
- Ying Fu, Yinqiang Zheng, Imari Sato, and Yoichi Sato. 2016. Exploiting Spectral-Spatial Correlation for Coded Hyperspectral Image Restoration. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
- Nahum Gat. 2000. Imaging spectroscopy using tunable filters: a review. In Proc. AeroSense 2000. 50--64.Google Scholar
- M E Gehm, R John, D J Brady, R M Willett, and T J Schulz. 2007. Single-shot compressive spectral imaging with a dual-disperser architecture. OSA Optics Express (OE) 15, 21 (2007), 14013--27.Google ScholarCross Ref
- Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Aistats, Vol. 9. 249--256.Google Scholar
- Mohammad Golbabaee, Simon Arberet, and Pierre Vandergheynst. 2013. Compressive source separation: Theory and methods for hyperspectral imaging. IEEE Transactions on Image Processing (TIP) 22, 12 (2013), 5096--5110. Google ScholarDigital Library
- Ralf Habel, Michael Kudenov, and Michael Wimmer. 2012. Practical spectral photography. In Computer Graphics Forum (Proc. EUROGRAPHICS 2012), Vol. 31. Wiley Online Library, 449--458. Google ScholarDigital Library
- Geoffrey E Hinton and Ruslan R Salakhutdinov. 2006. Reducing the dimensionality of data with neural networks. Science 313, 5786 (2006), 504--507.Google Scholar
- Gudrun Hoye and Andrei Fridman. 2013. Mixel camera - a new push-broom camera concept for high spatial resolution keystone-free hyperspectral imaging. OSA Optics Express (OE) 21, 9 (2013), 11057--11077.Google ScholarCross Ref
- Francisco Imai, Lawrence Taplin, and Ellen Day. 2002. Comparison of the accuracy of various transformations from multi-band images to reflectance spectra. Technical Report ISO 12233:2000. Rochester Institute of Technology.Google Scholar
- Daniel S. Jeon, Inchang Choi, and Min H. Kim. 2016. Multisampling Compressive Video Spectroscopy. Computer Graphics Forum 35, 2 (2016), 467--477.Google ScholarCross Ref
- Min H Kim. 2013. 3D Graphics Techniques for Capturing and Inspecting Hyperspectral Appearance. In Ubiquitous Virtual Reality (ISUVR), 2013 Int. Symp. on. IEEE, 15--18. Google ScholarDigital Library
- Min H. Kim, Todd Alan Harvey, David S. Kittle, Holly Rushmeier, Julie Dorsey, Richard O. Prum, and David J. Brady. 2012a. 3D Imaging Spectroscopy for Measuring Hyperspectral Patterns on Solid Objects. ACM Trans. Graph. (TOG) 31, 4 (2012), 38:1--11. Google ScholarDigital Library
- Min H. Kim, Holly Rushmeier, John ffrench, and Irma Passeri. 2012b. Developing Open-Source Software for Art Conservators. In VAST12: The 13th International Symposium on Virtual Reality, Archaeology and Intelligent Cultural Heritage. Eurographics Association, Brighton, England, 97--104.Google Scholar
- Min H. Kim, Holly Rushmeier, John ffrench, Irma Passeri, and David Tidmarsh. 2014. Hyper3D: 3D Graphics Software for Examining Cultural Artifacts. ACM Journal on Computing and Cultural Heritage 7, 3 (2014), 1:1--19. Google ScholarDigital Library
- Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. In The International Conference on Learning Representations (ICLR).Google Scholar
- David Kittle, Kerkil Choi, Ashwin Wagadarikar, and David J Brady. 2010. Multiframe image estimation for coded aperture snapshot spectral imagers. Applied Optics (AO) 49, 36 (2010), 6824--6833.Google ScholarCross Ref
- Kuldeep Kulkarni, Suhas Lohit, Pavan Turaga, Ronan Kerviche, and Amit Ashok. 2016. ReconNet: Non-Iterative Reconstruction of Images From Compressively Sensed Measurements. In The IEEE Conf. Computer Vision and Pattern Recognition (CVPR).Google Scholar
- HaebomLee and Min H. Kim. 2014. Building a Two-Way Hyperspectral Imaging System with Liquid Crystal Tunable Filters. In Proc. Int. Conf. Image and Signal Processing (ICISP). 26--34.Google Scholar
- Biao Leng, Shuang Guo, Xiangyang Zhang, and Zhang Xiong. 2015. 3D object retrieval with stacked local convolutional autoencoder. Signal Processing 112 (2015), 119--128. Google ScholarDigital Library
- Chengbo Li, Ting Sun, Kevin F. Kelly, and Yin Zhang. 2012. A Compressive Sensing and Unmixing Scheme for Hyperspectral Data Processing. IEEE Trans. Image Processing (TIP) 21, 3 (2012), 1200--1210. Google ScholarDigital Library
- Yunsong Li, Weiying Xie, and Huaqing Li. 2017. Hyperspectral image reconstruction by deep convolutional neural network for classification. Pattern Recognition (PR) 63 (2017), 371 -- 383. Google ScholarDigital Library
- Xing Lin, Yebin Liu, Jiamin Wu, and Qionghai Dai. 2014. Spatial-spectral Encoded Compressive Hyperspectral Imaging. ACM Trans. Graph. (TOG) 33, 6 (2014). Google ScholarDigital Library
- Zhouhan Lin, Yushi Chen, Xing Zhao, and Gang Wang. 2013. Spectral-spatial classification of hyperspectral image using autoencoders. In Proc. Int. Conf. Signal Process. Commun. 1--5.Google Scholar
- Jonathan Long, Evan Shelhamer, and Trevor Darrell. 2015. Fully Convolutional Networks for Semantic Segmentation. In Proc. IEEE Conf. Comput. Vis. Pattern. Recognit. (CVPR). 3431--3440.Google ScholarCross Ref
- Xiaorui Ma, Hongyu Wang, and Jie Geng. 2016. Spectral-Spatial Classification of Hyperspectral Image Based on Deep Auto-Encoder. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 9, 9 (2016), 4073--4085.Google ScholarCross Ref
- Emmanuel Maggiori, Yuliya Tarabalka, Guillaume Charpiat, and Pierre Alliez. 2016. Convolutional Neural Networks for Large-Scale Remote Sensing Image Classification. IEEE Trans. Geosci. Remote Sens. (2016), 645 -- 657.Google Scholar
- Alkhazur Manakov, John F Restrepo, Oliver Klehm, Ramon Hegedüs, Elmar Eisemann, Hans-Peter Seidel, and Ivo Ihrke. 2013. A reconfigurable camera add-on for high dynamic range, multispectral, polarization, and light-field imaging. ACM Trans. Graph. (Proc. SIGGRAPH 2013) 32, 4 (2013), 47. Google ScholarDigital Library
- Julio Marco, Quercus Hernandez, Adolfo MuÃśoz, Yue Dong, Adrian Jarabo, Min Kim, Xin Tong, and Diego Gutierrez. 2017. DeepToF: Off-the-Shelf Real-time Correction of Multipath Interference in Time-of-Flight Imaging. ACM Transactions on Graphics (Proc. SIGGRAPH Asia 2017) 36, 6 (2017). Google ScholarDigital Library
- Gabriel Martin, Jose M. Bioucas-Dias, and Antonio J. Plaza. 2015. HYCA: A New Technique for Hyperspectral Compressive Sensing. IEEE Trans. Geoscience and Remote Sensing 53, 5 (2015), 2819--2831.Google ScholarCross Ref
- Jonathan Masci, Ueli Meier, Dan Cireşan, and Jürgen Schmidhuber. 2011. Stacked convolutional auto-encoders for hierarchical feature extraction. In Proc. Int. Conf. Artificial Neural Networks (ICANN). 52--59. Google ScholarDigital Library
- Yi Peng, Deyu Meng, Zongben Xu, Chenqiang Gao, Yi Yang, and Biao Zhang. 2014. Decomposable Nonlocal Tensor Dictionary Learning for Multispectral Image De-noising. In Proc. IEEE Conf. Comput. Vis. Pattern. Recognit. (CVPR). Google ScholarDigital Library
- Wallace M Porter and Harry T Enmark. 1987. A system overview of the airborne visible/infrared imaging spectrometer (AVIRIS). In Technical Symposium. International Society for Optics and Photonics, 22--31.Google ScholarCross Ref
- Ajit Rajwade, David Kittle, Tsung-Han Tsai, David Brady, and Lawrence Carin. 2013. Coded Hyperspectral Imaging and Blind Compressive Sensing. SIAM Journal on Imaging Sciences 6, 2 (2013).Google ScholarCross Ref
- Konstantinos Rapantzikos and Costas Balas. 2005. Hyperspectral imaging: potential in non-destructive analysis of palimpsests. In Proc. IEEE Int. Conf. Image Processing (ICIP), Vol. 2. IEEE, II--618.Google ScholarCross Ref
- K. Simonyan and A. Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. In Proc. Int. Conf. Learning Representation (ICLR).Google Scholar
- Yaniv Taigman, Ming Yang, Marc'Aurelio Ranzato, and Lior Wolf. 2014. DeepFace: Closing the Gap to Human-Level Performance in Face Verification. In Proc. IEEE Conf. Comput. Vis. Pattern. Recognit. (CVPR). Google ScholarDigital Library
- Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. 2010. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. J. Mach. Learn. Res. 11 (2010), 3371--3408. Google ScholarDigital Library
- J. J. Vos. 1978. Colorimetric and photometric properties of a 2-deg fundamental observer. Color Res. Appl. 3 (1978), 125--128.Google ScholarCross Ref
- Ashwin Wagadarikar, Renu John, Rebecca Willett, and David Brady. 2008. Single disperser design for coded aperture snapshot spectral imaging. OSA Applied Optics (AO) 47, 10 (2008), B44--B51.Google ScholarCross Ref
- Lizhi Wang, Zhiwei Xiong, Dahua Gao, Guangming Shi, Wenjun Zeng, and Feng Wu. 2015. High-Speed Hyperspectral Video Acquisition With a Dual-Camera Architecture. In Proc. IEEE Conf. Comput. Vis. Pattern. Recognit. (CVPR).Google ScholarCross Ref
- S.J. Wright, R.D. Nowak, and M.A.T. Figueiredo. 2009. Sparse reconstruction by separable approximation. IEEE TSP 57, 7 (2009), 2479--2493. Google ScholarDigital Library
- Yuehao Wu, Iftekhar O Mirza, Gonzalo R Arce, and Dennis W Prather. 2011. Development of a digital-micromirror-device-based multishot snapshot spectral imaging system. OSA Optics Letters (OL) 36, 14 (2011), 2692--4.Google ScholarCross Ref
- Chen Xing, Li Ma, and Xiaoquan Yang. 2016. Stacked Denoise Autoencoder Based Feature Extraction and Classification for Hyperspectral Images. Journal of Sensors 2016 (2016), 1--10.Google Scholar
- F. Yasuma, T. Mitsunaga, D. Iso, and S. K. Nayar. 2010. Generalized Assorted Pixel Camera: Postcapture Control of Resolution, Dynamic Range, and Spectrum. IEEE Trans. Image Processing (TIP) 19, 9 (Sept 2010), 2241--2253. Google ScholarDigital Library
- Jaime Zabalza, Jinchang Ren, Jiangbin Zheng, Huimin Zhao, Chunmei Qing, Zhijing Yang, Peijun Du, and Stephen Marshall. 2016. Novel Segmented Stacked Autoencoder for Effective Dimensionality Reduction and Feature Extraction in Hyperspectral Imaging. Neurocomputing 214 (2016), 1--10. Google ScholarDigital Library
- Qiang Zhang, Robert Plemmons, David Kittle, David Brady, and Sudhakar Prasad. 2011. Joint segmentation and reconstruction of hyperspectral data with compressed measurements. OSA Applied Optics (AO) 50, 22 (2011), 4417--4435.Google ScholarCross Ref
- Yuting Zhang, Kibok Lee, and Honglak Lee. 2016. Augmenting Supervised Neural Networks with Unsupervised Objectives for Large-scale Image Classification. In Proc. International Conference on Machine Learning (ICML). Google ScholarDigital Library
Index Terms
- High-quality hyperspectral reconstruction using a spectral prior
Recommendations
Compact single-shot hyperspectral imaging using a prism
We present a novel, compact single-shot hyperspectral imaging method. It enables capturing hyperspectral images using a conventional DSLR camera equipped with just an ordinary refractive prism in front of the camera lens. Our computational imaging ...
Hyperspectral image super-resolution through clustering-based sparse representation
AbstractPromoting the spatial resolution of hyperspectral sensors is expected to improve computer vision tasks. However, due to the physical limitations of imaging sensors, the hyperspectral image is often of low spatial resolution. In this paper, we ...
Super-resolution reconstruction of hyperspectral images
Hyperspectral images are used for aerial and space imagery applications, including target detection, tracking, agricultural, and natural resource exploration. Unfortunately, atmospheric scattering, secondary illumination, changing viewing angles, and ...
Comments