Stereoscopic image quality assessment method based on binocular combination saliency model
Introduction
With the development of three-dimensional (3D) technologies, more and more 3D contents for 3DTV and 3D cinema are produced. However, it also brings us many new issues and challenges [1], [2], [3]. During the process of stereoscopic content creation, transmission, processing and display, various distortions that affect the quality perception may be introduced [4]. Consequently, it is very necessary to build an effective tool to measure the quality of stereoscopic images. Over the past decades, numerous 2D image quality assessment (IQA) methods have been proposed [5], [6], [7]. Gao et al. [8] proposed an image quality assessment model based on multiscale geometric analysis, which has strong links with the human visual system. Focusing on the problem of color distortion, He et al. [9] proposed a color fractal structure model to evaluate the quality of image. Later, they [10] proposed an effective universal blind image quality assessment method by using the sparse representation of the tertiary natural scene statistics, which achieved great results in image quality assessment area. However, most of existing methods fail to effectively evaluate the quality of 3D images. Compared with the two-dimensional (2D) images, 3D images provide the sense of depth perception, which makes the quality assessment for 3D images more difficult.
To solve the aforementioned problem, different stereoscopic image quality assessment (SIQA) algorithms have been designed, which can be categorized into the subjective and objective assessment models. A subjective assessment model represents the direct reflection of the human visual system (HVS), thus it is regarded as the most reasonable and precise assessment method. Lots of works have been progressing steadily [11], [12]. Zhou et al. [13] built a public 3D images database based on the subjective quality assessment method. Lee et al. [14] proposed a paired comparison based on a subjective experiment to minimize the effect of subject׳s limited 3D experience. However, the subjective model is time-consuming and impractical for online applications. Therefore, the objective model, which can be used to reliably predict the quality of stereo images, attracts more attentions [15], [16].
Because of the depth information, it is not an easy matter to design an effective objective method to evaluate the quality of stereoscopic images. Recently, lots of studies have been proposed which can be mainly divided into three categories. The first category is to apply the 2D IQA metrics to evaluate the quality of stereoscopic images, such as VIF [17], PSNR, SSIM [18], MS-SSIM [19], GSM [20] and others [21], [22]. Many researchers simply applied 2D IQA algorithms to the left and right images separately, and took the average of the left and right quality scores as the final score [23], [24]. However, these methods are inefficient in predicting the quality of stereoscopic images. The second category of models takes the depth information into account. For example, Campisi et al., by considering the additional depth information in stereoscopic images, built a quality evaluation method [25]. Benoit et al. [26] combined the image quality and the depth quality together to calculate the final image quality. Hewage et al. [27] proposed an approach by comparing the contours of original and impaired depth maps. Similarly, Xing et al. [28] and Boev et al. [29] also developed the effective metrics to measure the quality of stereoscopic images based on depth information.
Although great efforts have been conducted to evaluate the quality of stereoscopic images, it is still difficult to build a precisely evaluation method by using the real 3D information, such as disparity map or cyclopean image. The third approach is then developed based on the perceptual characteristics of human binocular visual system (HVS) [30], [31]. Ha et al. [32] built a perceptual quality assessment metric by considering the factors of temporal variation and disparity distribution. Maalouf et al. [33] integrated the left and right images into a cyclopean image to simulate human brain perception to derive a quality index. Shao et al. [34], based on binocular energy responses, proposed another effective quality assessment method of stereoscopic images. What׳s more, Chen et al. [35] addressed binocular rivalry issues by modeling the binocular suppression behavior and developed an effective model to measure the quality of stereoscopic images. Lin et al. [36] integrated the binocular combination behavior into the existing 2D objective metrics and built the final quality evaluation metric. These models achieve better results than the above two types of approaches, which indicates that the human binocular characteristics play an important role in stereoscopic image quality assessment. However, all the above SIQA models selectively neglect human visual sensitivity characteristics. When human views the stereoscopic images, human eyes attempt to focus on the object that they are interested in, which is called visual saliency attention [37], [38]. Xiu et al. [39] then proposed an objective saliency structure stereoscopic image quality assessment model based on the saliency map of each eye view and the texture sensitivity, and the experimental results indicated that the proposed metric achieves high consistent with human vision perception. It also indicates that human visual saliency map is helpful to predict the quality of stereoscopic images.
In this paper, inspired by previous works, we take advantage of the relationship between binocular combination perception and visual stereoscopic saliency information to deal with the stereoscopic image quality assessment problem. In particular, the effect of binocular rivalry between left and right eyes is introduced to get a reasonable binocular combination model. A novel stereoscopic saliency detection framework is also derived and incorporated to the binocular combination information by assigning various weights to different regions with different levels of importance. Since binocular combination perception quality intrinsically reflects the quality of stereoscopic image, so the method of combining stereoscopic saliency sense with the binocular combination model attains much accurate quality assessment results. The main contributions of this work are as follows: (1) a developed 3D visual saliency map for stereoscopic images is built which greatly reduces the computational complexity. (2) By considering the binocular combination properties, we use the proposed 3D visual saliency map to assign higher weights to more perceptually important area, which plays an important role in precisely quality assessment of stereoscopic images.
The remainder of this paper is organized as follows. In Section 2, a review of related works and backgrounds on human visual combination behavior and visual saliency detected models is presented. The overall proposed 3D QA framework is described in Section 3. Section 4 describes the experimental results and performance analysis. Finally, Section 5 concludes this paper with a discussion and the imagination of our future work.
Section snippets
Related works and background
In order to explain the proposed SIQA model in Section 3, here we give a brief review of relevant works and backgrounds.
The proposed work
The most effective way of predicting the quality of stereoscopic images is to directly estimate the quality of the true combined image formed within human brain. However, it is difficult to obtain the true combined image that people really perceived in mind. Therefore, we propose a combination model that is close to the true combined image to reflect the stereoscopic image quality. Based on the above physiological discoveries, binocular combination and 3D visual saliency map are both taken into
Stereoscopic image quality database
To verify the performance of the proposed metric, LIVE 3D Image Quality Database of the University of Texas at Austin is used in the experiment, which contains 365 distorted images generated from 20 reference images, shown in Fig. 3. Five types of distortions are applied to the reference images at various levels (80 for JP2K, JPEG, WN and FF respectively; 45 for Blur). All distortions are symmetric in nature, and each distorted image is assigned a quantitative subjective quality score [73].
To
Conclusions
This paper proposes an effective quality assessment method of stereoscopic images based on human 3D visual saliency map and binocular visual characteristics, which can precisely predict the quality of 3D images that are contaminated by different types of symmetric distortions. A great contribution of this work is that it provides a novel quality assessment method where the binocular combination behavior and visual saliency characteristics are both considered. What׳s more, our model has two
Acknowledgment
The authors would like to thank Prof. Alan C. Bovik for providing the LIVE 3D IQA Database. This research is partially supported by the National Natural Science Foundation of China (Nos. 61471260 and 61271324), and Program for New Century Excellent Talents in University (NCET-12-0400).
References (81)
- et al.
Fully automatic 3D facial expression recognition using polytypic multi-block local binary patterns
Signal Process.
(2015) - et al.
Multiview coding and error correction coding for 3D video over noisy channels
Signal Process.: Image Commun.
(2015) - et al.
Learning to rank for blind image quality assessment
IEEE Trans. Neural Netw. Learn. Syst.
(2015) - et al.
Objective image quality assessment of 3D synthesized views
Signal Process.: Image Commun.
(2015) - et al.
Natural image statistics based 3D reduced reference image quality assessment in contourlet domain
Neurocomputing
(2015) - et al.
A natural image quality evaluation metric
Signal Process.
(2009) - et al.
A perceptual stereoscopic image quality assessment model accounting for binocular combination behavior
J. Vis. Commun. Image Represent.
(2015) - et al.
Binocular energy response based quality assessment of stereoscopic images
Digit. Signal Process.
(2014) - et al.
Saliency structure stereoscopic image quality assessment method
Optik
(2014) - et al.
Neural dynamics of binocular brightness perception
Vis. Res.
(1999)
The visual processes underlying binocular brightness summation
Vis. Res.
Human binocular interactiontowards a neural model
Vis. Res.
Predictive coding as a model of the V1 saliency map hypothesis
Neural Netw.
The objective quality assessment of stereo image
Neurocomputing
Binocular rivalrysuppression depends on orientation and spatial frequency
Vis. Res.
Contour integration by the human visual systemevidence for a local association field
Vis. Res.
Improvement in visual sensitivity by changes in local contextparallel studies in human observers and in V1 of alert monkeys
Neuron
Grouping visual features during binocular rivalry?
Vis. Res.
Model for simultaneous face identification and facial expression recognition
Neurocomputing
The five-parameter logistica characterization and comparison with the four-parameter logistic
Anal. Biochem.
On the performance of Manhattan nonnegative matrix factorization
IEEE Trans. Neural Netw. Learn. Syst. PP
Objective evaluation criteria for stereo camera shooting quality under different shooting parameters and shooting distances
IEEE Sens. J.
Blind image quality assessment via deep learning
IEEE Trans. Neural Netw. Learn. Syst.
VSIa visual saliency-induced index for perceptual image quality assessment
IEEE Trans. Image Process.
Image quality assessment based on multiscale geometric analysis
IEEE Trans. Image Process.
Viewing comfort with stereoscopic picturesan experimental study on the subjective effects of disparity magnitude and depth of focus
J. Soc. Inf. Disp.
Subjective evaluation of stereoscopic imageseffects of camera parameters and display duration
IEEE Trans. Circuits Syst. Video Technol.
Paired comparison-based subjective quality assessment of sereoscopic image
Multim. Tools Appl.
Image information and visual quality
IEEE Trans. Image Process.
Image quality assessment: from error visibility to structural similarity
IEEE Trans. Image Process.
Image quality assessment based on gradient similarity
IEEE Trans. Image Process.
Universal blind image quality assessment metrics via natural scene statistics and multiple kernel learning
IEEE Trans. Neural Netw. Learn. Syst.
Cited by (106)
Jomodevi: A joint motion and depth visibility prediction algorithm for perceived stereoscopic 3D quality
2022, Signal Processing: Image CommunicationCitation Excerpt :In this section, we review the relevant scientific literature on S3D video quality assessment. The early works in the field of S3D quality assessment [5–11] present solutions that use 2D image quality assessment (IQA) and video quality assessment (VQA) algorithms on the individual views of S3D videos in order to estimate the perceived quality. In these objective metrics, the well-known 2D IQA and VQA models (SSIM [12], MS-SSIM [13], VIF [14], STAMD [15], VQM [16], etc.) are applied on the frame-wise individual views to predict the final quality scores of S3D videos.
Multiscale multilevel context and multimodal fusion for RGB-D salient object detection
2021, Signal ProcessingCitation Excerpt :Salient object detection (SOD), in which the aim is to extract predominant objects from a natural scene, has been a fundamental goal in computer vision. It can be used as a preprocessing step for a variety of multimedia applications, such as compression [1,2], segmentation [3,4], retargeting [5], retrieval [6], recognition [7], quality assessment [8–10], object detection [11,12], and photo cropping [13]. Over the last few years, several SOD models for red–green–blue (RGB) images [14–20] or videos [21–24] have been proposed and improved, but challenges related to similar surroundings, complex backgrounds, and varying illumination still hinder the advancement of SOD.
Perceptual objective quality assessment of stereoscopic stitched images
2020, Signal ProcessingCitation Excerpt :Compared with 2D images, stereoscopic images deliver an additional dimension of depth information. Currently, stereoscopic image quality assessment approaches [33–35,41–46] have been extensively studied. These methods related to imaging, coding, delivery, display.
Texture synthesis quality assessment using perceptual texture similarity
2020, Knowledge-Based SystemsPerceived quality measurement of stereoscopic 3D images based on sparse representation and binocular combination
2019, Digital Signal Processing: A Review JournalCitation Excerpt :In [14], You et al. predicted the perceptual quality of an S3D image by nonlinearly combining 2D quality estimate with disparity quality estimate. ( 3) Recently, studies have exploited deeper understanding of binocular visual perception to devise FR S3D-IQM metrics [15–30]. In [15], Lin et al. presented an FR S3D-IQM framework by considering binocular rivalry.