Elsevier

Signal Processing

Volume 125, August 2016, Pages 237-248
Signal Processing

Stereoscopic image quality assessment method based on binocular combination saliency model

https://doi.org/10.1016/j.sigpro.2016.01.019Get rights and content

Highlights

  • An effective 3D visual saliency model is proposed.

  • A full-reference SIQA model is built based on human binocular characteristics.

  • It has less computing complexity and higher consistency with the subjective values.

  • It can precisely assess the quality of images with different types of distortions.

Abstract

The objective quality assessment of stereoscopic images plays an important role in three-dimensional (3D) technologies. In this paper, we propose an effective method to evaluate the quality of stereoscopic images that are afflicted by symmetric distortions. The major technical contribution of this paper is that the binocular combination behaviors and human 3D visual saliency characteristics are both considered. In particular, a new 3D saliency map is developed, which not only greatly reduces the computational complexity by avoiding calculation of the depth information, but also assigns appropriate weights to the image contents. Experimental results indicate that the proposed metric not only significantly outperforms conventional 2D quality metrics, but also achieves higher performance than the existing 3D quality assessment models.

Introduction

With the development of three-dimensional (3D) technologies, more and more 3D contents for 3DTV and 3D cinema are produced. However, it also brings us many new issues and challenges [1], [2], [3]. During the process of stereoscopic content creation, transmission, processing and display, various distortions that affect the quality perception may be introduced [4]. Consequently, it is very necessary to build an effective tool to measure the quality of stereoscopic images. Over the past decades, numerous 2D image quality assessment (IQA) methods have been proposed [5], [6], [7]. Gao et al. [8] proposed an image quality assessment model based on multiscale geometric analysis, which has strong links with the human visual system. Focusing on the problem of color distortion, He et al. [9] proposed a color fractal structure model to evaluate the quality of image. Later, they [10] proposed an effective universal blind image quality assessment method by using the sparse representation of the tertiary natural scene statistics, which achieved great results in image quality assessment area. However, most of existing methods fail to effectively evaluate the quality of 3D images. Compared with the two-dimensional (2D) images, 3D images provide the sense of depth perception, which makes the quality assessment for 3D images more difficult.

To solve the aforementioned problem, different stereoscopic image quality assessment (SIQA) algorithms have been designed, which can be categorized into the subjective and objective assessment models. A subjective assessment model represents the direct reflection of the human visual system (HVS), thus it is regarded as the most reasonable and precise assessment method. Lots of works have been progressing steadily [11], [12]. Zhou et al. [13] built a public 3D images database based on the subjective quality assessment method. Lee et al. [14] proposed a paired comparison based on a subjective experiment to minimize the effect of subject׳s limited 3D experience. However, the subjective model is time-consuming and impractical for online applications. Therefore, the objective model, which can be used to reliably predict the quality of stereo images, attracts more attentions [15], [16].

Because of the depth information, it is not an easy matter to design an effective objective method to evaluate the quality of stereoscopic images. Recently, lots of studies have been proposed which can be mainly divided into three categories. The first category is to apply the 2D IQA metrics to evaluate the quality of stereoscopic images, such as VIF [17], PSNR, SSIM [18], MS-SSIM [19], GSM [20] and others [21], [22]. Many researchers simply applied 2D IQA algorithms to the left and right images separately, and took the average of the left and right quality scores as the final score [23], [24]. However, these methods are inefficient in predicting the quality of stereoscopic images. The second category of models takes the depth information into account. For example, Campisi et al., by considering the additional depth information in stereoscopic images, built a quality evaluation method [25]. Benoit et al. [26] combined the image quality and the depth quality together to calculate the final image quality. Hewage et al. [27] proposed an approach by comparing the contours of original and impaired depth maps. Similarly, Xing et al. [28] and Boev et al. [29] also developed the effective metrics to measure the quality of stereoscopic images based on depth information.

Although great efforts have been conducted to evaluate the quality of stereoscopic images, it is still difficult to build a precisely evaluation method by using the real 3D information, such as disparity map or cyclopean image. The third approach is then developed based on the perceptual characteristics of human binocular visual system (HVS) [30], [31]. Ha et al. [32] built a perceptual quality assessment metric by considering the factors of temporal variation and disparity distribution. Maalouf et al. [33] integrated the left and right images into a cyclopean image to simulate human brain perception to derive a quality index. Shao et al. [34], based on binocular energy responses, proposed another effective quality assessment method of stereoscopic images. What׳s more, Chen et al. [35] addressed binocular rivalry issues by modeling the binocular suppression behavior and developed an effective model to measure the quality of stereoscopic images. Lin et al. [36] integrated the binocular combination behavior into the existing 2D objective metrics and built the final quality evaluation metric. These models achieve better results than the above two types of approaches, which indicates that the human binocular characteristics play an important role in stereoscopic image quality assessment. However, all the above SIQA models selectively neglect human visual sensitivity characteristics. When human views the stereoscopic images, human eyes attempt to focus on the object that they are interested in, which is called visual saliency attention [37], [38]. Xiu et al. [39] then proposed an objective saliency structure stereoscopic image quality assessment model based on the saliency map of each eye view and the texture sensitivity, and the experimental results indicated that the proposed metric achieves high consistent with human vision perception. It also indicates that human visual saliency map is helpful to predict the quality of stereoscopic images.

In this paper, inspired by previous works, we take advantage of the relationship between binocular combination perception and visual stereoscopic saliency information to deal with the stereoscopic image quality assessment problem. In particular, the effect of binocular rivalry between left and right eyes is introduced to get a reasonable binocular combination model. A novel stereoscopic saliency detection framework is also derived and incorporated to the binocular combination information by assigning various weights to different regions with different levels of importance. Since binocular combination perception quality intrinsically reflects the quality of stereoscopic image, so the method of combining stereoscopic saliency sense with the binocular combination model attains much accurate quality assessment results. The main contributions of this work are as follows: (1) a developed 3D visual saliency map for stereoscopic images is built which greatly reduces the computational complexity. (2) By considering the binocular combination properties, we use the proposed 3D visual saliency map to assign higher weights to more perceptually important area, which plays an important role in precisely quality assessment of stereoscopic images.

The remainder of this paper is organized as follows. In Section 2, a review of related works and backgrounds on human visual combination behavior and visual saliency detected models is presented. The overall proposed 3D QA framework is described in Section 3. Section 4 describes the experimental results and performance analysis. Finally, Section 5 concludes this paper with a discussion and the imagination of our future work.

Section snippets

Related works and background

In order to explain the proposed SIQA model in Section 3, here we give a brief review of relevant works and backgrounds.

The proposed work

The most effective way of predicting the quality of stereoscopic images is to directly estimate the quality of the true combined image formed within human brain. However, it is difficult to obtain the true combined image that people really perceived in mind. Therefore, we propose a combination model that is close to the true combined image to reflect the stereoscopic image quality. Based on the above physiological discoveries, binocular combination and 3D visual saliency map are both taken into

Stereoscopic image quality database

To verify the performance of the proposed metric, LIVE 3D Image Quality Database of the University of Texas at Austin is used in the experiment, which contains 365 distorted images generated from 20 reference images, shown in Fig. 3. Five types of distortions are applied to the reference images at various levels (80 for JP2K, JPEG, WN and FF respectively; 45 for Blur). All distortions are symmetric in nature, and each distorted image is assigned a quantitative subjective quality score [73].

To

Conclusions

This paper proposes an effective quality assessment method of stereoscopic images based on human 3D visual saliency map and binocular visual characteristics, which can precisely predict the quality of 3D images that are contaminated by different types of symmetric distortions. A great contribution of this work is that it provides a novel quality assessment method where the binocular combination behavior and visual saliency characteristics are both considered. What׳s more, our model has two

Acknowledgment

The authors would like to thank Prof. Alan C. Bovik for providing the LIVE 3D IQA Database. This research is partially supported by the National Natural Science Foundation of China (Nos. 61471260 and 61271324), and Program for New Century Excellent Talents in University (NCET-12-0400).

References (81)

  • G.R. Engel

    The visual processes underlying binocular brightness summation

    Vis. Res.

    (1967)
  • A.I. Cogan

    Human binocular interactiontowards a neural model

    Vis. Res.

    (1987)
  • M.W. Spratling

    Predictive coding as a model of the V1 saliency map hypothesis

    Neural Netw.

    (2012)
  • N. Yun et al.

    The objective quality assessment of stereo image

    Neurocomputing

    (2013)
  • M. Fahle

    Binocular rivalrysuppression depends on orientation and spatial frequency

    Vis. Res.

    (1982)
  • D.J. Field et al.

    Contour integration by the human visual systemevidence for a local association field

    Vis. Res.

    (1993)
  • M.K. Kapadia et al.

    Improvement in visual sensitivity by changes in local contextparallel studies in human observers and in V1 of alert monkeys

    Neuron

    (1995)
  • D. Alais et al.

    Grouping visual features during binocular rivalry?

    Vis. Res.

    (1999)
  • H. Zheng et al.

    Model for simultaneous face identification and facial expression recognition

    Neurocomputing

    (2016)
  • P.G. Gottschalk et al.

    The five-parameter logistica characterization and comparison with the four-parameter logistic

    Anal. Biochem.

    (2005)
  • T.L. Liu et al.

    On the performance of Manhattan nonnegative matrix factorization

    IEEE Trans. Neural Netw. Learn. Syst. PP

    (2015)
  • J.C. Yang et al.

    Objective evaluation criteria for stereo camera shooting quality under different shooting parameters and shooting distances

    IEEE Sens. J.

    (2015)
  • D. Choi, M. Cho, Three-dimensional image transmission and reconstruction for multisensor imaging system using...
  • W.L. Hou et al.

    Blind image quality assessment via deep learning

    IEEE Trans. Neural Netw. Learn. Syst.

    (2015)
  • L. Zhang et al.

    VSIa visual saliency-induced index for perceptual image quality assessment

    IEEE Trans. Image Process.

    (2014)
  • X.B. Gao et al.

    Image quality assessment based on multiscale geometric analysis

    IEEE Trans. Image Process.

    (2009)
  • L. He, D. Wang, X. Li, D. Tao, X. Gao, F. Gao, Color fractal structure model for reduced-reference colorful image...
  • L. He, D. Tao, X. Li, X. Gao, Sparse representation for blind image quality assessment, in: 2012 IEEE Conference on...
  • M. Wöpking

    Viewing comfort with stereoscopic picturesan experimental study on the subjective effects of disparity magnitude and depth of focus

    J. Soc. Inf. Disp.

    (1995)
  • W.A. IJsselsteijn et al.

    Subjective evaluation of stereoscopic imageseffects of camera parameters and display duration

    IEEE Trans. Circuits Syst. Video Technol.

    (2000)
  • J. Zhou, G, Jiang, X. Mao, Subjective quality analyses of stereoscopic images in 3DTV system, in: Visual Communications...
  • J.S. Lee et al.

    Paired comparison-based subjective quality assessment of sereoscopic image

    Multim. Tools Appl.

    (2012)
  • H.R. Sheikh et al.

    Image information and visual quality

    IEEE Trans. Image Process.

    (2006)
  • Z. Wang et al.

    Image quality assessment: from error visibility to structural similarity

    IEEE Trans. Image Process.

    (2004)
  • Z. Wang, E.P. Simoncelli, A.C. Bovik, Multi-scale structural similarity for image quality assessment, in: IEEE Asilomar...
  • A. Liu et al.

    Image quality assessment based on gradient similarity

    IEEE Trans. Image Process.

    (2012)
  • X.B. Gao et al.

    Universal blind image quality assessment metrics via natural scene statistics and multiple kernel learning

    IEEE Trans. Neural Netw. Learn. Syst.

    (2013)
  • P. Gorley, N. Holliman, Stereoscopic image quality metrics and compression. In: Proceedings of SPIE, 2008, p....
  • J. You, L. Xing, A. Perkis, X. Wang, Assessment for stereoscopic images based on 2D image quality metrics and disparity...
  • P. Campisi, P. LeCallet, E. Marini, Stereoscopic images quality assessment, in: European Signal Processing Conference,...
  • Cited by (106)

    • Jomodevi: A joint motion and depth visibility prediction algorithm for perceived stereoscopic 3D quality

      2022, Signal Processing: Image Communication
      Citation Excerpt :

      In this section, we review the relevant scientific literature on S3D video quality assessment. The early works in the field of S3D quality assessment [5–11] present solutions that use 2D image quality assessment (IQA) and video quality assessment (VQA) algorithms on the individual views of S3D videos in order to estimate the perceived quality. In these objective metrics, the well-known 2D IQA and VQA models (SSIM [12], MS-SSIM [13], VIF [14], STAMD [15], VQM [16], etc.) are applied on the frame-wise individual views to predict the final quality scores of S3D videos.

    • Multiscale multilevel context and multimodal fusion for RGB-D salient object detection

      2021, Signal Processing
      Citation Excerpt :

      Salient object detection (SOD), in which the aim is to extract predominant objects from a natural scene, has been a fundamental goal in computer vision. It can be used as a preprocessing step for a variety of multimedia applications, such as compression [1,2], segmentation [3,4], retargeting [5], retrieval [6], recognition [7], quality assessment [8–10], object detection [11,12], and photo cropping [13]. Over the last few years, several SOD models for red–green–blue (RGB) images [14–20] or videos [21–24] have been proposed and improved, but challenges related to similar surroundings, complex backgrounds, and varying illumination still hinder the advancement of SOD.

    • Perceptual objective quality assessment of stereoscopic stitched images

      2020, Signal Processing
      Citation Excerpt :

      Compared with 2D images, stereoscopic images deliver an additional dimension of depth information. Currently, stereoscopic image quality assessment approaches [33–35,41–46] have been extensively studied. These methods related to imaging, coding, delivery, display.

    • Perceived quality measurement of stereoscopic 3D images based on sparse representation and binocular combination

      2019, Digital Signal Processing: A Review Journal
      Citation Excerpt :

      In [14], You et al. predicted the perceptual quality of an S3D image by nonlinearly combining 2D quality estimate with disparity quality estimate. ( 3) Recently, studies have exploited deeper understanding of binocular visual perception to devise FR S3D-IQM metrics [15–30]. In [15], Lin et al. presented an FR S3D-IQM framework by considering binocular rivalry.

    View all citing articles on Scopus
    View full text