1.1.

General Approaches to VQA

One reason frame-by-frame IQA performs less well for VQA is because it ignores temporal information, which is important for video quality due to temporal effects, such as temporal masking and motion perception.^3,4 Many researchers have incorporated temporal information into their VQA algorithms by supplementing frame-by-frame IQA with a model of temporal masking and/or temporal weighting.^5–8 For example, in Refs. 6 and 7, motion-weighting and temporal derivatives have been used to extend structural similarity (SSIM)⁹ and visual information fidelity (VIF)¹⁰ for VQA.

Modern VQA algorithms often estimate video quality by extracting and comparing visual/quality features from localized space-time regions or groups of video frames. For example, in Refs. 11 and 12, video quality is estimated based on spatial gradients, color information, and the interaction of contrast and motion from spatiotemporal blocks; motion-based temporal pooling is employed to yield the quality estimate. In Ref. 4, video quality is estimated via measures of spatial quality, temporal quality, and spatiotemporal quality for groups of video frames via a three-dimensional (3-D) Gabor filter-bank; the spatial and temporal components are combined into an overall estimate of quality. In Ref. 13, spatial edge features and motion characteristics in localized space-time regions are used to estimate quality.

Furthermore, it is known that the subjective assessment of video quality is time-varying,¹⁴ and this temporal variation can strongly influence the overall quality ratings.^15,16 Models of VQA that consider these effects have been proposed in Refs. 1617.18. to 19. For example, in Ref. 19, Ninassi et al. measured temporal variations of spatial visual distortions in a short-term pooling for groups of frames through a mechanism of visual attention; the global video quality score is estimated via a long-term pooling. In Ref. 16, Seshadrinathan et al. proposed a hysteresis temporal pooling model of spatial quality values by studying the relation between time-varying quality scores and the final quality score assigned by human subjects.

1.2.

Different Approach for VQA: Analysis of Spatiotemporal Slices

Traditional analyses of temporal variation in VQA tend to formulate methods to compute spatial distortion of a standalone frame,^5,7 of local space-time regions,^12,13 or of groups of adjacent frames^4,19 and then measure the changes of spatial distortion over time. An alternative approach, which is the technique we adopt in this paper, is to use spatiotemporal slices (as illustrated in Fig. 1), which allows one to analyze longer temporal variations.^20,21 In the context of general motion analysis, Ngo et al.²¹ stated that analyzing the visual patterns of spatiotemporal slices could characterize the changes of motion over time and describe the motion trajectories of different moving objects. Inspired by this result, in this paper, we present an algorithm that estimates quality based on the differences between the spatiotemporal slices of the reference and distorted videos.

Fig. 1

A video can be envisaged as a rectangular cuboid in which two of the sides represent the spatial dimensions ($x$ and $y$), and the third side represents the time dimension ($t$). If one takes slices of the cuboid from front-to-back, then the extracted slices correspond to normal video frames. Slicing the cuboid vertically and horizontally yields spatiotemporal slice images (STS images). Examples of three different slice types are presented in part (b) of the figure.

As shown in Fig. 1(a), a video can be envisaged as a rectangular cuboid in which two of the sides represent the spatial dimensions ($x$ and $y$), and the third side represents the time dimension ($t$). If one takes slices of the cuboid from front-to-back, then the extracted slices correspond to normal video frames. However, it is also possible to take the slices of the cuboid from other directions (e.g., from left-to-right or top-to-bottom) to extract images that contain spatiotemporal information, hereafter called the STS images. As shown in Fig. 1(b), if the cuboid is sliced vertically (left-to-right or right-to-left), then the extracted slices represent time along one dimension and vertical space along the other dimension, hereafter called the vertical STS images. If the cuboid is sliced horizontally (top-to-bottom or bottom-to-top), then the extracted slices represent time along one dimension and horizontal space along the other dimension, hereafter called the horizontal STS images.

Figure 2 shows examples of STS images from some typical videos. At one extreme, if the video contains no changes across time (e.g., no motion, as in a static video), then the STS images will contain only horizontal lines [see Fig. 2(a)] or only vertical lines [see Fig. 2(b)]. In both Figs. 2(a) and 2(b), the perfect temporal relationship in the video content manifests as perfect spatial relationship along the dimension that corresponds to time in the STS images. At the other extreme, if the video is rapidly changing (e.g., each frame contains vastly different content), the STS images will appear as random patterns. In both Figs. 2(c) and 2(d), the randomness of temporal content in the video manifests as spatially random pixels along the dimension that corresponds to time in the STS images. The STS images for normal videos [Figs. 2(e) and 2(f)] are generally well structured due to the joint spatiotemporal relationship of neighboring pixels and the smooth frame-to-frame transition.

Fig. 2

Demonstrative STS images extracted from a static video [(a) and (b)], from a video with a vastly different content for each frame [(c) and (d)], and from a typical normal natural video [(e) and (f)]. The STS images for the atypical videos in (a) to (d) appear similar to textures, whereas the STS images for normal videos are generally smoother and more structured due to the joint spatial and temporal (spatiotemporal) relationship.

The STS images have been effectively used in a model of human visual-motion sensing,²² in energy models of motion perception,²³ and in video motion analysis.^20,21 Here, we argue that the temporal variation of spatial distortion is exhibited as spatiotemporal dissimilarity in the STS images, and thus, these STS images can also be used to estimate video quality. To illustrate this, Fig. 3 shows sample STS images from a reference video (reference STS image) and from a distorted video (distorted STS image), where some dissimilar regions are clearly visible in the close-ups. As we will demonstrate, by quantifying the spatiotemporal dissimilarity between the reference and distorted STS images, it is possible to estimate video quality.

Fig. 3

Demonstrative STS images extracted from the reference and distorted videos. The close-ups show some dissimilar regions between the STS images.

Figure 4 shows sample STS images from two distorted videos of the LIVE video database²⁴ and the normalized absolute difference images between the reference and distorted STS images. The associated estimates ${\mathrm{PSNR}}_{\mathrm{sts}}$ and ${\mathrm{MAD}}_{\mathrm{sts}}$ are computed by applying peak SNR (PSNR)²⁵ and the most apparent distortion (MAD) algorithm²⁶ to each pair of the reference and distorted STS images and by averaging the results across all STS images. The higher the ${\mathrm{PSNR}}_{\mathrm{sts}}$ value, the better the video quality; and the lower the ${\mathrm{MAD}}_{\mathrm{sts}}$ value, the better the video quality. As seen from Fig. 4, the ${\mathrm{PSNR}}_{\mathrm{sts}}$ and ${\mathrm{MAD}}_{\mathrm{sts}}$ values show promise for VQA by comparing the STS images, whereas the frame-by-frame MAD fails to predict the qualities of these videos. However, it is important to note that, although PSNR and MAD show promise when applied to the STS images, neither PSNR nor MAD were designed for use with STS images. In particular, PSNR and MAD do not account for the responses of the human visual system (HVS) to temporal changes of spatial distortion. Consequently, ${\mathrm{PSNR}}_{\mathrm{sts}}$ and ${\mathrm{MAD}}_{\mathrm{sts}}$ can yield predictions that correlate poorly with mean opinion score (MOS)/ difference mean opinion score (DMOS). Thus, we propose an alternative method of quantifying degradation of the STS images via a measure of correlation and a model of motion perception.

Fig. 4

Sample STS images and their absolute difference STS images (relative to the STS images of the reference videos) extracted from videos (a) pa2_25fps.yuv and (b) pa8_25fps.yuv for vertical STS images (upper row) and for horizontal STS images (lower row). The videos are from the LIVE video database.²⁴ The values obtained by applying frame-by-frame most apparent distortion (MAD) on normal (front-to-back) frames are shown for comparison. The ${\mathrm{PSNR}}_{\mathrm{sts}}$ and ${\mathrm{MAD}}_{\mathrm{sts}}$ values, which are computed from the STS images, show promise in estimating video quality. However, neither peak SNR nor MAD account for human visual system responses to temporal changes of spatial distortion, and thus we propose an alternative method of quantifying degradation of the STS images.

1.3.

Proposal and Contributions

In this paper, we propose a VQA algorithm that estimates video quality by measuring spatial distortion and spatiotemporal dissimilarity separately. To estimate perceived video quality degradation due to spatial distortion, both the detection-based strategy and the appearance-based strategy of our MAD algorithm are adapted and applied to groups of normal video frames. A simple model of temporal weighting using optical-flow motion estimation is employed to give greater weights to distortions in the slow-moving regions.^5,18 To estimate spatiotemporal dissimilarity, we extend the models of Watson–Ahumada²⁷ and Adelson–Bergen,²³ which have been used to measure energy of motion in videos, to the STS images and measure the local variance of spatiotemporal neural responses. The spatiotemporal response is measured by filtering the STS image via one one-dimensional (1-D) spatial filter and one 1-D temporal filter.^23,27 The overall estimate of perceived video quality degradation is given by a geometric mean of the spatial distortion and spatiotemporal dissimilarity values.

We have named our algorithm ${\mathrm{ViS}}_3$ according to its two main stages: the first stage estimates video quality degradation based on spatial distortion (${\mathrm{ViS}}_1$), and the second stage estimates video quality degradation based on the dissimilarity between spatiotemporal slice images (${\mathrm{ViS}}_2$). The final estimate of perceived video quality degradation ${\mathrm{ViS}}_3$ is a combination of ${\mathrm{ViS}}_1$ and ${\mathrm{ViS}}_2$. The ${\mathrm{ViS}}_3$ algorithm is an improved and extended version of our previous VQA algorithms presented in Refs. 28 and 29. We demonstrate the performance of this algorithm on various video-quality databases and compare to some recent VQA algorithms. We also analyze the performance of ${\mathrm{ViS}}_3$ on different types of distortion by measuring its performance on each subset of videos.

The major contributions of this paper are as follows. First, we provide a simple yet effective extension of our MAD algorithm for use in VQA. Specifically, we show how to apply MAD’s detection- and appearance-based strategies to groups of video frames and how to modify the combination to take into account temporal masking. This contribution is presented in the first stage of the ${\mathrm{ViS}}_3$ algorithm. Second, we demonstrate that the spatiotemporal dissimilarity exhibited in the STS images can be used to effectively estimate video quality degradation. We specifically provide in the second stage of the ${\mathrm{ViS}}_3$ algorithm a technique to quantify the spatiotemporal dissimilarity by measuring spatiotemporal correlation and by applying an HVS-based model to the STS images. Finally, we demonstrate that a combination of the measurements obtained from these two stages is able to estimate video quality quite accurately.

This paper is organized as follows. In Sec. 2, we provide a brief review of current VQA algorithms. In Sec. 3, we describe details of the ${\mathrm{ViS}}_3$ algorithm. In Sec. 4, we present and compare the results of applying ${\mathrm{ViS}}_3$ to different video databases. General conclusions are presented in Sec. 5.

2.1.

Frame-by-Frame IQA

As stated in Sec. 1, the most straightforward technique to estimate video quality is to apply existing IQA algorithms on a frame-by-frame basis. These per-frame quality estimates can then be collapsed across time to predict an overall quality estimate of the video. It is common to find these frame-by-frame IQA algorithms used as a baseline for comparison,^24,31 and some authors implement this technique as a part of their VQA algorithms.^32,33 However, due to the lack of temporal information, this technique often fails to correlate with the perceived quality measurements obtained from human observers.

2.2.

Algorithms Based on Visual Features

An approach commonly used in VQA is to extract spatial and temporal visual features of the videos and then estimate quality based on the changes of these features between the reference and distorted videos.^{11,12,34–40}

One of the earliest approaches to feature-based VQA was proposed by Pessoa et al.³⁴ Their VQA algorithm employs segmentation along with segment-type-specific error measures. Frames of the reference and distorted videos are first segmented into smooth, edge, and texture segments. Various pixel-based and edge-detection-based error measures are then computed between corresponding regions of the reference and distorted videos for both the luminance and chrominance components. The overall estimate of quality is computed via a weighted linear combination of logistic-normalized versions of these error measures, using segment-category-specific weights, collapsed across all segments and all frames.

One of the most popular feature-based VQA algorithms, called the video quality metric (VQM), was developed by Pinson and Wolf.^11,12 The VQM algorithm employs quality features that capture spatial, temporal, and color-based differences between the reference and distorted videos. The VQM algorithm consists of four sequential steps. The first step calibrates videos in terms of brightness, contrast, and spatial and temporal shifts. The second step breaks the videos into subregions of space and time, and then extracts a set of quality features for each subregion. The third step compares features extracted from the reference and distorted videos to yield a set of quality indicators. The last step combines these indicators into a video quality index.

Okamoto et al.³⁵ proposed a VQA algorithm that operates based on the distortion of edges in both space and time. Okamoto et al. employ three general features: (1) blurring in edge regions, which is quantified by using the average edge energy difference described in ANSI T1.801.03; (2) blocking artifacts, which are quantified based on the ratio of horizontal and vertical edge distortions to other edge distortions; and (3) the average local motion distortion, which is quantified based on the average difference between block-based motion measures of the reference and distorted frames. The overall video quality is estimated via a weighted average of these three features.

In Ref. 36, Lee and Sim propose a VQA algorithm that operates under the assumption that visual sensitivity is greatest near edges and block boundaries. Accordingly, their algorithm applies both an edge-detection stage and a block-boundary detection stage to frames from the reference video to locate these regions. Separate measures of distortion for the edge regions and block regions are then computed between the reference and distorted frames. These two features are supplemented with a gradient-based distortion measure, and the overall estimate of quality is then obtained via a weighted linear sum of these three features averaged across all frames.

In the context of packet-loss scenarios, Barkowsky et al.³⁷ designed the TetraVQM algorithm by adding a model of temporal distortion awareness to the VQM algorithm. The key idea in TetraVQM is to estimate the temporal visibility of image areas and, therefore, weight the degradations in these areas based on their durations. TetraVQM employs block-based motion estimation to track image objects over time. The resulting motion vectors and motion-prediction errors are then used to estimate the temporal visibility, and this information is used to supplement VQM for estimating the overall quality. In Ref. 39, Engelke et al. demonstrated that significant improvements to VQM and TetraVQM can be realized by augmenting these techniques with information regarding visual saliency.

Various features have also been combined via machine-learning for improved VQA. In Ref. 8, Narwaria et al. proposed the temporal quality variation (TQV) algorithm, a low-complexity VQA algorithm that employs a machine-learning mechanism to determine the impact of the spatial and temporal factors as well as their interactions on the overall video quality. Spatial quality factors are estimated by a singular value decomposition (SVD)-based algorithm,⁴¹ and the temporal variation of spatial quality factors is used as a feature to estimate video quality.

2.3.

Algorithms Based on Statistical Measurements

Another class of VQA algorithms has been proposed that estimate quality based on differences in statistical features of the reference and distorted videos.^5–7

In Ref. 5, Wang et al. proposed the video structural similarity (VSSIM) index. VSSIM computes various SSIM⁹ indices at three different levels: the local region level, the frame level, and the video sequence level. In the local region level, the SSIM index of each region is computed for the luminance and chrominance components, with greater weight given to luminance component. These SSIM indices are weighted by local luminance intensity to yield the frame-level SSIM index. Finally, at the sequence level, the frame SSIM index is weighted by global motion to yield an estimate of video quality.

Another extension of SSIM to VQA, called speed SSIM, was also proposed by Wang and Li.⁶ There, they augmented SSIM⁹ with an additional stage that employs Stocker and Simoncelli’s statistical model⁴² of visual speed perception. The speed perception model is used to derive a spatiotemporal importance weight function, which specifies a relative weighting at each spatial location and time instant. The overall estimate of video quality is obtained by using this weight function to compute a weighted average of SSIM over all space and time.

In Ref. 7, Sheikh and Bovik augmented the VIF IQA algorithm¹⁰ for use in VQA. VIF estimates quality based on the amount of information that the distorted image provides about the reference image. VIF models images as realizations of a mixture of marginal Gaussian densities of wavelet subbands, and quality is then determined based on the mutual information between the subband coefficients of the reference and distorted images. To account for motion, V-VIF quantifies loss in motion information by measuring deviations in the spatiotemporal derivatives of the videos, the latter of which are estimated by using separable bandpass filters in space and time.

Tao and Eskicioglu³³ proposed a VQA algorithm that estimates quality based on SVD. Each frame of the reference and distorted videos are divided into $8\times 8$ blocks, and then the SVD is applied to each block. Differences in the SVDs of corresponding blocks of the reference and distorted frames, weighted by the edge-strength in each block, are used to generate a frame-level distortion estimate. Both luminance and chrominance SVD-based distortions are combined via a weighted sum. These combined frame-level estimates are then averaged across all frames to derive an overall estimate of video quality.

Peng et al. proposed a motion-tuned and attention-guided VQA algorithm based on a space-time statistical texture representation of motion. To construct the spacetime texture representation, the reference and distorted videos are filtered via a bank of 3-D Gaussian derivative filters at multiple scales and orientations. Differences in the energies within local regions of the filtered outputs between the reference and distorted videos are then computed along 13 different planes in space-time to define their temporal distortion measure. This temporal distortion measure is then combined with a model of visual saliency and multiscale SSIM⁴³ (averaged across frames) to estimate quality.

2.4.

Algorithms Based on Models of Human Vision

Another widely adopted approach to VQA is to estimate video quality via the use of various models of the HVS.^4,44–,55

One of the earliest VQA algorithms based on a vision model was developed by Lukas and Budrikis.⁴⁴ Their technique employs a spatiotemporal visual filter that models visual threshold characteristics on uniform backgrounds. To account for nonuniform backgrounds, the model is supplemented with a masking function based on the spatial and temporal activities of the video.

The digital video quality algorithm, developed by Watson et al.,⁴⁹ also models visual thresholds to estimate video quality. The authors employ the concept of just noticeable differences (JNDs), which are computed via a discrete cosine transform (DCT)-based model of early vision. After sampling, cropping, and color conversion, each $8\times 8$ block of the videos is transformed to DCT coefficients, converted to local contrast, and filtered by a model of the temporal contrast sensitivity function. JNDs are then measured by dividing each DCT coefficient by its respective visual threshold. Contrast masking is estimated based on the differences between successive frames, and the masking-adjusted differences are pooled and mapped to a visual quality estimate.

Other HVS-based approaches to VQA have employed various subband decompositions to model the spatiotemporal response properties of populations of visual neurons, which are assumed to underlie the multichannel nature of the HVS.^{4,45–47,53,55} These algorithms generally compute simulated neural responses to the reference and distorted videos and then estimate quality based on the extent to which these responses differ.

The moving picture quality metric algorithm, proposed by Basso et al.,⁴⁵ employs a spatiotemporal multichannel HVS model by using 17 spatial Gabor filters and two temporal filters on the luminance component. After contrast sensitivity and masking adjustments, distortion is measured within each subband and pooled to yield the quality estimate. The color MPQM algorithm, proposed by Lambrecht,⁴⁶ extends and applies the MPQM algorithm to both luminance and chrominance components with a reduced number of filters for the chrominance components (nine spatial filters and one temporal filter).

The normalization video fidelity metric algorithm, proposed by Lindh and Lambrecht,⁴⁷ implements a visibility prediction model based on the Teo–Heeger gain-control model.⁵⁶ Instead of using Gabor filters, the multichannel decomposition is performed by using the steerable pyramid with four scales and four orientations. An excitatory-inhibitory stage and a pooling stage are performed to yield a map of normalized responses. The distortion is measured based on the squared error between normalized response maps generated for the reference and the distorted videos.

Masry et al.⁵³ developed a VQA algorith that employs a multichannel decomposition and a masking model implemented via a separable wavelet transform. A training step was performed on a set of videos and associated subjective quality scores to obtain the masking parameters. Later in Ref. 55, Li et al. utilized this algorithm as part of a VQA algorithm that measures and combines detail losses and additive impairments within each frame; optimal parameters were determined by training the algorithm on a subset of the LIVE video database.²⁴

Seshadrinathan and Bovik⁴ proposed the motion-based video integrity evaluation (MOVIE) algorithm that estimates spatial quality, temporal quality, and spatiotemporal quality via a 3-D subband decomposition. MOVIE decomposes both the reference and distorted videos by using a 3-D Gabor filter-bank with 105 spatiotemporal subbands. The spatial component of MOVIE uses the outputs of the spatiotemporal Gabor filters and a model of contrast masking to capture spatial distortion. The temporal component of MOVIE employs optical-flow motion estimation to determine motion information, which is combined with the outputs of the spatiotemporal Gabor filters to capture temporal distortion. These spatial and temporal components are combined into an overall estimate of video quality.

2.5.

Summary

In summary, although previous VQA algorithms have analyzed the effects of spatial and temporal interactions on video quality, none have estimated video quality based on spatiotemporal slices (STS images), which contain important spatiotemporal information on a longer time scale. Earlier related work was performed by Péchard et al.,⁵⁷ where spatiotemporal tubes rather than slices were used for VQA. Their algorithm employs a segmentation to create spatiotemporal tubes, which are coherent in terms of motion and spatial activity. Similar to our STS images, the spatiotemporal tubes permit analysis of spatiotemporal information on a long time scale, and Pechard et al. demonstrated the superiority of their approach compared to other VQA algorithms on videos containing H.264 artifacts.

In the following section, we describe our HVS-based VQA algorithm, ${\mathrm{ViS}}_3$, which employs measures of both motion-weighted spatial distortion and spatiotemporal dissimilarity of the STS images to estimate perceived video quality degradation.

3.1.

Spatial Distortion

In the spatial distortion stage, we employ and extend our MAD algorithm,²⁶ which was designed for still images, to measure spatial distortion in each GOF of the video. The MAD algorithm is composed of two separate strategies: (1) a detection-based strategy, which computes the perceived distortion due to visual detection (denoted by $d_{\text{detect}}$) and (2) an appearance-based strategy, which computes the perceived distortion due to visual appearance changes (denoted by $d_{\text{appear}}$). The perceived distortion due to visual detection is measured by using a masking-weighted block-based mean-squared error in the lightness domain. The perceived distortion due to visual appearance changes is measured by computing the average differences between the block-based log-Gabor statistics of the reference and distorted images.

The MAD index of the distorted image is computed via a geometric weighted mean.

To extend MAD for use with video, we take the $Y$ components of the videos and perform the following steps (shown in Fig. 6) on each group of $N$ consecutive frames:

Fig. 6

Block diagram of the spatial distortion stage. The extracted frames from the reference and distorted videos are used to compute a visible distortion map and a statistical difference map of each group of frames (GOF). Motion estimation is performed on the reference video frames and used to model the effect of motion on the visibility of distortion. All maps are combined and collapsed to yield a spatial distortion value ${\mathrm{ViS}}_1$.

The video frames are extracted from the $Y$ components of the reference and distorted videos. Let $I_t(x,y)$ denote the $t$’th frame of the reference video $\mathbf{I}$, and let ${\widehat{I}}_t(x,y)$ denote the $t$’th frame of the distorted video $\widehat{\mathbf{I}}$, where $t\in [1,T]$ denotes the frame (time) index, and $T$ denotes the number of frames in video $\mathbf{I}$. These video frames are then divided into groups of $N$ consecutive frames for both the reference and the distorted video. The following subsections describe the details of each step.

3.1.1.

Compute visible distortion map

We apply the detection-based strategy from Ref. 26 to all pairs of respective frames from the reference video and the distorted video. A block diagram of this detection-based strategy is provided in Fig. 7.

Fig. 7

Block diagram of the detection-based strategy used to compute a visible distortion map. Both the reference and the distorted frames are converted to perceived luminance and filtered by a contrast sensitivity function. By comparing the local contrast of the reference frame $L$ and the error frame $\mathrm{\Delta }L$, we obtain a local distortion visibility map. This map is then weighted by local mean squared error to yield a visible distortion map.

Detection-based strategy

As illustrated in Fig. 7, a preprocessing step is first performed by using the nonlinear luminance conversion and spatial contrast sensitivity function filtering. Then, models of luminance and contrast masking are used to compute a local distortion visibility map. Next, this map is weighted by local mean squared error (MSE) to yield a visible distortion map. The specific steps are given below (see Ref. 26 for additional details).

First, to account for the nonlinear relationship between digital pixel values and physical luminance of typical display media, the video $\mathbf{I}$ is converted to a perceived luminance video $\mathbf{L}$ via

Eq. (3)

$$\mathbf{L}={(a+k\mathbf{I})}^{\gamma /3},$$

where the parameters $a$, $k$, and $\gamma$ are constants specific to the device on which the video is displayed. For 8-bit pixel values and an sRGB display, these parameters are given by $a=0$, $k=0.02874$, and $\gamma =2.2$. The division by 3 attempts to take into account the nonlinear HVS response to luminance by converting luminance into perceived luminance (relative lightness).

Next, the contrast sensitivity function (CSF) is applied by filtering both the reference frame $L$ and the error frame $\mathrm{\Delta }L=L-\widehat{L}$. The filtering is performed in the frequency domain via

Eq. (4)

$$\tilde{L}={\mathbb{F}}^{-1}[H(u,v)\times \mathbb{F}[L]],$$

where $\mathbb{F}$ and ${\mathbb{F}}^{-1}$ denote the discrete fourier transform (DFT) and inverse DFT, respectively; $H(u,v)$ is the DFT-based version of the CSF function defined by Eq. (3) in Ref. 26.

To account for the fact that the presence of an image can reduce the detectability of distortions, MAD employs a simple spatial-domain measure of contrast masking.

First, a local contrast map is computed for the reference frame in the lightness domain by dividing $\tilde{L}$ into $16\times 16$ blocks (with 75% overlap between neighboring blocks) and then measuring the RMS contrast of each block. The RMS contrast of block $b$ of $\tilde{L}$ is computed via

Eq. (5)

$$C_{\mathrm{ref}}(b)={\tilde{\sigma }}_{\mathrm{ref}}(b)/{\mu }_{\mathrm{ref}}(b),$$

where ${\mu }_{\mathrm{ref}}(b)$ denotes the mean of block $b$ of $\tilde{L}$, and ${\tilde{\sigma }}_{\mathrm{ref}}(b)$ denotes the minimum of the standard deviations of the four $8\times 8$ subblocks of $b$. The block size of $16\times 16$ was chosen because it is large enough to accommodate division into reasonably sized subblocks (to avoid overestimating the contrast around edges), but small enough to yield decent spatial localization (see Appendix A in Ref. 26).

$C_{\mathrm{ref}}(b)$ is a measure of the local RMS contrast in the reference frame and is thus independent of the distortions. Accordingly, we next compute a local contrast map for the error frame to account for the spatial distribution of the distortions in the distorted frame. The error frame $\mathrm{\Delta }L$ is divided into $16\times 16$ blocks (with 75% overlap between blocks), and then the RMS contrast $C_{\mathrm{err}}(b)$ for each block $b$ is computed via

Eq. (6)

$$C_{\mathrm{err}}(b)=\{\begin{array}{ll}{\sigma }_{\mathrm{err}}(b)/{\mu }_{\mathrm{ref}}(b)& \mathrm{if}{\mu }_{\mathrm{ref}}(b)>0.5\\ 0& \text{otherwise},\end{array}$$

where ${\sigma }_{\mathrm{err}}(b)$ denotes the standard deviation of block $b$ of $\mathrm{\Delta }L$. A lightness threshold of 0.5 is employed to account for the fact that the HVS is relatively insensitive to changes in extremely dark regions.

The local contrast maps are computed for both the reference frame and the error frame for every block $b$ of size $16\times 16$ with 75% overlap between neighboring blocks. The two local contrast maps $\{C_{\mathrm{ref}}\}$ and $\{C_{\mathrm{err}}\}$ are used to compute a local distortion visibility map denoted by $\xi (b)$ via

Eq. (7)

$$\xi (b)=\{\begin{array}{ll}\ln [C_{\mathrm{err}}(b)]-\ln [C_{\mathrm{ref}}(b)]& \mathrm{if}\ln [C_{\mathrm{err}}(b)]>\ln [C_{\mathrm{ref}}(b)]>-5\\ \ln [C_{\mathrm{err}}(b)]+5& \mathrm{if}\ln [C_{\mathrm{err}}(b)]>-5\ge \ln [C_{\mathrm{ref}}(b)]\\ 0& \text{otherwise}\end{array}.$$

The local distortion visibility map $\xi$ is then point-by-point multiplied by the local MSE to determine a visible distortion map denoted by ${\mathrm{\Upsilon }}^{\mathrm{D}}$, where the superscript $\mathrm{D}$ is used to imply that the map is computed from the detection-based strategy. The visible distortion at the location of block $b$ is given by

Eq. (8)

$${\mathrm{\Upsilon }}^{\mathrm{D}}(b)=\xi (b)·\mathrm{MSE}(b).$$

Note that in Ref. 26, the visible distortion map ${\mathrm{\Upsilon }}^{\mathrm{D}}$ is collapsed into a single scalar that represents the perceived distortion due to visual detection $d_{\text{detect}}$, which is computed via $d_{\text{detect}}=\sqrt{\sum _b{[{\mathrm{\Upsilon }}^{\mathrm{D}}(b)]}^2}$, where the summation is over all blocks. In the current paper, we do not collapse ${\mathrm{\Upsilon }}^{\mathrm{D}}$.

Apply to groups of video frames

Let ${\mathrm{\Upsilon }}_t^{\mathrm{D}}$ denote the visible distortion map computed from the $t$’th frame of the reference video and the $t$’th frame of the distorted video. The visible distortion maps computed from all frames in the $k$’th GOF will be $\{{\mathrm{\Upsilon }}_{N(k-1)+1}^{\mathrm{D}},{\mathrm{\Upsilon }}_{N(k-1)+2}^{\mathrm{D}},\dots ,{\mathrm{\Upsilon }}_{Nk}^{\mathrm{D}}\}$, where $k\in \{1,2,\dots ,K\}$ is the GOF index and $K$ is the number of GOFs in the video. These maps are combined via a point-by-point average to yield a GOF-based visible distortion map of the $k$’th GOF, which is denoted by ${\overline{\mathrm{\Upsilon }}}_k^{\mathrm{D}}$.

Eq. (9)

$${\overline{\mathrm{\Upsilon }}}_k^{\mathrm{D}}=\frac{1}{N}\sum _{\tau =1}^N{\mathrm{\Upsilon }}_{N(k-1)+\tau }^{\mathrm{D}}.$$

3.1.2.

Compute statistical difference map

As argued in Ref. 26, when the distortions in the image are highly suprathreshold, perceived distortion is better modeled by quantifying the extent to which the distortions degrade the appearance of the image’s subject matter. The appearance-based strategy measures local statistics of multiscale log-Gabor filter responses to capture changes in visual appearance. Figure 8 shows a block diagram of the appearance-based strategy used to compute a statistical difference map between the reference and the distorted frame.

Fig. 8

Block diagram of the appearance-based strategy used to compute a statistical difference map. The reference and the distorted frames are decomposed into different subbands using a two-dimensional log-Gabor filter-bank. Local standard deviation, skewness, and kurtosis are computed for each subband of both the reference and the distorted frames. The differences of local standard deviation, skewness, and kurtosis between each subband of the reference frame and the respective subband of the distorted frame are combined into a statistical difference map.

Appearance-based strategy

The appearance-based strategy employs a computational neural model using a log-Gabor filter-bank (with five scales $s\in \{1,2,3,4,5\}$ and four orientations $o\in \{1,2,3,4\}$), which implements both even-symmetric (cosine-phase) and odd-symmetric (sine-phase) filters. The even and odd filter outputs are then combined to yield magnitude-only subband values. Let $\{R^{s,o}\}$ and $\{{\widehat{R}}^{s,o}\}$ denote the sets of log-Gabor subbands computed for a reference and a distorted frame, respectively, where each subband is the same size as the frames.

The standard deviation, skewness, and kurtosis are then computed for each block $b$ of size $16\times 16$ (with 75% overlap between blocks) for each log-Gabor subband of the reference frame and the distorted frame. Let ${\sigma }^{s,o}(b)$, ${\varsigma }^{s,o}(b)$, and ${\kappa }^{s,o}(b)$ denote the standard deviation, skewness, and kurtosis computed from block $b$ of subband $R^{s,o}$. Let ${\widehat{\sigma }}^{s,o}(b)$, ${\widehat{\varsigma }}^{s,o}(b)$, and ${\widehat{\kappa }}^{s,o}(b)$ denote the standard deviation, skewness, and kurtosis computed from block $b$ of subband ${\widehat{R}}^{s,o}$. The statistical difference map is computed as the weighted combination of the differences in standard deviation, skewness, and kurtosis for all subbands. We denote ${\mathrm{{\rm Y}}}^{\mathrm{A}}$ as the statistical difference map, where the superscript $\mathrm{A}$ is used to imply that the map is computed from the appearance-based strategy. Specifically, the statistical difference at the location of block $b$ is given by

Eq. (10)

$${\mathrm{\Upsilon }}^{\mathrm{A}}(b)=\sum _{s=1}^5\sum _{o=1}^4{w}_s[|{\sigma }^{s,o}(b)-{\widehat{\sigma }}^{s,o}(b)|+2|{\varsigma }^{s,o}(b)-{\widehat{\varsigma }}^{s,o}(b)|+|{\kappa }^{s,o}(b)-{\widehat{\kappa }}^{s,o}(b)|],$$

where the scale-specific weights $w_s=\{0.5,0.75,1,5,6\}$ (for the finest to coarsest scales, respectively) are chosen the same as in Ref. 26 to account for the HVS’s preference for coarse scales over fine scales (see Ref. 26 for more details).

Note that in Ref. 26, the statistical difference map ${\mathrm{\Upsilon }}^{\mathrm{A}}$ is collapsed into a single scalar that represents the perceived distortion due to visual appearance changes $d_{\text{appear}}$, which is computed via $d_{\text{appear}}=\sqrt{\sum _b{[{\mathrm{\Upsilon }}^{\mathrm{A}}(b)]}^2}$, where the summation is over all blocks. In the current paper, we do not collapse ${\mathrm{\Upsilon }}^{\mathrm{A}}$.

Apply to groups of video frames

Let ${\mathrm{{\rm Y}}}_t^{\mathrm{A}}$ denote the statistical difference map computed from the $t$’th frame of the reference video and the $t$’th frame of the distorted video. The statistical difference maps computed from all frames in the $k$’th GOF will be $\{{\mathrm{\Upsilon }}_{N(k-1)+1}^{\mathrm{A}},{\mathrm{\Upsilon }}_{N(k-1)+2}^{\mathrm{A}},\dots ,{\mathrm{\Upsilon }}_{Nk}^{\mathrm{A}}\}$, where $k\in \{1,2,\dots ,K\}$ is the GOF index and $K$ is the number of GOFs in the video. These maps are combined via a point-by-point average to yield a GOF-based statistical difference map of the $k$’th GOF, which is denoted by ${\overline{\mathrm{\Upsilon }}}_k^{\mathrm{A}}$.

Eq. (11)

$${\overline{\mathrm{\Upsilon }}}_k^{\mathrm{A}}=\frac{1}{N}\sum _{\tau =1}^N{\mathrm{\Upsilon }}_{N(k-1)+\tau }^{\mathrm{A}}.$$

3.1.4.

Combine maps and compute spatial distortion value

For each GOF, we have computed the GOF-based visible distortion map ${\overline{\mathrm{\Upsilon }}}^{\mathrm{D}}$, the GOF-based statistical difference map ${\overline{\mathrm{\Upsilon }}}^{\mathrm{A}}$, and the GOF-based motion magnitude map ${\overline{\mathrm{\Upsilon }}}^{\mathrm{M}}$. Now, we extend and apply Eq. (2) to respective regions of the visible distortion map and the statistical difference map to obtain the GOF-based most apparent distortion map. This map is then point-by-point weighted by the motion magnitude map ${\overline{\mathrm{\Upsilon }}}_k^{\mathrm{M}}$ to yield the spatial distortion map of the $k$’th GOF. We denote ${\mathrm{\Delta }}_k(x,y)$ of size $W\times H$, the video frame size, as the spatial distortion map of the $k$’th GOF. Specifically, the value at $(x,y)$ of the spatial distortion map ${\mathrm{\Delta }}_k(x,y)$ is computed via

Eq. (13)

$$\widehat{\alpha }(x,y)=\frac{1}{1+{\beta }_1\times {[{\overline{\mathrm{\Upsilon }}}_k^{\mathrm{D}}(x,y)]}^{{\beta }_2}},$$

Eq. (14)

$${\mathrm{\Delta }}_k(x,y)=\frac{{[{\overline{\mathrm{\Upsilon }}}_k^{\mathrm{D}}(x,y)]}^{\widehat{\alpha }(x,y)}\times {[{\overline{\mathrm{\Upsilon }}}_k^{\mathrm{A}}(x,y)]}^{1-\widehat{\alpha }(x,y)}}{\sqrt{1+{\overline{\mathrm{\Upsilon }}}_k^{\mathrm{M}}(x,y)}}.$$

The division by ${\overline{\mathrm{\Upsilon }}}_k^{\mathrm{M}}(x,y)$ accounts for the fact that the distortion in slow-moving regions is generally more visible than the distortion in fast-moving regions. When the value in the motion magnitude map ${\overline{\mathrm{\Upsilon }}}_k^{\mathrm{M}}$ is relatively large or the corresponding spatial region is fast-moving, the visible distortion value in ${\mathrm{\Delta }}_k(x,y)$ is relatively small; when the value in the motion magnitude map ${\overline{\mathrm{\Upsilon }}}_k^{\mathrm{M}}$ is relatively small or the corresponding spatial region is slow-moving, the visible distortion value in ${\mathrm{\Delta }}_k(x,y)$ is relatively large. When there is no motion in the region, the visible distortion is determined solely by ${\overline{\mathrm{\Upsilon }}}_k^{\mathrm{D}}$ and ${\overline{\mathrm{\Upsilon }}}_k^{\mathrm{A}}$.

Figure 9 shows examples of the first frame (a) and the last frame (b) of a specific GOF of video mc2_50fps.yuv from the LIVE video database.²⁴ The visible distortion map (c), the statistical difference map (d), the motion magnitude map (e), and the spatial distortion map (f) computed for this GOF are also shown. As seen from the visible distortion map (c) and the statistical difference map (d), at the regions of high visible distortion level (i.e., the train, the numbers in the calendar), the spatial distortion map is weighted more by the statistical difference map. At the regions of low visible distortion level (i.e., the wall background), the spatial distortion map is weighted more by the visible distortion map.

Fig. 9

Examples of the first and last frames [(a) and (b)], the visible distortion map (c), the statistical difference map (d), the motion magnitude map (e), and the spatial distortion map (f) computed for a specific GOF of the video mc2_50fps.yuv from the LIVE video database.²⁴ All maps have been normalized in contrast to promote visibility. Note that the brighter the maps, the more distorted the corresponding spatial region of the GOF; for the motion magnitude map, the brighter the map, the faster the motion in the corresponding spatial region of the GOF.

As also seen from Figs. 9(c) and 9(d), the region corresponding to the train at the bottom of the frames is more heavily distorted than the other regions. However, due to the fast movement of the train, which is reflected in the bottom of the motion magnitude map (e), the visibility of distortion is reduced, making this region less bright in the spatial distortion map (f).

To estimate spatial distortion value of each GOF, we compute the RMS value of the spatial distortion map. The RMS value of the map ${\mathrm{\Delta }}_k(x,y)$ of size $W\times H$ is given by

Eq. (15)

$${\overline{\mathrm{\Delta }}}_k^{\mathrm{XY}}=\sqrt{\frac{1}{W\times H}\sum _{x=1}^W\sum _{y=1}^H{[{\mathrm{\Delta }}_k(x,y)]}^2},$$

where the superscript $\mathrm{XY}$ is used to remind readers that the value is computed from the normal frames with two dimensions $x$ and $y$. The overall perceived spatial distortion value, denoted by ${\mathrm{ViS}}_1$, is computed as the arithmetic mean of all spatial distortion values ${\overline{\mathrm{\Delta }}}_k^{\mathrm{XY}}$ via

Eq. (16)

$${\mathrm{ViS}}_1=\frac{1}{K}\sum _{k=1}^K{\overline{\mathrm{\Delta }}}_k^{\mathrm{XY}}.$$

Here, ${\mathrm{ViS}}_1$ is a single scalar that represents the overall perceived quality degradation of the video due to spatial distortion. The lower the ${\mathrm{ViS}}_1$ value, the better the video quality. A value ${\mathrm{ViS}}_1=0$ indicates that the distorted video is equal in quality to the reference video.

3.2.

Spatiotemporal Dissimilarity

In the distorted video, the distortion impacts not only the spatial relationship between neighboring pixels within the current frame, but also the transition between frames, which can be captured via the use of STS images. The difference between the STS images from the reference and distorted videos is referred to as the spatiotemporal dissimilarity in this paper. If the spatiotemporal dissimilarity between the STS images is small, the distorted video has high quality relative to the reference video; if the spatiotemporal dissimilarity between the STS images is large, the distorted video has low quality relative to the reference video. Figure 10 depicts a block diagram of the spatiotemporal dissimilarity stage, which estimates the spatiotemporal dissimilarity between the reference and the distorted video via the following steps:

Fig. 10

Block diagram of the spatiotemporal dissimilarity stage of the ${\mathrm{ViS}}_3$ algorithm. The STS images are extracted from the perceived luminance videos. The spatiotemporal correlation and the difference of spatiotemporal responses are computed in a block-based fashion and combined to yield a spatiotemporal dissimilarity map. All maps are then collapsed by using root mean square and combined to yield the spatiotemporal dissimilarity value ${\mathrm{ViS}}_2$ of the distorted video.

The following subsections describe the details of each step.

3.2.2.

Compute spatiotemporal correlation map

One simple way to measure the spatiotemporal dissimilarity is by using the local linear correlation coefficients of the STS images extracted from the reference and the distorted videos. If the distorted video has perfect quality relative to the reference video, these two videos should have high correlation in the STS images; if the distorted video has low quality relative to the reference video, the spatiotemporal correlation will be low.

Let $\rho (b)$ denote the linear correlation coefficient computed from block $b$ of the two STS images $S_x(t,y)$ and ${\widehat{S}}_x(t,y)$. We define the local spatiotemporal correlation coefficient $\tilde{\rho }(b)$ of these two blocks as

Eq. (17)

$$\tilde{\rho }(b)=\{\begin{array}{ll}0& \mathrm{if}\rho (\mathrm{b})<0\\ 1& \mathrm{if}\rho (\mathrm{b})>0.9\\ \rho (b)& \text{otherwise}\end{array}.$$

As shown in Eq. (17), if the two blocks are highly positively correlated, we set $\tilde{\rho }(b)=1$. The threshold value of 0.9 was chosen empirically so that a relatively high positive correlation ($\rho >0.9$) is still considered perfect by the algorithm. As we demonstrate in the online supplement to this paper,⁶⁰ the performance of the algorithm is relatively robust to small changes in this threshold value. On the other hand, if the two blocks are negatively correlated, we set $\tilde{\rho }(b)=0$ to reflect the dissimilarity between the two blocks.

This process is performed on every block of size $16\times 16$ with 75% overlap between neighboring blocks, yielding a spatiotemporal correlation map denoted by $P_x(t,y)$ between $S_x(t,y)$ and ${\widehat{S}}_x(t,y)$. Similarly, we compute a spatiotemporal correlation map denoted by $P_y(x,t)$ between $S_y(x,t)$ and ${\widehat{S}}_y(x,t)$. Examples of the correlation maps are shown in Fig. 11(c). The brighter the maps, the higher the spatiotemporal correlation between corresponding regions of the two STS images.

Fig. 11

Demonstrative maps for two pairs of STS images $S_y(x,t)$ and ${\widehat{S}}_y(x,t)$ from videos mc2_50fps.yuv (LIVE) and PartyScene_dst_09.yuv (CSIQ) with the correlation maps $P_y(x,t)$, the log of response difference maps $D_y(x,t)$, and spatiotemporal dissimilarity maps ${\mathrm{\Delta }}_y(x,t)$. All maps have been normalized to promote visibility. Note that the brighter the spatiotemporal dissimilarity maps ${\mathrm{\Delta }}_y(x,t)$, the more dissimilar the corresponding regions in the STS images.

3.3.

Combine Spatial Distortion and Spatiotemporal Dissimilarity Values

Finally, the overall estimate of perceived video quality degradation, denoted by ${\mathrm{ViS}}_3$, is computed from the spatial distortion value ${\mathrm{ViS}}_1$ and the spatiotemporal dissimilarity value ${\mathrm{ViS}}_2$. Specifically, ${\mathrm{ViS}}_3$ is computed as a geometric mean of ${\mathrm{ViS}}_1$ and ${\mathrm{ViS}}_2$, which is given by

Here, ${\mathrm{ViS}}_3$ is a single scalar that represents the overall perceived quality degradation of the video. The smaller the ${\mathrm{ViS}}_3$ value, the better the video quality. A value of ${\mathrm{ViS}}_3=0$ indicates that the distorted video is equal in quality to the reference video.

Note that the values of ${\mathrm{ViS}}_1$ and ${\mathrm{ViS}}_2$ occupy different ranges. Thus, the use of a geometric mean in Eq. (33) allows us to combine these values without the need for custom weights (which would be required when using an arithmetic mean). Other combinations are also possible, e.g., using a weighted geometric mean with possibly adaptive weights. However, our preliminary attempts to select such weights have not yielded significant improvements (see also Sec. 4.3.4).

4.1.

Video Quality Databases

To evaluate the performance of ${\mathrm{ViS}}_3$ and other quality assessment algorithms, we used the following three publicly available video-quality databases that have multiple types of distortion:

4.2.

Algorithms and Performance Measures

We compared ${\mathrm{ViS}}_3$ with PSNR²⁵ and recent full-reference video quality assessment algorithms for which code is publicly available, VQM,¹² MOVIE,⁴ and TQV,⁸ on the three video databases. PSNR was applied on a frame-by-frame basis, VQM and MOVIE were applied using their default implementations and settings, and TQV was applied using its original training parameters. For ${\mathrm{ViS}}_3$, we used a GOF size of $N=8$.

Before evaluating the performance of each algorithm on each video database, we applied a four-parameter logistic transform to the raw predicted scores, as recommended by video quality experts group (VQEG) in Ref. 31. The four-parameter logistic transform is given by

Following VQEG recommendations in Ref. 31, we employed the Spearman rank-order correlation coefficient (SROCC) to measure prediction monotonicity, and employed the Pearson linear correlation coefficient (CC) and the root mean square error (RMSE) to measure prediction accuracy. The prediction consistency of each algorithm was measured by two additional criteria: the outlier ratio (OR⁵) and the outlier distance (OD²⁶). OR is the ratio of number of false scores predicted by the algorithm to the total number of scores. A false score is defined as the transformed score lying outside the 95% confidence interval of the associated subjective score.⁵ In addition, OD indicates how far the outliers fall outside of the confidence interval. The OD is measured by the total distance from all outliers to their closest edge points of the corresponding confidence interval.²⁶

4.3.

Overall Performance

The performance of each algorithm on each video database is shown in Table 1 in terms of the five criteria (SROCC, CC, RMSE, OR, and OD). The best-performing algorithm is bolded, and the second best-performing algorithm is italicized and bolded. These data indicate that ${\mathrm{ViS}}_3$ is the best-performing algorithm on all three video databases in terms of all five evaluation criteria. The performances of ${\mathrm{ViS}}_1$ and ${\mathrm{ViS}}_2$ are also noteworthy.

Table 1

Performances of ViS3 and other algorithms on the three video databases. The best-performing algorithm is bolded and the second best-performing algorithm is italicized. Note that ViS3 is the best-performing algorithm on all three databases.

In terms of prediction monotonicity (SROCC), ${\mathrm{ViS}}_3$ is the best-performing algorithm on all three databases. On the LIVE and CSIQ databases, ${\mathrm{ViS}}_3$ and TQV are the two best-performing algorithms. On the IVPL database, ${\mathrm{ViS}}_3$ and MOVIE are the two best-performing algorithms. A similar trend in performance is observed in terms of prediction accuracy (CC and RMSE).

In terms of prediction consistency measured by OR, on the LIVE database, three algorithms (MOVIE, TQV, and ${\mathrm{ViS}}_3$) have an OR of zero, which indicates that they do not yield any outliers. On the IVPL database, both ${\mathrm{ViS}}_3$ and VQM have only one outlier. On the CSIQ database, ${\mathrm{ViS}}_3$ and MOVIE are the two algorithms with the least number of outliers.

In terms of OD, on the LIVE database, three algorithms (MOVIE, TQV, and ${\mathrm{ViS}}_3$) have an OD of zero because they do not have any outliers. On the IVPL database, MOVIE and VQM have the smallest OD. Although ${\mathrm{ViS}}_3$ yields only one outlier on the IVPL database as well as VQM, ${\mathrm{ViS}}_3$ has larger OD because this outlier lies further away from its confidence interval. This indicates that ${\mathrm{ViS}}_3$ has a weakness on the IPPL distortion, to which the outlier belongs. Furthermore, on the CSIQ database, ${\mathrm{ViS}}_3$ and TQV yield the smallest OD values.

Observe that ${\mathrm{ViS}}_1$ and ${\mathrm{ViS}}_2$ yield different relative performances depending on the database. ${\mathrm{ViS}}_1$ shows better predictions than ${\mathrm{ViS}}_2$ on the LIVE and IVPL databases. However, ${\mathrm{ViS}}_2$ shows better predictions than ${\mathrm{ViS}}_1$ on the CSIQ database. Generally, ${\mathrm{ViS}}_3$ shows higher SROCC and CC and lower RMSE, OR, and OD than either ${\mathrm{ViS}}_1$ or ${\mathrm{ViS}}_2$ alone. Nonetheless, it may be possible to combine ${\mathrm{ViS}}_1$ and ${\mathrm{ViS}}_2$ in an adaptive fashion for even better prediction performance, and such an adaptive combination remains an area for future research.

The scatter-plots of logistic-transformed ${\mathrm{ViS}}_3$ values versus DMOS on the three databases are shown in Fig. 12. The plots show a highly correlated trend between the logistic-transformed ${\mathrm{ViS}}_3$ values versus DMOS values. For all the three databases, the predictions are homoscedastic; i.e., there are generally no subpopulations of videos/distortion types for which ${\mathrm{ViS}}_3$ yields lesser or greater residual variance in the predictions. These residuals are used for an analysis of statistical significance in Sec. 4.3.3.

Fig. 12

Scatter-plots of logistic-transformed scores predicted by ${\mathrm{ViS}}_3$ versus subjective scores on the three databases. Notice that all the plots are homoscedastic. The $R$ values denote correlation coefficient between the logistic-transformed scores and subjective scores (DMOS).

4.3.1.

Performance on individual types of distortion

We measured the performance of ${\mathrm{ViS}}_3$ and other algorithms on individual types of distortion for videos from the three databases. For this analysis, we applied the logistic transform function to all predicted scores of each database, then divided the transformed scores into separate subsets according to the distortion types, and then measured the performance criteria in terms of SROCC and CC for each subset. Table 2 shows the resulting SROCC and CC values.

Table 2

Performances of ViS3 and other quality assessment algorithms measured on different types of distortion on the three video databases. The best-performing algorithm is bolded and the second best-performing algorithm is italicized.

Database	Distortion	PSNR	VQM	MOVIE	TQV	${\mathrm{ViS}}_3$
SROCC
LIVE	WLPL	0.621	0.817	0.811	0.754	0.845
	IPPL	0.472	0.802	0.715	0.742	0.788
	H.264	0.473	0.686	0.764	0.769	0.757
	MPEG-2	0.383	0.718	0.772	0.785	0.730
IVPL	DIRAC	0.860	0.891	0.888	0.786	0.926
	H.264	0.866	0.862	0.823	0.672	0.876
	IPPL	0.711	0.650	0.858	0.629	0.807
	MPEG-2	0.738	0.791	0.823	0.557	0.834
CSIQ	H.264	0.802	0.919	0.897	0.955	0.920
	WLPL	0.851	0.801	0.886	0.842	0.856
	MJPEG	0.509	0.647	0.887	0.870	0.789
	SNOW	0.759	0.874	0.900	0.831	0.908
	AWGN	0.906	0.884	0.843	0.908	0.928
	HEVC	0.785	0.906	0.933	0.902	0.917
CC
LIVE	WLPL	0.657	0.812	0.839	0.777	0.846
	IPPL	0.497	0.800	0.761	0.794	0.816
	H.264	0.571	0.703	0.790	0.788	0.773
	MPEG-2	0.395	0.737	0.757	0.794	0.746
IVPL	DIRAC	0.878	0.898	0.870	0.811	0.936
	H.264	0.855	0.869	0.845	0.744	0.898
	IPPL	0.673	0.642	0.842	0.735	0.802
	MPEG-2	0.718	0.836	0.824	0.533	0.912
CSIQ	H.264	0.835	0.916	0.904	0.965	0.918
	WLPL	0.802	0.806	0.882	0.784	0.850
	MJPEG	0.460	0.641	0.882	0.871	0.800
	SNOW	0.769	0.840	0.898	0.846	0.908
	AWGN	0.949	0.918	0.855	0.930	0.916
	HEVC	0.805	0.915	0.937	0.913	0.933

In general, VQM, MOVIE, and ${\mathrm{ViS}}_3$ all perform well on the WLPL distortion; these three algorithms show competitive and consistent performance on the WLPL distortion for both the LIVE and CSIQ databases. For the H.264 compression distortion, ${\mathrm{ViS}}_3$ and MOVIE perform well and consistently across all subsets of H.264 videos on all three databases. ${\mathrm{ViS}}_3$ and MOVIE are also competitive on the MPEG-2 compression distortion and the IPPL distortion on both the LIVE and IVPL databases.

In particular, on the LIVE database, ${\mathrm{ViS}}_3$ has the best performance on the WLPL distortion; VQM and ${\mathrm{ViS}}_3$ have the best performance on the IPPL distortion; ${\mathrm{ViS}}_3$, MOVIE, and TQV are the three best-performing algorithms on the H.264 compression distortion; and TQV and MOVIE are the two best-performing algorithms on the MPEG-2 compression distortion.

The low performance of the ${\mathrm{ViS}}_3$ algorithm on H.264 and MPEG-2 compression types in the LIVE video database is due to the outliers corresponding to specific videos as shown in Fig. 13; the outliers are marked by the red square markers. For H.264, the outliers correspond to the video riverbed where the water’s movement significantly masks the blurring imposed by the compression. However, ${\mathrm{ViS}}_3$ underestimates this masking and, thus, overestimates the DMOS. For MPEG-2, the sunflower seeds in the video sunflower generally impose signficant masking of the MPEG-2 blocking artifacts. However, there are select frames in this video in which the blocking artifacts become highly visible (owing perhaps to failed motion compensation), yet ${\mathrm{ViS}}_3$ does not accurately capture the visibility of these artifacts and, thus, underestimates the DMOS. These types of interactions between the videos and distortions are issues that certainly warrant future research.

Fig. 13

Scatter-plots of logistic-transformed scores predicted by ${\mathrm{ViS}}_3$ versus subjective scores on the H.264 and MPEG-2 distortion of the LIVE database. The second row shows representative frames of the two videos corresponding to the outliers (red square markers in the plots).

On the IVPL database, ${\mathrm{ViS}}_3$ yields the best performance on three types of distortion (DIRAC, H.264, and MPEG-2); ${\mathrm{ViS}}_3$ yields the second best performance on the IPPL distortion, on which MOVIE is the best-performing algorithm. VQM and MOVIE are the second best-performing algorithms on the MPEG-2 distortion. PSNR, VQM, and MOVIE are also competitive on both the DIRAC and H.264 distortion.

On the CSIQ database, TQV and ${\mathrm{ViS}}_3$ are the two best-performing algorithms on the H.264 compression distortion; ${\mathrm{ViS}}_3$ and MOVIE are the two best-performing algorithms on three types of distortion (WLPL, SNOW, and HEVC); MOVIE and TQV are the two best-performing algorithms on the MJPEG. On the AGWN distortion, ${\mathrm{ViS}}_3$ and TQV are competitive with PSNR, which is well known to perform well for white noise.

Generally, ${\mathrm{ViS}}_3$ excels on the H.264 compression distortion and the wavelet-based compression distortion (DIRAC, SNOW), and ${\mathrm{ViS}}_3$, VQM, and MOVIE excel on the WLPL distortion. ${\mathrm{ViS}}_3$ also performs well on the MPEG-2, HEVC, and AWGN distortion. However, ${\mathrm{ViS}}_3$ does not perform well on the MJPEG compression distortion compared to MOVIE and TQV.