Comparison of SSIM and PSNR for Image Quality Evaluation

Wang Weiqiang

SCE, Nanyang Technological University


 Abstract

I developed a software for comparing SSIM and PSNR the two methods of image quality evaluation, and compare the two algorithms by experimental data. The experimental results show that the SSIM algorithm is more suitable for the human visual system characteristics than the PSNR and it can get better image quality evaluation results.

 


1 Introduction 

SSIM and PSNR are two kinds of image quality assessment, PSNR is one of traditional objective methods, it has simple calculation method and wide range of applications in practice. Because it does not consider the human visual perception and image characteristics, in some case it is out of touch with people’s subjective evaluation, but SSIM(Structural Similarity) is easy to implement and has a strong applicability. Experiment shows that it has a better correlation with subjective. SSIM algorithm is excellent for image quality evaluation. In this report  I try to give out the compare result between this two algorithms.

2 PSNR algorithm

PSNR(Peak Signal-to-Noise Ratio) is the ratio between the maximum power of signal and power of noise which affects the fidelity of the signal’s representation. Usually PSNR is used to measure the quality of compressed images, the signal is from original image and the noise is the error created by compression codecs. PSNR is defined via MSE(Mean Squared Error) and MSE is defined as follow.

Figure 1

 

x is signal image with MxN dimension, x^ is noisy approximation of x, m and n define the position of the pixel which need to be calculated in two images. x(m,n) and x^(m,n) is the luminance value of the 2 pixels at (m,n).

Figure 2

 

From the definition of PSNR in figure 2, we can know if the quality of noisy image is very closed to original image, the value of MSE will be very small even close to zero, the value of 2552/MSE will be very large, that’s why use logarithmic decibel scale in the formula.

 

3 SSIM algorithm

 

SSIM(Structural SIMilarity) consists of three parts, brightness measurement compare, contrast measurement compare and structure information measurement. The formula is defined as figure 3.

Figure 3

the average of

the average of

the variance of

the variance of

the covariance of  and

Figure 4

wherein: x and y are the original image block and the distorted block; and  denote the mean value of the x and y; δx and δy are respectively standard deviation of x and y; δxy represents the covariance of x and y; parameters α, β, γ are greater than zero, so that can adjust the proportion of three parts. C1, C2 and C3 are small constants and in order to avoid zero denominator, wherein C1=(K1L)2, C2=(K2L)2, C3=C2/2,K1,K2<<1; L is the dynamic range of the pixel.

 

4 Experimental and results

I) Constants initial

During experiment, the constants of SSIM are set as below:

K1=0.01, K2=0.03, L=255 for 8-bit grayscale image, so C1=5.06,C2=45.56 and C3=22.78, set  α, β, γ  as 1.

II) Database

I choose LIVE Multiply Distorted Image Quality Database for experiment in this report, the version is release 2 which contains 175 Jpeg200 compression images, 169 Jpeg compression images, 145 gaussian blur images, 145 white noise images and 145 images which are with bit errors in JPEG2000 bit stream. It can be downloaded from http://live.ece.utexas.edu/research/quality/subjective.htm. The image library also gives Difference Mean Opinion Scores (DMOS). The bigger is the DMOS value, the lower is the image quality. I used MOS(100-DMOS) values in the  experiment to make scatter plots are more visualized. That means bigger is MOS value, the higher is the image quality. In this database, all image name, distorted image list and their DMOS values are in Matlab file, but it’s not easy to read them from .net programme, so I decided to copy them out manually and hard-code them as constants in my software, that mean the software can not be commonly used, but it’s enough just for this report.

III)Software/Tool

In order to understand SSIM and PSNR well, I developed a simple software base on .NET platform, programing language is C#.

Because all data information of the image library is in Matlab file, and it’s troublesome to read them from my programer, so I have to copy them from Matlab file and re-organize them in C# code. First of all, I defined images list, reference images and distorted images DMOS values in three arrays instead of reading them from Matlab file.

When the software is running, I can locate the image library folder by clicking “Browse” button, and then select distortion category from 5 radio buttons, follow by clicking “Compute” button to tell the software to scan the folder to find out all distorted images according the category selected, the software will load original image and distorted image one by one and calculate out the PSNR and SSIM values. These values will be shown at window form with MOS value, and also will be marked as points in two charts.

Finally I can get regression line for each chart by clicking “Trendline” button. The UI of the software is shown at figure 5.

Figure 5

 

In the charts of the software, abscissa is PSNR/SSIM value, ordinate is the MOS(Mean Opinion Scores) value, the core code of this software is attached in appendix A.

In this report all scatter plots are the screenshot of this software, in order to make the charts more clear, I removed table lines and added legend to flag data meaning.

IV) Experiment Results

 Here list out some example data of single image and all distortion category scatter plots.

Figure 6:  image “bikes.bmp” and it’s JPEG2000 compression distorted images.

Figure 7 :image “bikes.bmp” and it’s fast fading distorted images.

Figure 8: Image “bikes” and it’s JPEG compression distorted image

i) Scatter plots of subjective mean opinion score(MOS) versus PSNR and SSIM for Jpeg2000 compression.

 

ii) Scatter plots of subjective mean opinion score(MOS) versus PSNR and SSIM for Jpeg compression.

iii) Scatter plots of subjective mean opinion score(MOS) versus PSNR and SSIM for gaussian blur.

iv) Scatter plots of subjective mean opinion score(MOS) versus PSNR and SSIM for white noise.

v) Scatter plots of subjective mean opinion score(MOS) versus PSNR and SSIM for fading fast.

5 Conclusion, analyzing and discussion

Compare above scatter plots, we can find that in some case PSNR does not do well, for example, the values in fast fading and jpeg compression are very discreted, and for gaussian blur it’s values are agminated to middle range, actually we can get the answer from it’s definition which is only focus on the measure of difference of pixel. In another case, if we try to compute the MSE between two identical images, the value will be zero and hence the PSNR will be undefined (division by zero).  The main limitation of this metric is that it relies strictly on numeric comparison and does not actually take into account any level of biological factors of the human vision system.

SSIM does better than PSNR from all testing, especially for fast fading, gaussian blur and Jpeg2000 compression, it can evaluate the image from three parts and this method accord with expect of human visual system, but in Jpeg compression and white noise testing it’s result is not good enough to match to MOS. Anyway the results show that the SSIM algorithm can better fit the human visual characteristics than PSNR.

The structural similarity (SSIM) index measures the similarity between two images in a manner that is more consistent with human perception than traditional techniques like mean square error (MSE).  For example, blurred images are perceived as bad quality by the human eye, and this is consistent with results from the SSIM metric, unlike the MSE method, which claims a blurred image is similar to its focused original.

Because of its correlation to human perception, SSIM has become an accepted part of image quality and video analysis practices for analyzing compressed video data.

In conclusion, PSNR is more efficient than SSIM in algorithm. SSIM has strong ability on image quality assessment, but in some case it can not do well, in some paper SSIM was improved to contains more and more image feature, such as contourlet, gradient etc. From those experiment conclusion more image feature included in algorithm the result is more close to mean opinion scores.

Although PSNR has some limit in image quality assessment, but it still is an efficient method, in some situation that request is not very rigorous, PSNR is economic choice. SSIM is much better than PSNR, but the main issue for SSIM is efficiency, so the optimization of the SSIM index for various image processing algorithms needs to be studied.

References

 

  1. Zhou Wang, Member, IEEE, Alan Conrad Bovik, Fellow, IEEE, Hamid Rahim Sheikh, Student Member, IEEE, and Eero P. Simoncelli “Image Quality Assessment: From Error Visibility to Structural Similarity” in IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 13, NO. 4, APRIL 2004
    https://ece.uwaterloo.ca/~z70wang/publications/ssim.pdf

  2. Peak signal-to-noise ratio in Wikimedia
    http://en.wikipedia.org/wiki/Peak_signal-to-noise_ratio

  3. Structural similarity in Wikimedia
    http://en.wikipedia.org/wiki/SSIM

  4. Similarity check (PSNR and SSIM) on the GPU
    http://docs.opencv.org/doc/tutorials/gpu/gpu-basics-similarity/gpu-basics-similarity.html

The following two tabs change content below.
Wang Weiqiang is a senior web developer, and professional on ASP.NET, MVC, C#, Python, SQL Server, HTML5, Javascript, also interesting in machine learning and related algorithm.

Latest posts by Wang Weiqiang (王维强) (see all)

Wang Weiqiang is a senior web developer, and professional on ASP.NET, MVC, C#, Python, SQL Server, HTML5, Javascript, also interesting in machine learning and related algorithm.

Posted in Digital Multiple Media Tagged with: ,

Leave a Reply

Your email address will not be published. Required fields are marked *

*

Categories

Recent Posts

Related Posts