SINGLE IMAGE SUPER RESOLUTION: A COMPARATIVE STUDY

Jae-Kwang Lee et al. (Eds) : CCSEA, AIFU, DKMP, CLOUD, EMSA, SEA, SIPRO - 2017
pp. 139– 147, 2017. © CS & IT-CSCP 2017 DOI : 10.5121/csit.2017.70213
SINGLE IMAGE SUPER RESOLUTION: A
COMPARATIVE STUDY
Aliaa Youssef1
, Sameh Zarif2
and Amr Ghoneim1
1
Department of Computer Science, Helwan University, Helwan, Egypt
2
Department of Information Technology, Menofia University, Menofia, Egypt
ABSTRACT
The majority of applications requiring high resolution images to derive and analyze data
accurately and easily. Image super resolution is playing an effective role in those applications.
Image super resolution is the process of producing high resolution image from low resolution
image. In this paper, we study various image super resolution techniques with respect to the
quality of results and processing time. This comparative study introduces a comparison between
four algorithms of single image super-resolution. For fair comparison, the compared algorithms
are tested on the same dataset and same platform to show the major advantages of one over the
others.
KEYWORDS
Super resolution, Interpolation, Neighbour filling, Resizing, Low resolution
1. INTRODUCTION
Supper resolution (SR) is the process of enhancement the resolution of images. Resolution is a
measure of frequency content in an image. There are always requests for good quality images
from low one, although the cameras for high resolution (HR) images are expensive. Also image
capturing setting is not ideal so the resulting images are blurred and noisy. Regarding that using
of supper resolution techniques to enhance resolution of images and maintain the details of them
is preferable [1-4].
HR images are frequently used in large applications such as satellite imaging, sports images,
medical imaging, computer vision, remote sensing, surveillance systems, object detection and
recognition. The need of zooming of images to analyze visual information also increases the
request for super-resolution [5-7].
In general, super resolution techniques are divided into two categories, which are multi image
super-resolution and single image super-resolution [2]. Multi image super-resolution is the
process of generation high resolution image from multiple low resolution images. Single image
super-resolution is the process of generating high-resolution image from its low resolution image
[8]. This study focuses on single image super-resolution techniques.

140 Computer Science & Information Technology (CS & IT)
Many researchers have developed algorithms for solving super resolution issues. These
algorithms could be classified into three types: interpolation-based, learning-based and
reconstruction-based [9-10]. Interpolation based algorithms such as nearest neighbour
interpolation, bilinear interpolation, bicubic interpolation, and lanczos interpolation are simple
but the resulted image is blurred [8]. The learning-based algorithms main idea is that the lost
details in (LR) images could be retrieved from a dictionary or a data set retrieved from fixed (HR)
images set or website [11-13]. The reconstruction based algorithms enforce a constraint that the
version of the estimated (HR) image should be consistent with its (LR) image according to
predefined values [1].
The sections of paper are organized as follow. Section II presents image super resolution related
techniques. Section III introduces a brief description about four compared techniques that are
used in the comparison. Section IV describes the experimental results that show the advantages
and disadvantages of each technique. Conclusion is presented in section V.
2. RELATED WORK
The authors in [14] proposed fast and robust multi frame super-resolution. This method based on
normalization and Gaussian model. In normalization stage, the output images are generated with
sharp edges. The sharp images are followed by Gaussian model to remove noise.
J. Sun et al [1] presented an image super-resolution using gradient profile prior. The gradient
profile prior is learned from a huge number of natural images. It provides a constraint on image
gradients when it estimates a high-resolution image from a low-resolution one. This gradient
constraint helps to sharpen the details and putdown ringing along edges.
M. Bevilacqua et al [15] presented super-resolution through neighbour embedding. In this method
the generation of a high-resolution image patch does not depend on only one of the nearest
neighbours in the training set. Instead, it depends simultaneously on multiple nearest neighbours
in a way similar to LLE for manifold learning.
The SR algorithms which depend on reconstruction-based require image patches from one or
several images. This is achieved by registration and alignment of multiple LR image patches of
the same scene with sub-pixel level accuracy [5], [7]. If the images haven‟t insufficient patch
self-similarity, these methods are not able to produce satisfying results [9]. A recent methods
proposed in [17] moderate this limitation by learning image prior models via kernel principal
component analysis from multiple image frames.
Another type of super resolution is learning based methods. The information is learned/observed
from the training image data. Chang et al. [18] introduced the method of locally linear embedding
(LLE) for super resolution dedications. Support vector regression (SVR) is proposed by Ni et al.
[2] to fit the patches of low resolution image and the corresponding pixel value of the high
resolution image in DCT and spatial domains. In order to achieve better SR results, one needs to
carefully/manually select the training data. In such cases, the computation complexity of training
and difficulty of training data selection take into account. Sparse Representation, it was first
applied to SR by Yang et al. [8], [9]. They considered the image patch from HR images as a
sparse representation with respect to an over-complete dictionary composed of signal-atoms. Kim

Computer Science & Information Technology (CS & IT) 141
and Kwon [19] proposed an example-based single image SR for learning the mapping function
between the low and high resolution images by using sparse regression and natural image priors.
S. Derin et al [20] presented a novel Bayesian formulation for joint image registration and super
resolution. The unknown high resolution image, motion parameters and algorithm parameters,
including the noise variances, are modelled within a hierarchical Bayesian framework. The
proposed framework can be extended to more general super resolution applications with more
complex motion models. Y. Zhu et al [21] introduced a single image super resolution method
using deformable patches. By considering each patch as a deformable field rather than a fixed
vector, the patch dictionary is more expressive. This algorithm doesn‟t have the ability for
various of texture e.g. logo, animal, flowers.
Recently many researchers have been used sparse super-resolution algorithm for image
interpolation and inpainting. Sparse super-resolution Estimators algorithm introduced a group of
inverse problem estimators computed by mixing adaptively a group of linear estimators. Sparse
mixing weights are calculated over a blocks of coefficients in a frame providing a sparse signal
representation [22]. Computing adaptive directional image interpolations over a wavelet frame
provides effective nonparametric of inverse problems. Curvelet frames and contour let frames
build sparse image approximations by taking advantage of the image directional regularity. The
instability of these algorithms come from their flexibility .Sparse super-resolution algorithm in
[22] can be improved by Computing an adaptive signal representation in blocks. In which they
are obtained as an adaptive mixing of linear Tikhonov estimators, over blocks of vectors in a
frame. A fast orthogonal block matching pursuit algorithm is introduced to reduce the number of
process by Applying of mixing directional interpolators over oriented blocks in a wavelet frame.
In spatial domain, multi-images SR algorithms are mostly working on aliasing artifacts that are
present in LR images. The representative methods in this category include iterative back
projection (IBP), Projection onto convex sets (POCS), Maximum Likelihood (ML). Iterative back
projection (IBP) based method in [16] is initially a guess the HR targeted image. It is needed and
then it is refined. A guess can be obtained by registering the LR images over an HR grid and then
averaging them. This initial guess can be refined by using the simulated imaging model with a set
of available LR observations. Then the error between the simulated LR images and the observed
ones is obtained and back-projected to the coordinates of the HR image to improve the initial
guess. In this method the back-projected error is the mean of the errors that each LR image
causes.
3. COMPARED METHODS
This section describes four image super resolution methods. The compared methods that used in
our comparative study are selected to be two states of art super resolution methods, and the other
two are recently developed methods. The compared four methods are:
• Image Super-Resolution via Sparse Representation [2010] [9]
• Generative Bayesian Image Super-Resolution with Natural Image Prior [2012] [23]
• Super-resolution from Transformed Self-Exemplars [24]
• Deep Networks for Image Super-Resolution with sparse Prior [2015] [17]

3.1. Image Super-Resolution via Sparse Representation
Super resolution could be discussing from other point of view. J.Yang et al [9] introduced image
super-resolution via sparse representation to generate SR based upon sparse representations by
make training of coupled dictionaries from high and low resolution patch pairs. Let X is the high
resolution image, Y the low resolution image, x the high resolution image patch, y the low
resolution image patch, Dh dictionaries for high resolution and Dl dictionaries for low resolution.
In order to recover the high resolution image (X), the method is based on two constrains. First
constrain is the reconstruction constrain Y=SHX where S is down-sampling filter and H is
blurring filter. Second constrain is Sparse prior, which patches (x) of the high resolution image
(X) and patches (y) of low resolution image can represented like a sparse linear grouped in a
dictionaries (Dh) and (Dl). The high resolution feature can be reconstructed as a combination of
the sparse linear of the learned Dh dictionaries for high resolution atoms by assuming that low
resolution and high resolution features share the same sparse recovered coefficients. The main
disadvantage of this algorithm is the highly exhaustive computation caused by the used
optimization function.
3.2. Generative Bayesian Image Super-Resolution with Natural Image Prior
In this method, initially a guess the high resolution image is needed and then it is refined. This
guess can be obtained with the posterior mean, rather than the posterior mode [23] .To estimate
the high resolution image, we take the advantage of sampling of high dimensional data for
Gaussian Model and develop for MMSE for the HR image. In this method, Results compared
with SR algorithms verify its effectiveness. This method has flexibility in using natural images
priors in Bayesian model, and it use MCMC sampling based generative approach. The main
drawback of this method is that it is not fast as MAP Solution.
3.3. Super-resolution from Transformed Self-Exemplars
The authors in [24] proposed an image super resolution method based on transformed self-
exemplars. The algorithm produces the high resolution image by using the following steps:
1) Compute a transformation matrix T (homography) that warps target patch P to its best
matching patch Q (source patch) in the down sampled image ID for each patch P in the
low resolution image I. To obtain the parameters of such a transformation, we estimate a
nearest neighbor field between image I and down sampled image ID using a modified
Patch Match algorithm.
2) Extract the high resolution patch QH from the image I, which is the high resolution
version of the source patch Q.
3) To obtain the self-exemplar PH, the inverse of the computed transformation matrix T to
„unwarp‟ the high resolution patch QH, which is estimated HR version of the target
patch P. After that, the algorithm paste PH in the HR image at the location corresponding
to the LR patch P.
4) For all target patches, we repeat the above steps to obtain an estimation of the HR image.

5) Run the iterative back projection method to ensure that the estimated high resolution
image satisfies the rebuilding constraint with the given low resolution.
Finally this method is difficulty conducting with fine details when the planes are not well
detected. In addition to that, it is computationally complex due to the training procedure.
3.4. Deep Networks for Image Super-Resolution with Sparse Prior
Deep networks learning algorithms have been successfully applied in many areas of computer
vision and image processing, including low-level image restoration issues. Several models
depend on deep neural networks have been recently shown up for image super-resolution and
gained better performance that overcome all previous invented models. We argue that domain
expertise offered by the conventional sparse coding technique is still valuable, and how it can be
concatenated with the key ingredients of deep networks learning to achieve further enhanced
results. The method in [17] assumed that a sparse coding technique designed particularly for
super-resolution could be shaped as a neural network, and trained in an ordered structure from
end to end. The performance of the deep network based on sparse coding technique leads to much
more efficient and effective training, as well as a reduced model size.
4. EXPERIMENTAL RESULTS
We conducted a subjective evaluation of the super resolution results for several state of art
methods, for comprehensive and fair comparison between the compared methods, they have been
tested on a laptop with core i5 CPU and six GB of ram. To demonstrate the strengths and
drawbacks of each of them, the comparison is done on the same dataset. According to the changes
that happened to the super resolution image after resizing, it is very difficult to evaluate the
quality by traditional objective evaluation such as pixel to noise ratio (PSNR). So, the quality
depends on the human visual perception system rather than mathematical measures. Figure 1
shows an example of resizing low resolution face and natural images by using the four state-of-art
methods. First row in figure 1 shows the original images. Second row shows the super resolution
output from the method in [9]. The super resolution output from method in [23] is shown in third
row. Fourth row introduces the super resolution output from method in [24]. Finally, Fifth row
represents the super resolution output from method in [17].
In this comparative study, we conducted a mathematical quality measure by using PSNR.
Therefore, we created artificial low resolution images. Then, we applied the four state-of-art
methods to reconstruct the high resolution images from the artificial low resolution images. After
that, we calculated the PSNR between the original high resolution images and the reconstructed
high resolution images. Table I shows PSNR comparison for the images in figure 1. Table II
presents processing time comparisons between the four methods for the images in figure 1.

Table I. PSNR comparison between the four state-of-art methods.
Table II. Processing time comparison between the four compared methods (in seconds).

Figure. 1 Examples of recovering super resolution images. Row 1 represents the original images, row 2
represents the results of method [9], row 3 represents the results of method [23], row 4 represents the
results of method [24], and row 5 represents the results of method [17].
5. CONCLUSIONS
In this research paper, we introduce image super resolution comparative study between four
recent methods. We conducted to be fair as much as possible by comparing the four image super
resolution methods in the same images as well as device hardware. The experimental results
illustrate the strengths and drawbacks of each method. According to the PSNR table and Time
table we can conclude that, Method [24] produces the higher PSNR value over the other methods.
In the second table Method [17] produces the minimum Time. Therefore, we recommend them
for all researchers to be applied in their super resolution applications future work.
ACKNOWLEDGEMENTS
The authors gratefully would like to thank the Pre-Master students of Faculty of Computers and
Information, Menofia University, year 2016 for their efforts.

REFERENCES
[1] J. Sun, Z. Xu, and H. Shum, “Image super-resolution using gradient profile prior,” in IEEE
Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1–8, 2008.
[2] K. S. Ni and T. Q. Nguyen, “Image super-resolution using support vector regression,” IEEE Trans.
Image Process., vol. 16, no. 6, pp. 1596–1610, 2007.
[3] J. Sun, J. Zhu, and M. F. Tappen, “Context-constrained hallucination for image super-resolution,” in
Proc. IEEE Conf. Comput. Vision and Pattern Recognition, pp. 1-8, 2010.
[4] Zeyde, R., Elad, M., Protter, M “single image scale-up using Sparse-representations” Curves and
Surfaces, pp. 711–730, 2012.
[5] A. Chakrabarti, A. N. Rajagopalan, and R. Chellappa, “Super-resolution of face images using kernel
PCA-based prior,” IEEE Trans. Multimedia, vol. 9, no. 4, pp. 888–892, 2007.
[6] M. Protter and M. Elad, “Image sequence denoising via sparse and redundant representations,” IEEE
Trans. Image Process., vol. 18, no.1, pp. 27–35, 2009.
[7] J. Mairal, F. Bach, J. Ponce, G. Sapiro, and A. Zisserman, “Non-local sparse models for image
restoration,” in Proc. ICCV, pp. 2272–2279, 2009.
[8] J. Yang, J. Wright, T. S. Huang, and Y. Ma, “Image super-resolution as sparse representation of raw
image patches,” in IEEE Conf. Comput. Vision and Pattern Recognition, pp. 1-8, 2008.
[9] J. Yang, J. Wright, T. Huang, and Y. Ma, “Image super-resolution via sparse representation,” IEEE
Trans. Image Process., vol. 19, no. 11, pp. 2861–2873, 2010.
[10] K. I. Kim and Y. Kwon, “Example-Based Learning for Single-Image Super-Resolution and jpeg
Artifact Removal” Max-Planck-Institute for Biological Cybernetics, pp. 1-28, 2008.
[11] Yang, J., Wang, Z., Lin, Z., Cohen, S., Huang, T. “Coupled dictionary training for image super-
resolution” IEEE Transactions on Image Processing, vol. 21(8), 3467–3478, 2012.
[12] Dong, C., Loy, C.C., He, K., Tang, X. “Learning a deep convolutional network for image super-
resolution” uropean Conference on Computer Vision, pp. 184–199, 2014.
[13] Yang, J., Lin, Z., Cohen, S.: Fast image super-resolution based on in-place example regression. In:
IEEE Conference on Computer Vision and Pattern Recognition. pp. 1059–1066 (2013).
[14] S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, “Fast and Robust Multi-frame Super-resolution”,
IEEE Transactions on Image Processing, vol. 13, no. 10, pp. 1327-1344,
[15] M. Bevilacqua, A. Roumy, C. Guillemot, and M. Morel. “Low-Complexity Single-Image Super-
Resolution based on Nonnegative Neighbor Embedding” Proceedings of the British Machine Vision
Conference (BMVC). , pp. 135.1–135.10, 2012
[16] Patel Shreyas A,”Novel Iterative Back Projection Approach ”, Journal of Computer Engineering ,
Volume 11, Issue 1, PP 65-69, 2013.
[17] Z. Wang, D. Liu, J. Yang, W. Han and T. Huang, "Deep Networks for Image Super-Resolution with
Sparse Prior," IEEE International Conference on Computer Vision (ICCV), pp. 370-378, 2015

[18] H. Chang, D.-Y. Yeung, and Y. Xiong, “Super-resolution through neighbor embedding,” in Proc.
IEEE Conf. Comput. Vision and Pattern Recognition, pp. 1-8, 2004.
[19] K. I. Kim and Y. Kwon, “Single-image super-resolution using sparse regression and natural image
prior,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 6, pp. 1127–1133, Jun. 2010.
[20] S. Derin, R. Molina and A. K. Katsaggelos “Variational Bayesian Super Resolution” IEEE
TRANSACTIONS ON IMAGE PROCESSING, VOL. 20, NO. 4, APRIL 2011.
[21] Y. Zhu, Y. Zhang, A. L. Yuille “Single Image Super-resolution using Deformable Patches” in Proc.
IEEE Conf. Comput. Vision and Pattern Recognition, pp. 1-8, 2014.
[22] S. Mallat, G Yu “Super-resolution with sparse mixing estimators”, IEEE Transactions on Image
Processing, VOL. 19, NO. 11, pp. 2889 – 2900, 2010.
[23] H Zhang, Y Zhang, H Li, “Generative Bayesian image super resolution with natural image prior”
IEEE Transactions on Image processing, vol. 21, no. 9, pp. 4054-4067, 2012.
[24] J. Huang, A. Singh, N. Ahuja “Single Image Super-resolution from Transformed Self-Exemplars”
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5197 - 5206, 2015.
AUTHORS
Sameh Zarif received his BSc and MSc degrees in information technology from
Menofia University, Egypt, in 2005 and 2009 respectively. He completed his Doctor
of Philosophy from centre of intelligent signal & imaging research (CISIR),
Universiti Teknologi PETRONAS (UTP), Malaysia 2015. Currently he is an
assistant Professor in Department of Information Technology at Menofia University
Egypt. In addition to his current research into image super resolution, texture
synthesis, and image completion, his interests lie in image processing, computer
vision, and pattern recognition.

SINGLE IMAGE SUPER RESOLUTION: A COMPARATIVE STUDY

More Related Content

What's hot

Viewers also liked

Similar to SINGLE IMAGE SUPER RESOLUTION: A COMPARATIVE STUDY

Recently uploaded

SINGLE IMAGE SUPER RESOLUTION: A COMPARATIVE STUDY