(Ebook) Image Statistics in Visual Computing by Tania

Pouli, Erik Reinhard, Douglas W. Cunningham ISBN

9781439874905, 9781568817255, 1439874905, 1568817258 Pdf
Download

https://ebooknice.com/product/image-statistics-in-visual-
computing-4747968

★★★★★
4.6 out of 5.0 (88 reviews )

DOWNLOAD PDF

ebooknice.com
 (Ebook) Image Statistics in Visual Computing by Tania Pouli,
Erik Reinhard, Douglas W. Cunningham ISBN 9781439874905,
9781568817255, 1439874905, 1568817258 Pdf Download

EBOOK

Available Formats

■ PDF eBook Study Guide Ebook

EXCLUSIVE 2025 EDUCATIONAL COLLECTION - LIMITED TIME

INSTANT DOWNLOAD VIEW LIBRARY

Here are some recommended products for you. Click the link to
download, or explore more at ebooknice.com

(Ebook) Biota Grow 2C gather 2C cook by Loucas, Jason; Viles,

James ISBN 9781459699816, 9781743365571, 9781925268492,
1459699815, 1743365578, 1925268497

https://ebooknice.com/product/biota-grow-2c-gather-2c-cook-6661374

(Ebook) Matematik 5000+ Kurs 2c Lärobok by Lena Alfredsson, Hans

Heikne, Sanna Bodemyr ISBN 9789127456600, 9127456609

https://ebooknice.com/product/matematik-5000-kurs-2c-larobok-23848312

(Ebook) SAT II Success MATH 1C and 2C 2002 (Peterson's SAT II

Success) by Peterson's ISBN 9780768906677, 0768906679

https://ebooknice.com/product/sat-ii-success-math-1c-and-2c-2002-peterson-
s-sat-ii-success-1722018

(Ebook) Master SAT II Math 1c and 2c 4th ed (Arco Master the SAT
Subject Test: Math Levels 1 & 2) by Arco ISBN 9780768923049,
0768923042

https://ebooknice.com/product/master-sat-ii-math-1c-and-2c-4th-ed-arco-
master-the-sat-subject-test-math-levels-1-2-2326094
(Ebook) Cambridge IGCSE and O Level History Workbook 2C - Depth
Study: the United States, 1919-41 2nd Edition by Benjamin
Harrison ISBN 9781398375147, 9781398375048, 1398375144,
1398375047
https://ebooknice.com/product/cambridge-igcse-and-o-level-history-
workbook-2c-depth-study-the-united-states-1919-41-2nd-edition-53538044

(Ebook) High Dynamic Range Imaging: Acquisition, Display, and

Image-Based Lighting by Erik Reinhard, Greg Ward, Sumanta
Pattanaik, Paul Debevec ISBN 9780125852630, 0125852630

https://ebooknice.com/product/high-dynamic-range-imaging-acquisition-
display-and-image-based-lighting-2011388

(Ebook) Introduction to Visual Computing: Core Concepts in

Computer Vision, Graphics, and Image Processing by Aditi
Majumder, M. Gopi ISBN 9781482244915, 1482244918

https://ebooknice.com/product/introduction-to-visual-computing-core-
concepts-in-computer-vision-graphics-and-image-processing-6992588

(Ebook) High Dynamic Range Imaging: Acquisition, Display, and

Image-Based Lighting by Erik Reinhard, Wolfgang Heidrich, Paul
Debevec, Sumanta Pattanaik, Greg Ward, Karol Myszkowski ISBN
9780123749147, 012374914X
https://ebooknice.com/product/high-dynamic-range-imaging-acquisition-
display-and-image-based-lighting-1964620

(Ebook) Image Bite Politics: News and the Visual Framing of

Elections (Series in Political Psychology) by Maria Elizabeth
Grabe, Erik Page Bucy ISBN 9780195372076, 9780199707270,
0195372077, 0199707278
https://ebooknice.com/product/image-bite-politics-news-and-the-visual-
framing-of-elections-series-in-political-psychology-1839392
 Image Statistics
in Visual Computing
To achieve the complex task of interpreting what we see, our brains rely on statistical
regularities and patterns in visual data. Knowledge of these regularities can also
be considerably useful in visual computing disciplines, such as computer vision,
computer graphics, and image processing. The field of natural image statistics studies
the regularities to exploit their potential and better understand human vision. With
numerous color figures throughout, Image Statistics in Visual Computing covers
all aspects of natural image statistics, from data collection to analysis to applications
in computer graphics, computational photography, image processing, and art.

The authors keep the material accessible, providing mathematical definitions where
appropriate to help you understand the transforms that highlight statistical regularities
present in images. The book also describes patterns that arise once the images are
transformed and gives examples of applications that have successfully used statistical
regularities. Numerous references enable you to easily look up more information
about a specific concept or application. A supporting website also offers additional
information, including descriptions of various image databases suitable for statistics.
Tania Pouli
Collecting state-of-the-art, interdisciplinary knowledge in one source, this book Erik Reinhard
explores the relation of natural image statistics to human vision and shows how natural
image statistics can be applied to visual computing. It encourages you to develop novel Douglas W. Cunningham
insights and applications in all disciplines that relate to visual computing.

K13473

Computer Graphics
Image Statistics
in Visual Computing
 i i

i i

i i

i i
Image Statistics
in Visual Computing

Tania Pouli
Erik Reinhard
Douglas W. Cunningham

AN A K PETERS BOOK
CRC Press
Taylor & Francis Group
6000 Broken Sound Parkway NW, Suite 300
Boca Raton, FL 33487-2742

© 2014 by Taylor & Francis Group, LLC

CRC Press is an imprint of Taylor & Francis Group, an Informa business

No claim to original U.S. Government works

Version Date: 20130712

International Standard Book Number-13: 978-1-4665-3982-2 (eBook - Kindle)

This book contains information obtained from authentic and highly regarded sources. Reasonable efforts have been
made to publish reliable data and information, but the author and publisher cannot assume responsibility for the valid-
ity of all materials or the consequences of their use. The authors and publishers have attempted to trace the copyright
holders of all material reproduced in this publication and apologize to copyright holders if permission to publish in this
form has not been obtained. If any copyright material has not been acknowledged please write and let us know so we may
rectify in any future reprint.

Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmitted, or uti-
lized in any form by any electronic, mechanical, or other means, now known or hereafter invented, including photocopy-
ing, microfilming, and recording, or in any information storage or retrieval system, without written permission from
the publishers.

For permission to photocopy or use material electronically from this work, please access www.copyright.com (http://
www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood Drive, Danvers, MA 01923,
978-750-8400. CCC is a not-for-profit organization that provides licenses and registration for a variety of users. For
organizations that have been granted a photocopy license by the CCC, a separate system of payment has been arranged.

Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for
identification and explanation without intent to infringe.
Visit the Taylor & Francis Web site at
http://www.taylorandfrancis.com

and the CRC Press Web site at

http://www.crcpress.com
 i i

i i

ER & TP: To our parents Dora & Panagiotis and

Ineke & Franz, for their support, patience, and love
despite our always forgetting to call them.

DWC: I would like to thank my wife, Monica, and

my son, Samuel, not only for their support but also
the useful reminders that there is a world out there
beyond book-writing.

i i

i i
 i i

i i

i i

i i
 i i

i i

Contents

Foreword by Raanan Fattal . . . . . . . . . . . . . . . xiii

Preface . . . . . . . . . . . . . . . . . . . . . xv
Cover Image . . . . . . . . . . . . . . . . . . xvi
Acknowledgments . . . . . . . . . . . . . . . . xvi

I Background 1
1 Introduction . . . . . . . . . . . . . . . . . . 3
1.1 Statistics as Priors . . . . . . . . . . . . . . . 6
1.2 Statistics as Image Descriptors . . . . . . . . . . 7
1.3 Statistical Pipeline . . . . . . . . . . . . . . 10
1.4 Natural Images . . . . . . . . . . . . . . . . 11
1.5 Discussion . . . . . . . . . . . . . . . . . 11
2 The Human Visual System. . . . . . . . . . . . . . 15
2.1 Radiometric and Photometric Terms . . . . . . . . . 15
2.2 Human Vision . . . . . . . . . . . . . . . . 17
2.3 The Eyes . . . . . . . . . . . . . . . . . . 18
2.3.1 Optical Media and the Retina . . . . . . . . 19
2.3.2 Photoreceptors . . . . . . . . . . . . . 20
2.3.3 Horizontal and Bipolar Cells . . . . . . . . 25
2.3.4 Amacrine and Ganglion Cells . . . . . . . . 26
2.4 The Lateral Geniculate Nucleus and Cortical Processing . . 27
2.5 Implications of Human Visual Processing . . . . . . . 29
2.5.1 Visual Acuity . . . . . . . . . . . . . 30
2.5.2 Temporal Resolution . . . . . . . . . . . 31
2.5.3 Contrast . . . . . . . . . . . . . . . 31

vii

i i

i i
 i i

i i

viii Contents

2.5.4 Color Processing . . . . . . . . . . . . 32

2.5.5 Visual Illusions . . . . . . . . . . . . . 34
3 Image Collection and Calibration . . . . . . . . . . . 41
3.1 Image Capture . . . . . . . . . . . . . . . . 42
3.1.1 Photographer and Camera Bias. . . . . . . . 43
3.1.2 High Dynamic Range Imaging . . . . . . . . 43
3.1.3 Field of View . . . . . . . . . . . . . 46
3.1.4 Multispectral and Hyperspectral Imaging . . . . 49
3.1.5 Depth and Range Capture . . . . . . . . . 51
3.2 Post-processing and Calibration . . . . . . . . . . 54
3.2.1 Radiometric Calibration . . . . . . . . . . 54
3.2.2 Lens Aberrations . . . . . . . . . . . . 55
3.2.3 Noise . . . . . . . . . . . . . . . . 59
3.3 Image Databases . . . . . . . . . . . . . . . 60
3.3.1 Van Hateren’s Natural Image Database . . . . . 60
3.3.2 University of Texas at Austin Databases . . . . . 61
3.3.3 UPenn Natural Image Database . . . . . . . 61
3.3.4 The Barcelona Calibrated Images Database. . . . 62
3.3.5 HDR Photographic Survey . . . . . . . . . 62
3.3.6 IPL Calibrated Color Image Database . . . . . 63
3.3.7 McGill Calibrated Color Image Database . . . . 64
3.3.8 Hyperspectral Images of Natural Scenes. . . . . 64
3.3.9 Real-World Hyperspectral Images Database . . . 65
3.3.10 Bristol Hyperspectral Images Database . . . . . 66
3.3.11 Amsterdam Library of Object Images (ALOI) . . . 66
3.3.12 Caltech-256 Object Category Dataset . . . . . 67
3.3.13 Brown Range Image Database . . . . . . . . 67

II Image Statistics 69
4 First-Order Statistics . . . . . . . . . . . . . . . 71
4.1 Histograms and Moments . . . . . . . . . . . . 72
4.1.1 Image Moments and Moment Invariants . . . . . 74
4.1.2 Histogram Adjustments . . . . . . . . . . 75
4.2 Moment Statistics and Average Distributions . . . . . . 77
4.3 Material Properties . . . . . . . . . . . . . . 81
4.4 Nonlinear Compression in Art . . . . . . . . . . . 83
4.5 Dark-Is-Deep Paradigm . . . . . . . . . . . . . 85
4.6 Summary . . . . . . . . . . . . . . . . . 86
5 Gradients, Edges, and Contrast . . . . . . . . . . . . 89
5.1 Real-World Considerations . . . . . . . . . . . . 89
5.1.1 Perceptual Consequences . . . . . . . . . 90
5.1.2 Image Space Consequences . . . . . . . . . 90

i i

i i
 i i

i i

Contents ix

5.2 Gradients . . . . . . . . . . . . . . . . . 90
5.2.1 The Forward Difference Method . . . . . . . 91
5.2.2 The Backward Difference Method. . . . . . . 92
5.2.3 The Central Difference Method . . . . . . . 93
5.2.4 The Söbel Operator . . . . . . . . . . . 94
5.2.5 Second Derivative Methods . . . . . . . . . 95
5.2.6 Gradient Magnitude . . . . . . . . . . . 96
5.2.7 Gradient Statistics . . . . . . . . . . . . 97
5.2.8 Single-Image Gradient Statistics . . . . . . . 97
5.3 Edges . . . . . . . . . . . . . . . . . . . 99
5.3.1 Definition of an Edge . . . . . . . . . . . 100
5.3.2 Edge Detection Processes . . . . . . . . . 100
5.3.3 Edge Statistics . . . . . . . . . . . . . 103
5.4 Linear Scale Space . . . . . . . . . . . . . . 104
5.4.1 The N-Jet and Feature Detectors . . . . . . . 108
5.4.2 Implications for Human Perception . . . . . . 109
5.4.3 Scale-Space Statistics . . . . . . . . . . . 110
5.5 Contrast in Images . . . . . . . . . . . . . . 111
5.6 Image Deblurring . . . . . . . . . . . . . . . 114
5.7 Superresolution . . . . . . . . . . . . . . . 116
5.8 Inpainting . . . . . . . . . . . . . . . . . 116
6 Fourier Analysis . . . . . . . . . . . . . . . . . 119
6.1 Autocorrelation . . . . . . . . . . . . . . . 121
6.2 The Fourier Transform . . . . . . . . . . . . . 123
6.3 The Wiener-Khintchine Theorem . . . . . . . . . . 128
6.4 Power Spectra . . . . . . . . . . . . . . . . 128
6.4.1 Slope Computation . . . . . . . . . . . 129
6.4.2 Spectral Slope Analysis . . . . . . . . . . 132
6.4.3 Dynamic Range. . . . . . . . . . . . . 137
6.4.4 Dependence on Image Representation . . . . . 139
6.4.5 Angular Dependence . . . . . . . . . . . 140
6.4.6 Temporal Dependence . . . . . . . . . . 140
6.4.7 1/f Failures . . . . . . . . . . . . . . 141
6.5 Phase Spectra . . . . . . . . . . . . . . . . 142
6.6 Human Perception . . . . . . . . . . . . . . 144
6.7 Fractal Forgeries . . . . . . . . . . . . . . . 145
6.8 Image Processing and Categorization . . . . . . . . 145
6.9 Texture Descriptors . . . . . . . . . . . . . . 146
6.10 Terrain Synthesis . . . . . . . . . . . . . . . 147
6.11 Art Statistics . . . . . . . . . . . . . . . . 148

i i

i i
 i i

i i

x Contents

7 Dimensionality Reduction . . . . . . . . . . . . . . 153

7.1 Principal Component Analysis . . . . . . . . . . 156
7.1.1 Whitening . . . . . . . . . . . . . . 159
7.1.2 PCA on Pixels . . . . . . . . . . . . . 159
7.1.3 PCA on Patches. . . . . . . . . . . . . 159
7.1.4 PCA on Images . . . . . . . . . . . . . 163
7.1.5 Eigenfaces . . . . . . . . . . . . . . 164
7.2 Independent Components Analysis . . . . . . . . . 166
7.3 ICA on Natural Images . . . . . . . . . . . . . 169
7.4 Gaussian Mixture Models . . . . . . . . . . . . 172
8 Wavelet Analysis . . . . . . . . . . . . . . . . . 175
8.1 Wavelet Transform . . . . . . . . . . . . . . 176
8.2 Multiresolution Analysis . . . . . . . . . . . . 178
8.3 Signal Processing . . . . . . . . . . . . . . . 181
8.4 Other Bases . . . . . . . . . . . . . . . . . 185
8.5 2D Wavelets . . . . . . . . . . . . . . . . 187
8.6 Contourlets, Curvelets, and Ridgelets . . . . . . . . 190
8.7 Coefficient Histograms . . . . . . . . . . . . . 190
8.8 Scale Invariance . . . . . . . . . . . . . . . 193
8.9 Correlations between Coefficients . . . . . . . . . 194
8.10 Complex Wavelets . . . . . . . . . . . . . . 197
8.11 Correlations between Scales . . . . . . . . . . . 200
8.12 Application: Image Denoising. . . . . . . . . . . 202
8.13 Application: Progressive Reconstruction . . . . . . . 203
8.14 Application: Texture Synthesis . . . . . . . . . . 203
9 Markov Random Fields. . . . . . . . . . . . . . . 207
9.1 Image Interpretation . . . . . . . . . . . . . . 208
9.2 Graphs . . . . . . . . . . . . . . . . . . 210
9.2.1 Neighborhood Systems . . . . . . . . . . 210
9.2.2 Cliques . . . . . . . . . . . . . . . 212
9.3 Probabilities and Markov Random Fields . . . . . . . 212
9.3.1 Gibbs Distributions . . . . . . . . . . . 213
9.3.2 Auto-Models. . . . . . . . . . . . . . 214
9.4 MAP-MRF . . . . . . . . . . . . . . . . . 215
9.5 Applications . . . . . . . . . . . . . . . . 216
9.5.1 Image Restoration . . . . . . . . . . . . 216
9.5.2 Object Segmentation . . . . . . . . . . . 217
9.6 Complex Models and Patch-Based Regularities . . . . . 218
9.6.1 Products of Experts . . . . . . . . . . . 218
9.6.2 Fields of Experts . . . . . . . . . . . . 219
9.7 Statistical analysis of MRFs . . . . . . . . . . . 219

i i

i i
 i i

i i

Contents xi

III Beyond Two Dimensions 225

10 Color. . . . . . . . . . . . . . . . . . . . . 227
10.1 Trichromacy and Metamerism . . . . . . . . . . . 228
10.2 Color as a 3D Space . . . . . . . . . . . . . . 229
10.3 Opponent Processing . . . . . . . . . . . . . . 230
10.4 Color Transfer . . . . . . . . . . . . . . . . 235
10.4.1 Color Transfer through Simple Moments . . . . 236
10.4.2 Color Transfer through Higher-Order Manipulation . 238
10.4.3 Histogram Features at Different Scales . . . . . 239
10.4.4 Color Transfer as a 3D Problem . . . . . . . 241
10.5 Color Space Statistics . . . . . . . . . . . . . 242
10.5.1 Color Space Normalization . . . . . . . . . 244
10.5.2 Correlation Analysis . . . . . . . . . . . 245
10.6 Color Constancy and White Balancing . . . . . . . . 248
10.6.1 Computational Color Constancy as the Minkowski
Norm . . . . . . . . . . . . . . . . 251
10.6.2 White-Balance Algorithm Selection . . . . . . 254
10.7 Summary . . . . . . . . . . . . . . . . . 254
11 Depth Statistics . . . . . . . . . . . . . . . . . 257
11.1 The “Dead Leaves” Model . . . . . . . . . . . . 259
11.2 Perception of Scene Geometry. . . . . . . . . . . 260
11.2.1 Length Perception . . . . . . . . . . . . 261
11.2.2 Orientation and Angle Perception . . . . . . . 262
11.3 Correlations between 2D and Range Statistics . . . . . 264
11.4 Depth Reconstruction . . . . . . . . . . . . . 267
12 Time and Motion . . . . . . . . . . . . . . . . . 269
12.1 The Statistics of Time . . . . . . . . . . . . . 269
12.2 Motion . . . . . . . . . . . . . . . . . . 274
12.2.1 Correlation-Based Motion Detection . . . . . . 274
12.2.2 Gradient-Based Motion Detection . . . . . . . 275
12.3 Applications Using Statistical Motion Regularities . . . . 275
12.3.1 Specular Highlights . . . . . . . . . . . 276
12.3.2 Markov Random Fields . . . . . . . . . . 276
12.3.3 People Detection and Biological Motion . . . . 276
12.3.4 Motion Blur . . . . . . . . . . . . . . 277
12.3.5 Shape from Dynamic Occlusion . . . . . . . 277
12.4 Optical Flow . . . . . . . . . . . . . . . . 278
12.4.1 Probabilistic Optical Flow . . . . . . . . . 281
12.4.2 Statistics of Real-World Motion and Retinal Flow . 282
12.4.3 Statistics of Optical Flow . . . . . . . . . 283

i i

i i
 i i

i i

xii Contents

A Basic Definitions . . . . . . . . . . . . . . . . . 285

A.1 Probabilities and Bayes’ Rule . . . . . . . . . . . 285
A.2 Gaussian Distribution . . . . . . . . . . . . . 286
A.3 Kullback-Leibler Divergence . . . . . . . . . . . 286
Bibliography . . . . . . . . . . . . . . . . . . . 287
Inde x . . . . . . . . . . . . . . . . . . . . . . 341

i i

i i
 i i

i i

Foreword
To me, the term natural image statistics does not merely refer to a collection
of statistics one can collect from images but signifies a paradigm shift in the
way we reason about images of natural scenery. Instead of modeling them as
a deterministic outcome of a physical process using explicit geometric entities
and illumination laws, we started considering natural images as random variables
characterized by unique statistical regularities.
In fact, these two modeling approaches, integrated in the Bayesian framework,
have proved themselves most effective for a wide range of image restoration tasks.
For example, when removing noise or blur from an image, the signal-to-noise
level becomes very low across the high-frequency band and therefore a large num-
ber of unknowns must be recovered. State-of-the-art denoising and deblurring
methods fill in this missing data based on the sparse distribution that derivative
and wavelet coefficients exhibit in natural images. Similarly, when filling holes
in an image or increasing its resolution, millions of pixels must be determined.
Non-parametric patch-based statistical models show a remarkable ability to re-
cover this seemingly missing data by exploiting various self-similarities found
within and across natural images.
The statistical investigation of natural images has also proved itself valuable
for studying the human visual system, which was exposed and adapted to this
stimulus throughout its evolution. Researchers showed, for example, that im-
age transformations based on some optimal statistical criterion, such as compact
representation, provide important insights over the response of various sensory
neurons.
These practical and theoretical implications make the search for new and more
accurate natural image priors a very active line of research in image understand-
ing, processing, and neuroscience.
The book Image Statistics in Visual Computing offers a delightful coverage
of this fascinating subject. It starts by describing fundamental concepts of human
visual perception, as well as important calibration aspects behind the collection of
natural image databases. The book thoroughly covers key image transformations

xiii

i i

i i
 i i

i i

xiv Foreword

and describes their unique statistical behavior. It devotes a chapter to Markov

random field models, which constitute the most widely used framework for defin-
ing image priors. Finally, the book surveys several important extensions of the
statistical study to the behavior of color, motion, and range images.

Raanan Fattal
School of Computer Science and Engineering
The Hebrew University of Jerusalem

i i

i i
 i i

i i

Preface

Natural images exhibit statistical regularities that differentiate them from random
collections of pixels. Moreover, the human visual system appears to have evolved
to exploit such statistical regularities. The field of natural image statistics is tradi-
tionally concerned with trying to understand how human vision operates by iden-
tifying statistical regularities in natural images that may be used in some way.
There are some amazing examples of how the human visual system is beauti-
fully optimized to observe and interpret natural images. One example that put us
on the trail of natural image statistics is described in detail in Chapter 10: Take
a large set of pixels from a collection of natural images and transform them to
cone response space. If these three color channels are then rotated using principal
components analysis, the result is a new color space that is formally decorrelated.
As it happens, the color space created this way strongly resembles a color
opponent space. It is known from many sources that the human visual system
also employs a color opponent space to encode the visual signal prior to sending
it from the eye to the brain. The beauty of this is, therefore, that human vision
actively decorrelates its visual input, provided it is observing natural images. In
practice it appears that the three decorrelated channels are closer to independence,
which is an even stronger statistical claim.
This has subsequently given rise to applications in image processing that treat
the three color channels in color opponent space as independent, simplifying an
otherwise more difficult three-dimensional problem into three one-dimensional
problems. Color transfer algorithms form a good example of a class of algorithm
in which this appoach is successfully explored.
The lesson learned from this example—and the premise for this book—is that
the field of natural image statistics is not only helpful in the study of human vi-
sion but may also create opportunities that can be exploited in visual engineering
disciplines such as computer vision, computer graphics, and image processing.
Thus, as many areas of visual computing are concerned either with producing im-
agery for observation by humans or with processing images to obtain information

xv

i i

i i
 i i

i i

xvi Preface

similar to what humans would extract, it would be prudent to understand which

statistical regularities occur in nature so they can be emulated by image synthe-
sis and analysis methods. In this book we introduce all aspects of natural image
statistics, ranging from data collection to analysis and finally their applications in
computer graphics, computational photography, image processing, and art.
We do not aim to provide an exhaustive list of all applications that use natural
image statistics in some way; instead, we see this as a book that may give readers
those all-important “aha” moments. We think that there are many possibilities
for natural image statistics to be used in a variety of applications, of which the
currently known ones only scratch the surface. We hope that this book will serve
as a trigger for new and fresh ideas, leading to novel insights and applications in
all disciplines that relate to visual computing—both in academic and industrial
settings.
To serve this purpose, we have tried to keep the book as accessible as possible.
Wherever appropriate, we provide the basic mathematics to understand the trans-
forms that emphasize some of the many statistical regularities present in natural
images. The statistical regularities and the patterns that can arise once they are
transformed are described in some detail. We then give examples of applications
that have successfully used these patterns. To keep the book manageable, we def-
initely do not aim to provide an exhaustive list of examples, nor do we intend to
describe each example in full mathematical detail. We do, however, provide many
references so that more information about any topic and specific application dis-
cussed in this book can be easily found. In that sense, we hope that the book may
serve as an entry point into a highly interesting area of research that brings to-
gether aspects of statistics, perception, neuroscience, physics, image processing,
computer vision, and computer graphics.

Cover Image
One of the most fascinating aspects of natural image statistics is their ability to de-
construct complex images of natural environments into compact descriptors. The
seemingly endless variety of nature leads to striking—and repeatedly occurring—
regularities. The image used in the cover tries to show exactly this. Starting from
a photograph of a natural scene, the maple leaves are slowly transformed to pro-
gressively simpler shapes, eventually converging to circles arranged according to
the “Dead leaves model,” discussed in Chapter 11.

Acknowledgments
We are grateful for the help offered by many friends and colleagues who have
assisted either directly or indirectly with the production of this book. In particular,

i i

i i
 i i

i i

Preface xvii

we would like to thank A K Peters, the Max Planck Institute for Informatics,
and the Brandenburg Technical University for their support. We would also like
to thank Ayan Chakrabarti, Mark Fairchild, Bill Geisler, Jan-Mark Geusebroek,
and Dale Purves for allowing us to use their images. Further, we would like to
thank Karol Myszkowski, Patrick Perez, and Hans-Peter Seidel. We are also very
grateful for the great foreward that Ranaan Fattal has written for us.
Many researchers have made their work available online, and this has proved
to be a real help, allowing us to reproduce their results on our own images.
For instance, for the production of several figures we have adapted Stanford’s
Wavelab package,1 which is a suite of MATLAB tools for experimenting with
wavelets. The Wavelab team and list of contributors consists of David Donoho,
Arian Maleki, Morteza Sharam, Jon Buckheit, Maureen Clerc, Jerome Kalifa,
Stephane Mallat, Thomas Yu, Mark Reynold Duncan, Xiaoming Huo, Ofer Levi,
Shaobing Chen, Iain Johnstone, Jeffrey Scargle, Rainer von Sachs, Thomas Yu,
Jeffrey Scargle, and Eric Kolaczyk. We gratefully acknowledge their efforts.
Stéphane Mallat’s book A Wavelet Tour of Signal Processing2 is accompanied
on the internet by a large set of numerical experiments that can be used in the
classroom.3 These are designed and maintained by Gabriel Peyré. We thank him
for providing us with the answers to these excercises, significantly simplifying
our experiments and the production of further figures for our book. The “Dead
leaves” figures in Chapter 11 (and indeed part of the front cover) were made using
Gabriel’s MATLAB “Toolbox Image.”
We would also like to thank Ivan Selesnick, Shihua Cai, Keyong Li, Levent
Sendur, and A. Farras Abdelnour at the Department of Electrical and Computer
Engineering of Brooklyn Poly for making available on the internet a collection of
MATLAB routines to experiment with several wavelet transforms.4 In particular,
we have used their dual-tree complex wavelet transform implementation for ex-
perimenting with correlations across scale of the magnitude of complex wavelets.
Some of the ICA related figures in Chapter 7 were created by modifying a
version of the FastICA code that is kindly made available by Aapo Hyvärinen,
Jarmo Hurri, and Patrik Hoyer as part of their book on natural image statistics,5 a
book that served as an inspiration for ours, albeit with a different focus.
The texture synthesis examples were generated with the code kindly made
available by Javier Portilla and Eero Simoncelli.6 This code relies on the multi-
scale image processing toolbox made available by Eero Simoncelli.7

1 www-stat.standord.edu/~wavelab
2 www.ceremade.dauphine.fr/~peyre/wavelet-tour/
3 www.ceremade.dauphine.fr/~peyre/numerical-tour/
4 eeweb.poly.edu/iselesni/WaveletSoftware/index.html
5 www.naturalimagestatistics.net
6 www.cns.nyu.edu/lcv/texture
7 www.cns.nyu.edu/~eero/software.php

i i

i i
 i i

i i

xviii Preface

Chapter 9 (“Markov random fields”) contains some figures that were gener-
ated with code kindly made publicly available by Uwe Schmidt, Qi Gao, and
Stefan Roth.8
We would like to also thank Christian Winger, a student at the Brandenburg
Technical University in Cottbus for allowing us to use his implementation of the
inpainting algorithm by Crimini et al. [134] for the figure in Chapter 5. This
method was implemented and extended as part of Christian’s bachelor’s thesis.
Our gratitude also goes to all the great people at A K Peters and CRC Press
for their help and guidance throughout the publishing process. Our editors, Sarah
Chow and Charlotte Byrnes, have been a pleasure to work with, and their ideas
and corrections have helped make the book better. Finally, we’d like to especially
thank Alice and Klaus Peters as well for signing us up in the first place; we hope
they’re enjoying their well-earned retirement.

8 www.gris.tu-darmstadt.de/research/visinf/software/index.en.htm

i i

i i
 i i

i i

Part I

Background

i i

i i
 i i

i i

i i

i i
 i i

i i

Chapter 1

Introduction

For many reasons, including basic survival, we continuously sense the environ-
ment around us. Light is perhaps the premier carrier of information about our
environment. The signal it conveys is also the most complex to observe and in-
terpret. It should therefore not come as a surprise that a significant portion of the
human brain is dedicated to processing the light reaching our eyes (although there
appear to be significant differences between individuals in terms of how much of
the brain is devoted to the processing of visual stimuli [17]). Images—both still
and moving—contain a wealth of information: a quick look through the nearest
window is enough to tell us what the weather is like, whether we are in the coun-
tryside or the city, and what time of the day it might be. Movement in a scene
might tell us of an approaching predator or that the bus is approaching, while
color and texture may inform us about the state of ripeness of fruit or whether our
loaf of bread has gone off again.
Considering that human vision takes as input at any point in time an image
of the world through each eye—effectively two projections of the world onto the
retina—it is remarkable that we are able to infer so much of the world around us.
These two streams of flat images are sufficient to allow us to navigate complex en-
vironments, meaningfully respond to events, and recognize individuals. Anyone
who has tried to implement an algorithm to reconstruct a three-dimensional (3D)
environment from a pair of two-dimensional (2D) images knows how impossibly
difficult this is. Yet, human vision appears to accomplish this effortlessly.
Thus, human vision has the task of inferring at least some aspects of the 3D
configuration of our world, given 2D image inputs. This is known to be an un-
derconstrained problem: in essence, human vision has to invent one dimension
which has gone missing as a result of the projection of light on the retina. It
has been repeatedly shown that human vision employs many heuristics, some of
which are known. As an example, human vision generally expects a single light
source [598] that comes from above [407]. Face recognition, for instance, be-
comes significantly impaired if a human face is illuminated from below.

i i

i i
 i i

i i

4 1. Introduction

There is a long history of research on the processes taking place within the
human (and animal) visual system that enable us to perceive and interpret im-
ages. Vision scientists typically study such processes through carefully con-
structed psychophysical experiments that measure overt behavior to systematic
changes in simple light patterns [498, 554, 138] or by directly measuring electro-
physiological responses while the retina is stimulated with a simple light pattern
[354].1 In the latter case, the light pattern that maximally stimulates a given cell
is known as that cell’s preferred stimulus, and it carries considerable information
about the structure and size of the cell’s receptive field. Alternatively, one could
measure behavior or cell responses while the retina is stimulated with natural im-
ages, since that is what the retina usually processes [445, 616, 151, 30, 764].
Naturally, the ability of the visual system to analyze information is very much
related to the information that is there to be processed, which just happens to
be images. The field of natural image statistics has focused on exactly this rela-
tionship. Initially the field focused on describing images such as television sig-
nals [168, 424], but it has since expanded into explicitly examining the many
links between the structure of a wide variety of image types on the one side
and the workings of the visual system on the other. In general, natural image
statistics research collects a set of natural images, transforms them according to
a statistical optimization criterion, and then evaluates whether the result in some
sense forms a reasonable description of the response properties of some neurons
[645, 234, 26, 546, 691].
The argumentation is roughly as follows. The retina has a large number of
discrete photoreceptors, so that the projection of light is spatially discretized.
Equivalently, photographs contain a large number of discrete pixels.2 The val-
ues that these elements can take is not fully random. This can, for instance, be
seen in Figure 1.1, where at the top a natural image is displayed. Below, we have
shuffled the order of the pixels. All pixels in the top image are also present in the
bottom images. However, the bottom images do not carry meaningful informa-
tion. Moreover, human vision is not well equipped to discern differences between
images such as these.
As the images at the bottom of Figure 1.1 are effectively recombinations of the
pixels of the image at the top, one may ask how many combinations of pixels there
exist for every random image [637]. Assuming that we have a small-ish image of 4
Mpix = 4 × 106 pixels, then the number of ways these pixels could be recombined
would be (4 × 106 )!, which for practical purposes approaches infinity. This means
that the number of natural images that we might encounter in real life is infinitely
smaller than then number of possible combinations in which the photoreceptors
in the retina could be stimulated [395, 215, 154, 639].

1 See hubel.med.harvard.edu.
2 We are assuming for this discussion, as well as for the remainder of the book, that images are
digitally stored and represented, and that they are therefore spatially discretized in a pixel grid.

i i

i i
1. Introduction 5

Original image

A re-combination of pixels Another re-combination of pixels

Figure 1.1. The pixels of the top image were randomly shuffled twice to form the
images at the bottom. These are only two of an almost infinite number of recom-
binations possible. Note how human vision does not easily detect differences be-
tween these two images. (Andean Cock-of-the-rock, Jurong Bird Park, Singapore,
2012)

As a consequence we might argue that natural images are in fact incredibly

rare. For every natural image there is an infinite number of stimuli that are ran-
dom combinations of pixels. Most of these random combinations, however, do
not reveal structure that can be interpreted by human vision. The subset of natural
images is sometimes called an image manifold [704]. It therefore stands to rea-
son that human vision has evolved to observe, interpret, and make sense of this
incredibly rare subset of all images [257].
Despite being a rare subset of all possible images, natural images still contain
an immense degree of variation. Within this plethora of differences are statistical
regularities that make natural images amenable to human interpretation. What are
these features that natural images have in common with each other that differenti-
 i i

i i

6 1. Introduction

ate them from random images? Furthermore, in what ways does the human visual
system expect “naturalness”? In this book we look at these and similar questions,
which are core to this exciting area of research called natural image statistics.
Beyond helping to understand how the human visual system works, the regu-
larities in natural images are very valuable to disciplines that rely on images, such
as computer graphics, computer vision, or image processing. Similar in spirit to
the processing blocks within the visual system, computational models can take
advantage of the regularities in natural images to produce more plausible results
or to infer information that may otherwise be lost.
If, for instance, the aim of a particular technique is to reconstruct—or in-
paint—areas of an image that may be hidden, statistical information from nearby
regions can be used to create a patch that looks similar, yet not identical, to its
neighbors [568] (see Section 5.8). In another scenario, statistical regularities
computed over a collection of images can serve as priors, guiding (optimization)
algorithms [211, 453, 667]. The distribution of edge magnitudes in natural im-
ages, for example, can aid algorithms aiming to deblur images (see Section 5.6).
In effect, this produces an expectation of how edges are typically distributed in
images, helping the deblurring algorithm in question produce a solution that con-
forms to that expectation. Likewise, an analysis of art has shown that paintings
often have the same statistics as real-world images, and thus a number of exist-
ing techniques can be used to analyze, recover, or otherwise alter paintings (for
reviews, see [279, 602, 709]).
Although the distinction is not always as clear cut, images can be analyzed
in ensembles or individually. In the first case, the statistical analysis can provide
insights about regularities and properties of general scene types and situations,
which can in turn be applied when manipulating further images of that type. In
the latter case, the information acquired by looking at an individual image is only
useful for dealing with that particular image. However, this approach has shown
to provide powerful statistics internal to the image itself [829]. Statistics are of
course also commonly used in a third way; namely, for analyzing the results of
studies or experiments often performed for assessing imaging algorithms [138],
although this is somewhat outside the scope of this book. The following sec-
tions will briefly introduce the possible applications of statistics in the context of
imaging disciplines.

1.1 Statistics as Priors

In the same way that human vision has certain expectations or heuristics regarding
its visual input, we may employ natural image statistics to guide algorithm design
to produce solutions that may be perceived as more natural or realistic. Exam-
ples include image deblurring [211] and image restoration or denoising [831] (see

i i

i i
 i i

i i

1.2. Statistics as Image Descriptors 7

Sections 5.6–5.8). The most direct way in which this may happen is in optimization-
type algorithms.
An optimization algorithm requires an objective function—the function that is
minimized—which encodes a set of opposing characteristics that the result ought
to have. The algorithm typically begins by guessing an initial solution. The qual-
ity of the solution is then measured by inserting it into the objective function.
Then iteratively a new candidate solution is guessed and compared against the
previous best solution. There are several more or less standard choices for op-
timization processes, defining how the next guess is selected, but the main chal-
lenge is in the design of an appropriate objective function.
Objective functions usually consist of multiple terms. The first term is usually
a norm that compares the current guess against some desired goal. The closer the
image is to the goal, the smaller the number that this term returns. As this term
will admit many solutions that are mathematically equally good, extra conditions
are normally added to steer the optimization into a direction that is good according
to some other, higher-level criteria.
Thus, such additional terms are designed to penalize wrong solutions and pro-
mote more appropriate ones respectively. They act as priors in the sense that they
encode expectations regarding the mathematical shape of the solution. Often such
priors take the form of smoothness terms. When the goal of the optimization al-
gorithm is to filter or generate an image, natural image statistics have proven to
be a good source of priors, effectively allowing some statistical form of natural-
ness to be encoded. An example is the deblurring techniques mentioned earlier:
the average distribution of gradients in images can serve as a prior to suppress
spurious edges that might arise from the deblurring process [211, 667].
In the particular case of image gradients, they are found to often have small
values and only sometimes have large values, as shown in Figure 1.2. In other
words, the distribution of gradients is heavy-tailed rather than Gaussian. Note
that although here we show the results computed on a single image, this trend
holds in general for natural images. Thus, if we were to compute the gradient
distribution over a large set of natural images, we would find very similar results,
as we will see in Chapter 5. This means that it is possible to use this distribution as
a general prior in optimization problems, whenever we wish our resulting image
to have a gradient distribution that is in some sense “natural.” Accounting for the
non-Gaussian distribution of gradients has helped better solve several problems,
including image deblurring [211], image denoising [631, 686], superresolution,
and demosaicing [726].

1.2 Statistics as Image Descriptors

More often than not when manipulating images, the most useful source of infor-
mation is none other than the image itself. If the goal is to add objects to a photo-

i i

i i
8 1. Introduction

7 6
Bin count [5

Log bin count

6 Red channel
5
Green channel
5
Blue channel 4
4
3
3
 

 
 
ï ï    ï ï   
Gradient Gradient
Figure 1.2. Image gradients are heavy-tailed, as shown in this example. The
image at the top was first converted to log space, and then the horizontal gradients
were computed. They are plotted in a linear plot (bottom left) and for visualization
in a log-linear plot (bottom right). Note that most gradients are small, leading to a
heavy-tailed distribution. (Monument Valley, USA, 2012)

graph, for instance, clues as to how the new object would look if it were actually
placed in the real scene can be obtained by analyzing the image itself. This can,
for instance, provide information regarding color and illumination (mis-)matches.
Conversely, if we wish to remove an object from an image, it is possible to
analyze regions that neighbor the pixels to be removed. Here, statistics may help
to infer what would be the most likely scene that lies behind the removed object.
In this case, no general statistical assumptions are made about the input image,
i.e., it is not desirable to try and infer a specific background based on ensemble
statistics.
Instead, statistical tools are used to analyze the scene and purely based on
those, the missing parts can be reconstructed. For instance, in the image shown in
Figure 1.3, we may want to remove the camel without making it obvious that the
resulting image was manipulated in any way. Areas surrounding the camel con-
tain pixels similar to what would likely be behind it, and as such they can be used
to estimate the missing information. Since only a single image is available, an ac-
curate reconstruction would not be possible, but relying on statistical similarities
1.2. Statistics as Image Descriptors 9

Figure 1.3. Inpainting example. Note that this is only an illustration, created
semi-automatically with Photoshop. See Section 5.8 for statistics-based inpaint-
ing. (Giza, Egypt, 2007)

Figure 1.4. An example of a new texture (right) that was synthesized based on the
example texture given on the left. The texture on the right was tiled nine times.

could be sufficient to produce a plausible result. Image inpainting is discussed in

more detail in Section 5.8.
Another example of where the statistics of an individual image can be mean-
ingfully used is texture synthesis. Given an example image with a desired appear-
ance, the goal of texture synthesis algorithms is to generate a texture tile with sim-
ilar appearance. There are many algorithms based on resampling [787, 195, 157]
and some that explicitly take image statistics into account [692, 581, 582]. In
the latter case, statistics based on a complex wavelet decomposition in the output
texture are forced to match those of an example texture. An example is shown in
Figure 1.4 and further discussed in Chapter 8.
There are cases where statistics can serve both as a global prior, capturing
general assumptions and expectations about images, and as a tool to analyze spe-
cific input. If, for instance, the goal is to white balance an image such that color
casts of a non-white light source are removed, one can analyze the color content
 i i

i i

10 1. Introduction

of that image to that effect [69]. The statistical assumption made in general about
scenes is that the average reflectance of all materials is achromatic (the gray-world
assumption). This means that if we illuminate a scene with a colored light source,
the light reflected off objects (and subsequently captured in an image) would be
white only if the color of the illuminant were white. If the average color of an
image therefore deviates from gray, that difference is an indication of the color of
the light source.
In this case, the expectation that images will average to middle gray serves as
a global prior, albeit in a simple sense. On the other hand, the specific statistics
of each image, namely, the average of all its pixel values, describes that partic-
ular image and how it differs from the expected value. This technique for white
balancing, as well as variants thereof, are discussed further in Chapter 10.

1.3 Statistical Pipeline

Whether statistical regularities are computed over image ensembles or single im-
ages, a similar process takes place. First, a dataset is constructed (Chapter 3). This
may be a collection of images (an ensemble), a collection of patches selected from
one or more images, or even single pixels, depending on the application.
The dataset is then transformed from the original image space to a different
space, which allows the study of a particular aspect of the image data. This may be
a transform to the gradient domain if edges are of interest or to a different color
space, for instance. In effect, this transformation takes the form of a function
that operates on single or multiple pixels. As an example, consider a color space
transform from sRGB to the LMS color space. In this case, the function operating
on the image pixels is a matrix transformation applied to each triplet of color
values (Chapter 10). If one is studying gradients instead, two or more pixel values
are considered to determine the rate of change between them (Chapter 5). The
transform is usually designed to extract or emphasize some statistical regularity
present in images. Often, the transform is related to some aspect of human vision,
in that the transform is matched to the processing thought to occur in the human
visual system.
Once the data has been transformed to the appropriate space, the extracted
features need to be analyzed. This is achieved through statistical analysis similar
to what would be employed to compute trends in any type of data. One may aver-
age along one or more dimensions, look at the distribution of values, or construct
a model that predicts the measured data.
It is important to note that what this process typically aims to achieve is
twofold. First, statistical measures can reduce the dimensionality of the data while
retaining information relevant to a particular application. This is one of the key
factors making image statistics a very useful tool for analyzing images, which by

i i

i i
 i i

i i

1.4. Natural Images 11

their nature are rich in data. Second, the statistical analysis can aid in deriving
models for the data.

1.4 Natural Images

At this point, it is perhaps worthwhile to briefly discuss what we mean by natural
images. In the context of visual computing, an image is typically a 2D array
of (colored) pixel values. The content depicted in an image determines whether
this image could be considered natural or not. There are many images that are
not natural images, including X-rays, images produced with magnetic resonance
imaging (MRI), and forward looking infrared (FLIR) imagery. Often, an image is
considered to be the realization of a spatial stochastic process [704]. It is common
to assume that this stochastic process is stationary, which means that the statistics
are the same everywhere in the image.
It would also be prudent to limit oneself to images that have sufficient com-
plexity, ranging from surface reflectance and texture to object shapes, and to re-
quire that objects sit on a ground plane with the camera looking more or less
forward. This is consistent with how humans normally observe the world, and it
may therefore be reasonable to assume that human evolution, as well as personal
development, has occurred under similar circumstances.
To what extent human vision is the result of eons of evolution versus the result
of personal development may also impact the choice of images that are studied. If
it is assumed that the organization of human vision is due to evolution only, then
it would make sense to exclude images that depict built-up environments. On the
other hand, if individual development plays an important role, then natural image
ensembles may well include images of interiors and built-up environments.
In this book, we make no strong assumptions either way. We are interested in
natural image statistics and how they may help solve problems in visual comput-
ing. As the need arises, for certain applications it would make sense to restrict the
type of images analyzed further, or conversely expand the set of images consid-
ered. We have therefore deliberately left the title of this book at Image Statistics
rather than the perhaps more expected Natural Image Statistics. Nonetheless,
when capturing a dataset of subsequent analysis, certain care must be taken to not
inadvertently introduce bias. This topic is discussed further in Chapter 3.

1.5 Discussion
We have found that most transforms that could be applied to pixel data and which
have some bearing on human vision all more or less point into the same direction,
which is that of sparseness. Statistical distributions of transformed data tend to
have high kurtosis, which is another way of saying that the data tend to have most

i i

i i
Other documents randomly have
different content
Welcome to our website – the ideal destination for book lovers and
knowledge seekers. With a mission to inspire endlessly, we offer a
vast collection of books, ranging from classic literary works to
specialized publications, self-development books, and children's
literature. Each book is a new journey of discovery, expanding
knowledge and enriching the soul of the reade

Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.

Let us accompany you on the journey of exploring knowledge and

personal growth!

ebooknice.com