ECE359 SIGNAL PROCESSING FOR

MULTIMEDIA

INTRODUCTION & FUNDAMENTALS

Prof. Fatma Newagy

Prof. of Communications Engineering
[email protected]
Assessment Weights and Schedule
Assignments Weeks 3, 9

Quizzes Weeks 6, 10

Mid-Term Examination Week 7

Project Week 13

Final Exam End of semester

Any copied
document
(assignments,
reports, projects,…)
Assignments / presentation / project 20%
from internet/
Quizzes 10%
student
Mid-Term Exam 20%
Will get negative
Lab Reports 10% mark
Final Exam 40%
Total 100%
Article Reading and Presentation
• Article Reading and Presentation
• Medical image analysis
• Face, fingerprint, and other object recognition
• Image and/or video compression
• Image segmentation and/or denoising
• Digital image/video watermarking/steganography and
detection
• Whatever you’re interested …
 Evaluation of article reading and project
• Evaluation of article reading and presentation
• Report
Article reading
— Submit a survey of the articles you read and the list of the
articles
— Submit an article including introduction, methods, experiments,
results, and conclusions
— Submit the project code, the readme document, and some
testing samples (images, videos, etc.) for validation
• Presentation
Journals & Conferences
in Image Processing
• Journals:
— IEEE T IMAGE PROCESSING
— IEEE T MEDICAL IMAGING
— INTL J COMP. VISION
— IEEE T PATTERN ANALYSIS MACHINE INTELLIGENCE
— PATTERN RECOGNITION
— COMP. VISION AND IMAGE UNDERSTANDING
— IMAGE AND VISION COMPUTING
……
• Conferences:
— CVPR: Comp. Vision and Pattern Recognition
— ICCV: Intl Conf on Computer Vision
— ACM Multimedia
— ICIP
— SPIE
— ECCV: European Conf on Computer Vision
— CAIP: Intl Conf on Comp. Analysis of Images and Patterns
……

• IEEE Communications Surveys & Tutorials

• IEEE Multimedia
• IEEE Signal Processing Magazine
Text Books
• Rafael C. Gonzalez and Richard E. Woods, Digital Image
Processing, Prentice Hall, 2008, Third Edition.

• Rafael C. Gonzalez and Richard E. Woods, Digital Image

Processing Using Matlab, Prentice Hall, 2009, Second
Edition.
Extra materials
• https://web.stanford.edu/class/ee368/

• http://www.imageprocessingplace.com/

• ……
This lecture will cover
• Introduction
• State of the art examples of digital image processing
• Key stages in digital image processing
• Digital Image / Multimedia Processing Fundamentals
Introduction
• What is Digital Image Processing?
Digital Image
— a two-dimensional function
x and y are spatial coordinates
The amplitude of f is called intensity or gray level at the point (x, y)

Digital Image Processing

— process digital images by means of computer, it covers low-, mid-, and high-level
processes
low-level: inputs and outputs are images
mid-level: outputs are attributes extracted from input images
high-level: an ensemble of recognition of individual objects

Pixel
— the elements of a digital image
What is DIP? (cont…)
•The continuum from image processing to computer vision
can be broken up into low-, mid- and high-level processes

Low Level Process Mid Level Process High Level Process

Input: Image Input: Image Input: Attributes
Output: Image Output: Attributes Output: Understanding

Examples: Noise Examples: Object Examples: Scene

removal, image recognition, understanding,
sharpening segmentation autonomous navigation

In this course we will

stop here
Image Sharpening

(a) Original Image (b) After sharpening

Removing Noise

(a) Original Image (b) After removing noise

Image Deblurring

(a) Original Image (b) After removing the blur

Image Segmentation
History of DIP
•1980s - Today: The use of digital image processing
techniques has exploded and they are now used for all
kinds of tasks in all kinds of areas
• Image enhancement/restoration
• Artistic effects
• Medical visualisation
• Industrial inspection
• Law enforcement
• Human computer interfaces
Examples: Image Enhancement
•One of the most common uses of DIP techniques: improve
quality, remove noise etc
Examples: The Hubble Telescope
•Launched in 1990 the Hubble
telescope can take images of
very distant objects
•However, an incorrect mirror
made many of Hubble’s
images useless
•Image processing techniques
were used to fix this
Examples: Artistic Effects

•Artistic effects are used to

make images more
visually appealing, to add
special effects and to
make composite images
Examples: Medicine
•Take slice from MRI scan of canine heart, and find
boundaries between types of tissue
• Image with gray levels representing tissue density
• Use a suitable filter to highlight edges

Original MRI Image of a Dog Heart Edge Detection Image

Examples: GIS
•Geographic Information Systems
• Digital image processing techniques are used extensively to
manipulate satellite imagery
• Terrain classification
• Meteorology
Examples: GIS (cont…)

•Night-Time Lights of the

World data set
• Global inventory of human
settlement
• Not hard to imagine the kind of
analysis that might be done
using this data
Examples: Industrial Inspection
•Human operators are
expensive, slow and
unreliable
•Make machines do the
job instead
•Industrial vision systems
are used in all kinds of
industries
Examples: Law Enforcement

•Image processing techniques

are used extensively by law
enforcers
• Number plate recognition for speed
cameras/automated toll systems
• Fingerprint recognition
• Enhancement of CCTV images
Key Stages in Digital Image Processing
Image Morphological
Restoration Processing

Image
Segmentation
Enhancement

Image Object
Acquisition Recognition

Representation
Problem Domain
& Description
Colour Image Image
Processing Compression
Key Stages in Digital Image Processing:
Image Aquisition
Image Morphological
Restoration Processing

Image
Segmentation
Enhancement

Image Object
Acquisition Recognition

Representation
Problem Domain
& Description
Colour Image Image
Processing Compression
Key Stages in Digital Image Processing:
Image Enhancement
Image Morphological
Restoration Processing

Image
Segmentation
Enhancement

Image Object
Acquisition Recognition

Representation
Problem Domain
& Description
Colour Image Image
Processing Compression
Key Stages in Digital Image Processing:
Image Restoration
Image Morphological
Restoration Processing

Image
Segmentation
Enhancement

Image Object
Acquisition Recognition

Representation
Problem Domain
& Description
Colour Image Image
Processing Compression
Key Stages in Digital Image Processing:
Morphological Processing
Image Morphological
Restoration Processing

Image
Segmentation
Enhancement

Image Object
Acquisition Recognition

Representation
Problem Domain
& Description
Colour Image Image
Processing Compression
Key Stages in Digital Image Processing:
Segmentation
Image Morphological
Restoration Processing

Image
Segmentation
Enhancement

Image Object
Acquisition Recognition

Representation
Problem Domain
& Description
Colour Image Image
Processing Compression
Key Stages in Digital Image Processing:
Object Recognition
Image Morphological
Restoration Processing

Image
Segmentation
Enhancement

Image Object
Acquisition Recognition

Representation
Problem Domain
& Description
Colour Image Image
Processing Compression
Key Stages in Digital Image Processing:
Representation & Description
Image Morphological
Restoration Processing

Image
Segmentation
Enhancement

Image Object
Acquisition Recognition

Representation
Problem Domain
& Description
Colour Image Image
Processing Compression
Key Stages in Digital Image Processing:
Image Compression
Image Morphological
Restoration Processing

Image
Segmentation
Enhancement

Image Object
Acquisition Recognition

Representation
Problem Domain
& Description
Colour Image Image
Processing Compression
Key Stages in Digital Image Processing:
Colour Image Processing
Image Morphological
Restoration Processing

Image
Segmentation
Enhancement

Image Object
Acquisition Recognition

Representation
Problem Domain
& Description
Colour Image Image
Processing Compression
Computer Vision: Some Applications
• Optical character recognition (OCR)
• Face Detection
• Smile Detection
• Login without password using fingerprint scanners and
face recognition systems
• Object recognition in mobiles
• Smart Cars
• Vision in space
Fundamentals in DIP
A Simple Image Formation Model
f (x, y)  i(x, y) r(x, y)

f (x, y): intensity at the point (x, y)

i(x, y): illumination at the point (x, y)
(the amount of source illumination incident on the scene)
r(x, y): reflectance/transmissivity at the point (x, y)
(the amount of illumination reflected/transmitted by the object)
where 0 < i(x, y) <  and 0 < r(x, y) < 1
Some Typical Ranges of illumination

• Illumination
Lumen — A unit of light flow or luminous flux
Lumen per square meter (lm/m2) — The metric unit of measure
for illuminance of a surface

• On a clear day, the sun may produce in excess of 90,000 lm/m2 of

illumination on the surface of the Earth

• On a cloudy day, the sun may produce less than 10,000 lm/m2 of
illumination on the surface of the Earth

• On a clear evening, the moon yields about 0.1 lm/m2 of illumination

• The typical illumination level in a commercial office is about 1000 lm/m2

Some Typical Ranges of Reflectance

• Typical values of reflectance r(x,y)

• 0.01 for black velvet

• 0.65 for stainless steel

• 0.80 for flat-white wall paint

• 0.90 for silver-plated metal

• 0.93 for snow

Representing Digital Images
• The representation of an M×N numerical array as

 f (0,0) f (0,1) ... f (0, N 1) 

 f (1,0) f (1,1) ... f (1, N 1) 
f (x, y)  
 ... ... ... ... 
 f (M 1,0) f (M 1,1) ... f (M 1, N 1) 

Representing Digital Images
• The representation of an M×N numerical array as

 a0,0 a0,1 ... a0,N1 

a a1,1 ... a1,N1  
A  1,0
 ... ... ... ... 
a 
 M1,0 aM1,1 ... aM1,N1 
Representing Digital Images
• The representation of an M×N numerical array in
MATLAB

 f (1,1) f (1,2) ... f (1, N) 

 f (2,1) f (2,2) ... f (2, N) 
f (x, y)  
 ... ... ... ... 
 f (M,1) f (M,2) 
 ... f (M, N)
Representing Digital Images
• Discrete intensity interval [0, L-1], L=2k

• The number b of bits required to store a M × N

digitized image

b=M×N×k
Spatial and Intensity Resolution
• Spatial resolution
— A measure of the smallest discernible detail in an
image
— stated with line pairs per unit distance, dots (pixels)
per unit distance, dots per inch (dpi)

• Intensity resolution
— The smallest discernible change in intensity level
— stated with 8 bits, 12 bits, 16 bits, etc.
Spatial and Intensity Resolution
Spatial and Intensity Resolution
Spatial and Intensity Resolution
Image Interpolation

• Interpolation — Process of using known data to

estimate unknown values
e.g., zooming, shrinking, rotating, and geometric correction

• Interpolation (sometimes called resampling) — an

imaging method to increase (or decrease) the number of
pixels in a digital image.
Some digital cameras use interpolation to produce a larger image than
the sensor captured or to create digital zoom
 Image Interpolation:
Nearest Neighbor Interpolation

f1(x2,y2) = f(x1,y1)
f(round(x2), round(y2))
=f(x1,y1)

f1(x3,y3) =
f(round(x3), round(y3))
=f(x1,y1)
Image Interpolation:
Bilinear Interpolation

(x,y)

f2 (x, y)
 (1 a) (1b) f (l, k)  a (1b) f (l 1, k)
(1 a) b f (l, k 1)  a b f (l 1, k 1)
l  floor(x), k  floor( y), a  x l,b  y  k.
Image Interpolation:
Bicubic Interpolation
• The intensity value assigned to point (x,y) is obtained by the
following equation

f3(x, y)  aij x y
3 3
i j

i0 j 0
• The sixteen coefficients are determined by using the sixteen
nearest neighbors.
Examples: Interpolation
Examples: Interpolation
Examples: Interpolation
Examples: Interpolation
Examples: Interpolation
Examples: Interpolation
Examples: Interpolation
Examples: Interpolation
Basic Relationships Between Pixels

• Neighborhood

• Adjacency

• Connectivity

• Paths

• Regions and boundaries

Basic Relationships Between Pixels

• Neighbors of a pixel p at coordinates (x, y)

 4-neighbors of p, denoted by N4(p):
(x-1, y), (x+1, y), (x,y-1), and (x, y+1).
(x, y-1)
(x-1, y) P (x,y) (x+1, y)
(x, y+1)
 4 diagonal neighbors of p, denoted by ND(p):
(x-1, y-1), (x+1, y+1), (x+1,y-1), and (x-1, y+1).
(x-1, y+1) (x+1, y-1)
P (x,y)
(x-1, y-1) (x+1, y+1)

 8 neighbors of p, denoted N8(p)

N8(p) = N4(p) U ND(p)
Basic Relationships Between Pixels q

• Adjacency p
Let V be the set of intensity values

4-adjacency: Two pixels p and q with values from V are

4-adjacent if q is in the set N4(p).

8-adjacency: Two pixels p and q with values from V are

8-adjacent if q is in the set N8(p).
Basic Relationships Between Pixels q

p
• Adjacency
Let V be the set of intensity values

m-adjacency: Two pixels p and q with values from V are m-

adjacent if

(i) q is in the set N4(p),

or
(ii) q is in the set ND(p) and the set N4(p) ∩ N4(q) has no pixels whose
values are from V.
=>>>(mixed)

Important Note: the type of adjacency used must be specified

Basic Relationships Between Pixels

• Path
 A (digital) path (or curve) from pixel p with coordinates (x0, y0) to
pixel q with coordinates (xn, yn) is a sequence of distinct pixels with
coordinates

(x0, y0), (x1, y1), …, (xn, yn)

Where (xi, yi) and (xi-1, yi-1) are adjacent for 1 ≤ i ≤ n.

 Here n is the length of the path.

 If (x0, y0) = (xn, yn), the path is closed path.

 We can define 4-, 8-, and m-paths based on the type of adjacency
used.
Examples: Adjacency and Path
V = {1, 2}

0 1 1 0 1 1 0 1 1
0 2 0 0 2 0 0 2 0
0 0 1 0 0 1 0 0 1

8-adjacent m-adjacent
Examples: Adjacency and Path
V = {1, 2}

01,1 1
1,2 1
1,3 0 1 1 0 1 1
02,1 2
2,2 0
2,3 0 2 0 0 2 0
03,1 0
3,2 1
3,3 0 0 1 0 0 1
`
8-adjacent m-adjacent

The 8-path from (1,3) to (3,3): The m-path from (1,3) to (3,3):
(i) (1,3), (1,2), (2,2), (3,3) (1,3), (1,2), (2,2), (3,3)
(ii) (1,3), (2,2), (3,3)
Basic Relationships Between Pixels

• Connected in S
Let S represent a subset of pixels in an image. Two
pixels p with coordinates (x0, y0) and q with coordinates
(xn, yn) are said to be connected in S if there exists a
path

(x0, y0), (x1, y1), …, (xn, yn)

i,0  i  n,(xi , yi ) S
Basic Relationships Between Pixels

Let S represent a subset of pixels in an image

• For every pixel p in S, the set of pixels in S that are connected to p is

called a connected component of S.

• If S has only one connected component, then S is called Connected

Set.

• We call R a region of the image if R is a connected set

• Two regions, Ri and Rj are said to be adjacent if their union forms a

connected set.
• Regions that are not to be adjacent are said to be disjoint.
Basic Relationships Between Pixels

• Boundary (or border)

 The boundary of the region R is the set of pixels in the region that have
one or more neighbors that are not in R.
 If R happens to be an entire image, then its boundary is defined as the
set of pixels in the first and last rows and columns of the image.

• Foreground and background

 An image contains K disjoint regions, Rk, k = 1, 2, …, K. Let Ru denote

the union of all the K regions, and let (Ru)c denote its complement.
All the points in Ru is called foreground;
All the points in (Ru)c is called background.
Distance Measures
• Given pixels p, q and z with coordinates (x, y), (s, t),
(u, v) respectively, the distance function D has
following properties:

a. D(p, q) ≥ 0 [D(p, q) = 0, iff p = q]

b. D(p, q) = D(q, p)

c. D(p, z) ≤ D(p, q) + D(q, z)

Dm distance: is defined as the shortest m-path between the points.

Distance Measures q (s,t)

a. Euclidean Distance :
De(p, q) = [(x-s)2 + (y-t)2]1/2

p (x,y)

b. City Block Distance: q

D4(p, q) = |x-s| + |y-t|

p
c. Chess Board Distance:
D8(p, q) = max(|x-s|, |y-t|)
 Introduction to Mathematical Operations in DIP

• Array vs. Matrix Operation

a11 a12  b11 b12 

A   B
a21 a22  
Array
product  21 22 
b b
operator
a11b11 a12b12 
A .* B    Array product

Matrix
 21 21 22 22 
a b a b
product
operator
a11b11  a12b21 a11b12  a12b22 
A*B 
Matrix product
 a b a b 
 21 11 22 21 21 12 22 22 
a b a b
Introduction to Mathematical Operations in DIP

• Linear vs. Nonlinear Operation

H  f (x, y)  g(x, y)

H ai fi (x, y)  aj f j (x, y)
 H ai fi (x, y)  H aj f j (x, y)
Additivity

 ai H  fi (x, y)  aj H  f j (x, y) Homogeneity

 ai gi (x, y)  aj g j (x, y)
H is said to be a linear operator;
H is said to be a nonlinear operator if it does not meet the above
qualification.
Arithmetic Operations

• Arithmetic operations between images are array

operations. The four arithmetic operations are denoted
as

s(x,y) = f(x,y) + g(x,y)

d(x,y) = f(x,y) – g(x,y)
p(x,y) = f(x,y) × g(x,y)
v(x,y) = f(x,y) ÷ g(x,y)
Example: Addition of Noisy Images for Noise Reduction

Noiseless image: f(x,y)

Noise: n(x,y) (at every pair of coordinates (x,y), the noise is uncorrelated
and has zero average value)
Corrupted image: g(x,y)
g(x,y) = f(x,y) + n(x,y)

Reducing the noise by adding a set of noisy images,

g(x, y)  gi (x, y)

{gi(x,y)} K
1
K i1
Example: Addition of Noisy Images for Noise Reduction

g(x, y)  gi (x, y)

1 K
K i1
1 K 
Eg(x, y)  E  gi (x, y) 
2
 1 K 2
 K i1  g ( x, y )
 g ( x, y )
Ki1 i
1 K 
 E   f (x, y)  ni (x, y) 1 2
 K i1   1 K 2
 n(x, y)
1 K  K
 f (x, y)  E  ni (x, y)
 n ( x, y )
Ki1 i

 K i1 
 f (x, y)
Example: Addition of Noisy Images for Noise Reduction

► In astronomy, imaging under very low light levels

frequently causes sensor noise to render single
images virtually useless for analysis.

► In astronomical observations, similar sensors for noise

reduction by observing the same scene over long
periods of time. Image averaging is then used to
reduce the noise.
An Example of Image Subtraction: Mask Mode Radiography

Mask h(x,y): an X-ray image of a region of a patient’s body

Live images f(x,y): X-ray images captured at TV rates after injection of

the contrast medium

Enhanced detail g(x,y)

g(x,y) = f(x,y) - h(x,y)

The procedure gives a movie showing how the contrast medium

propagates through the various arteries in the area being observed.
An Example of Image Multiplication
Spatial Operations

• Performed directly on the pixels of a given image

• Single-pixel operations
Alter the values of an image’s pixels based on the intensity.
s  T(z)

e.g.,
Spatial Operations
• Neighborhood operations

The value of this pixel is determined

by a specified operation involving the
pixels in the input image with
coordinates in Sxy
Geometric Spatial Transformations

• Geometric transformation (rubber-sheet transformation)

— A spatial transformation of coordinates

(x, y)  T{(v, w)}

— intensity interpolation that assigns intensity values to the spatially
transformed pixels.

• Affine transform
t11 t12 0

x y 1  v w 1t21 t22 0 
t31 t32 1
Intensity Assignment

• Forward Mapping

(x, y)  T{(v, w)}

It’s possible that two or more pixels can be transformed to the same
location in the output image.

• Inverse Mapping

(v, w)  T1{(x, y)}

The nearest input pixels to determine the intensity of the output pixel
value.
Inverse mappings are more efficient to implement than forward
mappings.
Example: Image Rotation and Intensity Interpolation
Image Registration
• Input and output images are available but the
transformation function is unknown.
Goal: estimate the transformation function and use it to
register the two images.

• One of the principal approaches for image registration is

to use tie points (also called control points)
 The corresponding points are known precisely in the
input and output (reference) images.
Image Registration
• A simple model based on bilinear approximation:

 x  cv
1  c2w c3vw c4
y  c v  c w c vw c
 5 6 7 8

Where (v, w) and (x, y) are the coordinates of

tie points in the input and reference images.
Image Registration
Image Transform
• A particularly important class of 2-D linear transforms,
denoted T(u, v)
M 1 N1
T(u, v)   f (x, y)r(x, y, u, v)
x0 y0

where f (x, y) is the input image,

r(x, y, u, v) is the forward transformation ker nel,
variables u and v are the transform variables,
u = 0, 1, 2, ..., M-1 and v = 0, 1, ..., N-1.
Image Transform
• Given T(u, v), the original image f(x, y) can be recovered
using the inverse transformation of T(u, v).

M1 N1
f (x, y)  T(u, v)s(x, y, u, v)
u0 v0
where s(x, y, u, v) is the inverse transformation ker nel,
x = 0, 1, 2, ..., M-1 and y = 0, 1, ..., N-1.
Image Transform
Example: Image Denoising
Forward Transform Kernel
M 1 N1
T(u, v)   f (x, y)r(x, y, u, v)
x0 y0

The kernel r(x, y, u, v) is said to be SEPERABLE if

r(x, y, u, v)  r1(x, u)r2 ( y, v)

In addition, the kernel is said to be SYMMETRIC if

r1(x, u) is functionally equal to r2 ( y, v), so that
r(x, y, u, v)  r1(x, u)r1( y, u)
The Kernels for 2-D Fourier Transform
The forward kernel
r(x, y, u, v)  e j 2 (ux/ M vy/ N )

Where j= 1

The inverse kernel

1 j 2 (ux/Mvy/ N)
s(x, y, u, v)  e
MN
2-D Fourier Transform

M 1 N1
T(u, v)   f (x, y)e  j 2 (ux/ M vy/ N )

x0 y0

1 M1 N1
f (x, y)  
MN u0 v0
T(u, v)e j 2 (ux/ M vy/ N )
Probabilistic Methods
Let zi , i  0, 1, 2, ..., L -1, denote the values of all possible intensities
in an M  N digital image. The probability, p(zk ), of intensity level
zk occurring in a given image is estimated as
nk
p(zk )  ,
MN
where nk is the number of times that intensity zk occurs in the image.
L1
p(z ) 1
k 0
k

The mean (average) intensity is given by

L1
m = zk p(zk )
k0
Probabilistic Methods
The variance of the intensities is given by
L1
 = (zk  m) p(zk )
2 2

k0

The nth moment of the intensity variable z is

L1
un (z) = (zk  m) p(zk )
n

k 0
Example: Comparison of Standard Deviation Values

  31.6   49.2

 14.3 31.6 49.2