Segmentation?

Object delineation,
i.e. finding the contour / surface of the
object of interest
H06W8a: Medical image analysis
Class 3: Low-level segmentation Pixel / voxel classification,
i.e. assigning a (tissue class) label to each
pixel in the image (or more general: the
Prof. Frederik Maes probability for a pixel to belong to a
particular class)
[email protected]
Image partitioning,
i.e. dividing the image in non-overlapping
regions with distinct properties
(e.g. homogeneous intensity / texture)

Segmentation challenges Manual delineation

• imaging ambiguity (noise, limited contrast, …)
• large diversity of objects and images
• complex 3D shapes
• large variability within each object class
• pathological objects
• unknown shape and photometry Example: Right atrium of the
heart in cardiac MRI

2D delineation on consecutive slices è stack of contours in 3D ?

? Each slice delineated independently è 3D inconsistencies…
Anisotropic resolution (slice spacing > pixel size) è missing parts…

! ?

2D contours è 3D surface Manual segmentation

Axial Axial Session 1 Session 2
Coronal • tedious, time-consuming è usually
not practical in clinical routine (except
for RT planning)
Observer 1
• inter-/intra-observer variability è
clinically significant versus non-
significant differences?
• slice-by-slice 2D segmentation induces
3D inconsistencies Observer 2
• large, complex structures can’t be
1) 3D pointcloud based on a single stack of 2) Triangulated surface mesh created by reliably segmented by pure manual
contours or multiple stacks delineated in interpolation of 3D point cloud (cfr. delineation
images with different orientation (assuming computer graphics), visualized as a 3D Observer 3
correct registration of the images) shaded surface (surface rendering)
 Segmentation strategies Low-level segmentation strategies

• “Low-level” approaches: • Thresholding

– Based on generic, heuristic, simplistic assumptions with no or little • Morphological operators
specific a priori information about the object
• Edge detection
– Requires sufficient contrast, intensity homogeneity, no artefacts
• Region growing
– Limited applicability in medical imaging
• Watershed transform
• Model-based
– Prior knowledge of object appearance (photometry, geometry, context) • Interactive segmentation
– Must be flexible to deal with object variability (normal and pathology)
– Statistical models: learned from training data
– Different strategies:
• Deformable shape models
• Atlas-based segmentation
• Classification

Thresholding Contour extraction

Original CT slice: 0-2500 Intensity profile along line P
Binary image (0 / 1) Boundary positions
Background
Skull
Skull
Skull

P It Local
Soft tissue operator
Soft tissue Back- Back-
ground ground

Histogram Thresholded image: 0 or 1

Back- It It = selected
ground threshold value
Soft tissue

I ≥ It è 1
I < It è 0
Skull
Contour = sequence of connected Smoothed contour: connect midpoints
pixel edges (‘edgels’) of each edgel (instead of endpoints)

Surface extraction: marching squares Surface extraction: marching cubes

• Problem: extract the surface of an object specified by a threshold It

• Solution: the ‘marching squares’ algorithm (‘marching cubes’ in 3D) • In 2D: 4 neighbours
• Combines thresholding (binarization), boundary extraction and boundary è line segments
smoothing in one procedure, instead of consecutive steps
• Principle in 2D:
• In 3D: 8 neighours
• Inspect intensity values in adjacent pixels è surface patches
I1<It I2>It (p1,p2,p3,p4)
p1 q1 p2 • Consider pixels p1,p2 with values I1,I2 • Limited number of prespecified
below and above the threshold topologies possible
• Find boundary point location by linear (depending on pixel values)
interpolation:
q2 q1 = (1-a)*p1+a*p2 with a = (It-I1)/(I2-I1)
• Idem for p2,p4 è q2
p3 p4 • Add boundary segment (q1,q2)
• Repeat for every set (p1,p2,p3,p4)
I3<It I4<It
 Application: bone segmentation from CT Optimal threshold selection?

• Empirically: trial and error (e.g. interactive)

• Prior knowledge (e.g. based on similar previous images)
• Gradient-magnitude based
• Histogram-based
• Model-based threshold selection

It too low It about right It too high

Caveat: post-processing needed, e.g. to

separate different bone structures…
Oversegmentation Undersegmentation
(‘too much’) (‘not enough’)

Imaging an ideal step edge Image gradient operator

Ideal step edge Point-spread function Imaged step edge
-1 -2 -1 -1 0 1
W = 3 x3 Sobel edge filter = 0 0 0 or -2 0 2
1 2 1 -1 0 1
« à

vertical horizontal
edges edges
Gradient magnitude
Original Filtered (edge strength)
Image gradient Image Laplacian Heuristic observation: object boundary
corresponds to a local intensity change
or ‘edge’

If PSF is symmetric: true edge location

coincides with
• maximum of gradient magnitude
• zero crossing of Laplacian

(first derivative) (second derivative) Fx = dI/dx Fy = dI/dy F = sqrt(Fx2 + Fy2)

Gradient magnitude image Gradient-based threshold selection

P Gradient max
magnitude

Window 1: strong edges Window 2: weak edges

It
Intensity along P Histogram of gradient magnitude
Gradient magnitude
Strong Many spurious,
edges weak edges B(It) overlaid on Select intensity threshold It such that thresholded object
(due to noise) gradient
No obvious threshold boundary B(It) coincides with strong edges as much as
between spurious and magnitude possible, i.e. such that mean gradient magnitude is maximal
relevant edges
along the boundary

Spurious edges Few relevant, strong edges

 Histogram-based: Otsu’s method Example: mouse bone segmentation
t i max
Histogram split in 2 classes (object, n1 (t ) = å h(i ) n2 (t ) = å h(i) N = n1 (t ) + n2 (t ) Reconstructed image (2 μm) Global threshold
background) by threshold t: i =1 i =t +1 SkyScan 1172
w1 = n1 / N w2 = n2 / N
æ t
ö
µ1 (t ) = ç å h(i ).I (i ) ÷ / n1 (t )
è i =1 ø
æ i max ö
µ 2 (t ) = ç å h(i ).I (i ) ÷ / n2 (t )
è i =t +1 ø
æ t
ö trabecular bone
s 1 (t ) = ç å h(i ).I 2 (i ) ÷ / n1 (t ) - µ12 (t )
2
t è i =1 ø
Ex vivo microCT imaging
(excised bone sample)
æ i max
ö
n1,w1,µ1,s1 n2,w2,µ2,s2 s22 (t ) = ç å h(i ).I 2 (i ) ÷ / n2 (t ) - µ 22 (t )
è i =t +1 ø
BV/TV 14.86%
Select threshold t such that within-class
variance is minimal, i.e. between-class SMI 2.08
within-class variance cortical bone
variance is maximal:
Tb.N 125
topt = arg maxt s2 Segmented image
b(t) between-class variance Bone characteristics
(s2 = total variance = independent of t) 3D model
e.g. Animal model of osteoporosis vs. wild type animal

Example: mouse bone segmentation Example: mouse bone segmentation

Reconstructed image (10 μm) Global threshold
Histogram SkyScan 1076

Optimal threshold In vivo microCT imaging Otsu’s method

Original ex-vivo CT slice (living animal)

Otsu’s method
Image degradation due to lower dose, soft tissue, scanner geometry

s2b(t)

?
topt Segmented image
Registered ex vivo and in vivo microCT images

Example: mouse bone segmentation Example: mouse bone segmentation

Adaptive threshold ?
Higher threshold

Loss of trabeculae
Global threshold ?

Local threshold
= 0.5*(local minimum + local maximum)

Lower threshold
Caveat: overestimates width of thin
trabeculae… è heuristic adjustment needed
Thickening of cortex
Better solution: iterative CT reconstruction
methods, taking discrete nature of the
intensity histogram into account In vivo, adaptive Ex vivo, global
threshold threshold
 Model-based threshold selection Edge detection

• Histogram is modeled
as a mixture of Gaussian
distributions, one for
each class

•This allows to compute

the probability of each
class in a particualr
voxel given its intensity

• Threshold selected
according to maximum
likelihood criterion

Object boundary à local intensity change or ‘edge’ à large gradient magnitude

Thresholding the gradient magnitude image 1 Thresholding the gradient magnitude image 2

histogram histogram

Thresholding the gradient magnitude image 3 ‘Thinning’ the edge image

histogram

Remove boundary points until a single, one-pixel wide line remains

( = morphological operator)
 Cardiac MR image
Example Hysteresis thresholding

Global thresholding

Gradient magnitude
Threshold = 0.01 Threshold = 0.045

• Uses a double threshold t2 > t1

• Criterion:
Threshold = 0.01 Threshold = 0.045 • Gradient magnitude > t2 ?
• Mark pixel as “definitely an edge pixel”
• Gradient magnitude < t1 ?
Select strongest edges only, remove ‘noisy’ edges • Mark pixel as “definitely not an edge pixel”
• Gradient magnitude < t2 but > t1 ?
Threshold selection ? • Mark as edge pixel if adjacent to another edge pixel
Does not guarantee closed contours … (edge linking?) • Evaluate criterion for all not marked pixels until convergence
Hysteresis thresholding

Canny edge detection Canny edge detection

• 1: Smoothing with Gaussian kernel

An edge detection scheme with (theoretically) optimal

detection and localization performance

Canny edge detection Canny edge detection

• 2: Gradient magnitude and direction • 3: Non-maximum suppression:

select the point along the normal with maximal gradient
 Canny edge detection Canny edge detection: example

• 4: Hysteresis thresholding
Canny edges are not always closed !

50 50 50

100 100 100

150 150 150

200 200 200

250
250 250

300
300 300

High threshold Low threshold Result

Canny edge detection:

Region-based segmentation
influence of smoothing
• Segment into “coherent” regions based on pixel similarities
• Guaranteed to produce closed region boundaries (>< edge detection)
• Works from inside out, instead of outside in
• Objective: produce regions that are
– as large as possible and as few as needed
– “coherent “, but allowing for internal variation
• Inherent tradeoff:
– Over-segmentation: too similar, too small, too many regions
– Under-segmentation: too dissimilar, too large, too few regions

Smoothing can displace the edges

Region growing: principle Region growing: example

• Define a similarity measure S(i, j) for pixels i and j and a threshold T. Segmentation of the air-inflated colon for virtual colonoscopy based on CT
• Initiate the region R from a single seed pixel p (in 2D) or voxel (in 3D). seed point
• Consider each pixel q that is not in R but adjacent to a pixel r in R.
• If S(p,q)>T, add q to R.
• Iterate until R does not expand anymore.

Colon is segmented as a single

connected region in 3D
 Region growing: implementation Watershed regions

• Different features can be used to assess similarity S between pixels: • A watershed region or catchment basin is defined as the region in which
– intensity (color), texture, other derived features
water flows “downhill” to a common point.
• Threshold T:
– constant? updated during iterations? • Watersheds of gradient magnitude make useful region-based
– If too small è over-segmentation, if too large è under-segmentation segmentation primitives.
• Different comparisons possible:
– Compare q to seed pixel p: S(p,q) (sensitive to noise, choice of p …)
– Compare q to neighboring pixel r : S(r,q) (sensitive to drift …)
– Cumulative cost: S(p,q) + S(r,q)
– Compare q to region statistics (region mean and variance …)
• Choice of seed point p
– Single seed point for single region
– Multiple seed points for single region
• Considered one by one + subsequent region merging
• Considered jointly, e.g. to account for variability within the region of interest
– Multiple seed points for different regions
• Multiple regions grown simultaneously (‘region competition’)

Watershed computation Watershed: example

• Work on the gradient magnitude image .

• Link each pixel to the smallest of its neighbors.
• If no smaller neighbors, these are region seeds.
• All pixels that “flow” downhill (smallest neighbor) to the same point form
a single region.

Watershed: example Region merging

• Start with a heavily over-segmented image (or individual pixels)

• Evaluate similarity of neighbouring regions based on region statistics, e.g.
differences in mean region intensity compared to variance in each region
• Reduce the number of regions by merging neighbouring regions that are
sufficiently similar, assuming these belong to the same object
• Repeat until no more regions can be merged

Canny edge detection Watershed regions

 Example: Histogram-based similarity measure
Hierarchical representation
for region merging
Intensity histogram Intensity histogram
1 region

tree of hierarchically connected primitives

i
j

Union i+j

regions i and j with ‘similar’ intensity regions i and j with ‘dissimilar’

have ‘similar’ histogram intensity have ‘dissimilar’ histogram
è knowledge of region label i or j è knowledge of region label i or j
provides little additional provides significant additional
information about the region information about the region can be used for efficient user interaction
intensity è MI(L,I) is low è intensity è MI(L,I) is high è regions
regions can be merged should not be merged
N regions
MI(L,I) = mutual information between region label L and region intensity I
= H(I) – H(I|L) ~ H(I(i+j))*(ni+nj) – (H(Ii)*ni + H(Ij)*nj) merge until a single region remains
with H(Ir) entropy of intensities in region r and nr number of pixels

Interaction Interaction
‘Paintbrush’ ‘Point and click’

user drags cursor over object to be segmented, each region hit by the
mouse cursor is selected and added to the segmentation user indicates points that do or do not belong to the object of interest,
segmented region boundary is updated on the fly

hidden visible
Algorithm searches for region in region hierarchy that
best fits specified constraints…
- +

3-D propagation 3-D primitives

100

Primitives
20

Interaction
on one slice +
Propagation to other slices
 Morphological operations Connected component labeling

• Operations on a binary image that influence the Original Connected components Largest component
morphology (shape) of the object of interest
• Example:
– eliminating isolated pixels
– boundary smoothing
– filling holes
– connected component analysis
– …

Connectivity depends on
neighborhood system

Basic operations: erosion Erosion: example

Original Eroded 1 x Eroded 2 x

1 1
1

minimum

0
iterate

Remove pixel from object if at least one neighbor is background pixel

(which neighbors depends on chosen neighborhood system)

Basic operations: dilation Dilation: example

0 Original Dilated 1 x Dilated 2 x

1 0

maximum

Add pixel to object if at least one neighbor is object pixel

(which neighbors depends on chosen neighborhood system)
 Combining operations: Closing Combining operations: Opening

Original First dilation… … then erosion Original First erosion… … then dilation

Example: hole filling Example: hole filling

‘hole’ connected
with background

isolated hole

Example: hole filling Low-level segmentation: Conclusion

• Principle:
Original Dilation Background fill
– Detect (and link) image edges
or
– Combine pixels with similar global or local photometric characteristics
• No model of global object shape or context
• Simple, fast, general operators (cfr. Photoshop)
• Heuristic, often too simplistic
• Limited use for complex objects in complex medical images
– Bone segmentation in CT using thresholding
Invert Add original Erode