Cluster Analysis
Basic Concepts and Algorithms
What is Cluster Analysis?
Cluster: a collection of data objects
– Similar to one another within the same cluster
– Dissimilar to the objects in other clusters
Cluster analysis
– Grouping a set of data objects into clusters
Inter-cluster
Intra-cluster distances
distances are are
minimized maximized
Applications of Cluster Analysis
Understanding Discovered Clusters Industry Group
1
Applied-Matl-DOWN,Bay-Network-Down,3-COM-DOWN,
– Information Retrieval Cabletron-Sys-DOWN,CISCO-DOWN,HP-DOWN,
DSC-Comm-DOWN,INTEL-DOWN,LSI-Logic-DOWN,
Micron-Tech-DOWN,Texas-Inst-Down,Tellabs-Inc-Down,
Technology1-DOWN
Natl-Semiconduct-DOWN,Oracl-DOWN,SGI-DOWN,
Group related documents for Sun-DOWN
2
Apple-Comp-DOWN,Autodesk-DOWN,DEC-DOWN,
ADV-Micro-Device-DOWN,Andrew-Corp-DOWN,
browsing Computer-Assoc-DOWN,Circuit-City-DOWN,
Compaq-DOWN, EMC-Corp-DOWN, Gen-Inst-DOWN,
Technology2-DOWN
Motorola-DOWN,Microsoft-DOWN,Scientific-Atl-DOWN
– Finance
3
Fannie-Mae-DOWN,Fed-Home-Loan-DOWN,
MBNA-Corp-DOWN,Morgan-Stanley-DOWN Financial-DOWN
Group stocks with similar price 4
Baker-Hughes-UP,Dresser-Inds-UP,Halliburton-HLD-UP,
Louisiana-Land-UP,Phillips-Petro-UP,Unocal-UP, Oil-UP
Schlumberger-UP
fluctuations
– Biology
Group genes and proteins that have
similar functionality
– Marketing
Help marketers discover distinct
groups in their customer bases, and
develop targeted marketing programs
Clustering gene expression data
Applications of Cluster Analysis
Image segmentation
– Goal: Break up the image
into meaningful or
perceptually similar regions
Summarization
– Reduce the size of large
data sets
Clustering precipitation in
Australia
Notion of a Cluster can be Ambiguous
How many clusters? Six Clusters
Two Clusters Four Clusters
Clustering results are crucially dependent on the measure of similarity (or distance) between
“points” to be clustered
Measure the Quality of Clustering
Quality of clustering:
– There is usually a separate “quality” function that
measures the “goodness” of a cluster.
– It is hard to define “similar enough” or “good
enough”
The answer is typically highly subjective
6
Considerations for Cluster Analysis
Partitioning criteria
– Single level vs. hierarchical partitioning (often, multi-level hierarchical
partitioning is desirable)
Separation of clusters
– Exclusive (e.g., one customer belongs to only one region) vs. non-
exclusive (e.g., one document may belong to more than one class)
Similarity measure
– Distance-based (e.g., Euclidian, road network, vector) vs.
connectivity-
based (e.g., density or contiguity)
Clustering space (Partial versus complete)
– Full space (often when low dimensional) vs. subspaces (often in high-
dimensional clustering)
Heterogeneous versus homogeneous
– Cluster of widely different sizes, shapes, and densities
7
Types of Clusters
Well-separated clusters
Center-based clusters
Contiguous clusters
Density-based clusters
Property or Conceptual
Described by an Objective Function
Types of Clusters: Well-Separated
Well-Separated Clusters:
– A cluster is a set of points such that any point in a cluster is
closer (or more similar) to every point in the cluster than to any
point not in the cluster.
3 well-separated clusters
Types of Clusters: Center-Based
Center-based
– A cluster is a set of objects such that an object in a
cluster is closer (more similar) to the “center” of a cluster,
than to the center of any other cluster
– The center of a cluster is often a centroid
4 center-based clusters
Types of Clusters: Contiguity-Based
Contiguous Cluster (Nearest neighbor or Transitive)
– A cluster is a set of points such that a point in a cluster is
closer (or more similar) to one or more other points in the
cluster than to any point not in the cluster.
Types of Clusters: Density-Based
Density-based
– A cluster is a dense region of points, which is separated by
low-density regions, from other regions of high density.
– Used when the clusters are irregular or intertwined, and when
noise and outliers are present.
6 density-based clusters
Types of Clusters: Conceptual Clusters
Shared Property or Conceptual Clusters
– Finds clusters that share some common property or represent
a particular concept.
.
2 Overlapping Circles
Characteristics of the Input Data Are Important
Type of proximity or density measure
– This is a derived measure, but central to clustering
Sparseness
– Dictates type of similarity
– Adds to efficiency
Attribute type
– Dictates type of similarity
Type of Data
– Dictates type of similarity
– Other characteristics, e.g., autocorrelation
Dimensionality
Noise and Outliers
Type of Distribution
Similarity and
Dissimilarity
Type of data in clustering analysis
Interval-scaled variables
Binary variables
Nominal, ordinal, and ratio variables
Variables of mixed types
– Nominal attribute: distinctness
– Ordinal attribute: distinctness & order
– Interval attribute: distinctness, order & addition
– Ratio attribute: all 4 properties
Types of Attributes
There are different types of attributes
– Nominal
Examples: ID numbers, eye color, zip codes
– Ordinal
Examples: rankings (e.g., taste of potato chips on a scale
from 1-10), grades, height in {tall, medium, short}
– Interval
Examples: calendar dates, temperatures in Celsius or
Fahrenheit.
– Ratio
Examples: temperature in Kelvin, length, time, counts
Attribute Description Examples Operations
Type
Nominal The values of a nominal attribute are zip codes, employee ID mode, entropy,
just different names, i.e., nominal numbers, eye color, sex: contingency
attributes provide only enough {male, female} correlation, 2 test
information to distinguish one object
from another. (=, )
Ordinal The values of an ordinal attribute hardness of minerals, median, percentiles,
provide enough information to order {good, better, best}, rank correlation,
objects. (<, >) grades, street numbers run tests, sign tests
Interval For interval attributes, the differences calendar dates, mean, standard
between values are meaningful, i.e., temperature in Celsius deviation, Pearson's
a unit of measurement exists. or Fahrenheit correlation, t and F
(+, - ) tests
Ratio For ratio variables, both differences temperature in Kelvin, geometric mean,
and ratios are meaningful. (*, /) monetary quantities, harmonic mean,
counts, age, mass, percent
length, electrical current variation
Attribute Transformation Comments
Level
Nominal Any permutation of values If all employee ID numbers
were reassigned, would it
make any difference?
Ordinal An order preserving change of An attribute encompassing
values, i.e., the notion of good, better best
new_value = f(old_value) can be represented equally
where f is a monotonic well by the values {1, 2, 3} or
function. by { 0.5, 1, 10}.
Interval new_value =a * old_value + b where Thus, the Fahrenheit and
a and b are constants Celsius temperature scales
differ in terms of where their
zero value is and the size of a
unit (degree).
Ratio new_value = a * old_value Length can be measured in
meters or feet.
Similarity and Dissimilarity
Similarity
– Numerical measure of how alike two data objects are.
– Is higher when objects are more alike.
– Often falls in the range [0,1]
Dissimilarity
– Numerical measure of how different are two data
objects
– Lower when objects are more alike
– Minimum dissimilarity is often 0
– Upper limit varies
Proximity refers to a similarity or dissimilarity
Similarity/Dissimilarity for Simple Attributes
p and q are the attribute values for two data objects.
Distance Measures
Manhattan Distance
Euclidean Distance
n
2
dist ( pk q k)
k 1
Where n is the number of dimensions (attributes) and pk and qk
are, respectively, the kth attributes (components) or data
objects p and q.
Manhattan Distance
3
point x y
2 p1 p1 0 2
p3 p4 p2 2 0
1 p3 3 1
p2 p4 5 1
0
0 1 2 3 4 5 6
Acutal Points in 2D
Distance between p1 and p2 Manhattan Distance is the sum of
(X1,Y1) = (0 , 2) The absolute values of differences
(X2,Y2) = (2 , 0) Of the coordinates.
d=|0–2|+|2–0|
d= 2+2 =4
Distance between p1 and p3
(X1,Y1) = (0 , 2)
(X2,Y2) = (3 , 1)
d=|0–3|+|2–1|
d= 3+1 = 4 L1 p1 P2 p3 p4
Distance between p1 and p4 p1 0 4 4 6
(X1,Y1) = (0 , 2) p2 4 0 2 4
(X4,Y4) = (5 , 1) p3 4 2 0 2
d=|0–5|+|2–1| p4 6 4 2 0
d= 5+1 =6
Euclidean Distance
point x y
3 p1 0 2
p2 2 0
2 p1 p3 3 1
p3 p4 p4 5 1
1
p2 Acutal Points in 2D
0
0 1 2 3 4 5 6
p1 p2 p3 p4
p1 0 2.828 3.162 5.099
p2 2.828 0 1.414 3.162
p3 3.162 1.414 0 2
p4 5.099 3.162 2 0
Distance Matrix
Clustering Algorithms
K-means and its variants
Hierarchical clustering
Density-based clustering
Partitional Clustering
Divide data objects into non-overlapping subsets (clusters) such that
each data object is in exactly one subset
Typical methods: k-means, k-medoids, CLARANS
Original Points A Partitional Clustering
Hierarchical Clustering
A set of nested clusters organized as a hierarchical tree
p1
p3 p4
p2
p1 p2 p3
Traditional Hierarchical Clustering p4
Traditional Dendrogram
p1
p3 p4
p2
p1 p2
p3 p4
Non-traditional Hierarchical Clustering
Non-traditional Dendrogram
K-Means : Partitioning approach
An iterative clustering algorithm
Each cluster is associated with a centroid (center point)
Each point is assigned to the cluster with the closest centroid
Number of clusters, K, must be specified
Initialize: Pick K random points as cluster
centers
Repeat:
1.Assign data points to closest
cluster center
2.Change the cluster center to the
average of its assigned points
Until The centroids don’t change
K-Means : Partitioning approach
An iterative clustering algorithm
Each cluster is associated with a centroid (center point)
Each point is assigned to the cluster with the closest centroid
Number of clusters, K, must be specified
Initialize: Pick K random points as cluster
centers
Repeat:
1.Assign data points to closest
cluster center
2.Change the cluster center to the
average of its assigned points
Until The centroids don’t change
K-means clustering Example
K-means clustering Example
Iteration 1
Iterative Step 2:
Iterative Step 1: Change the cluster
Assign data points to center to average of the
closest cluster center assigned points
K-means clustering Example
Iteration 2
Iterative Step 1:
Assign data points to
closest cluster center
Repeat until convergence Iterative Step 2:
Change the cluster
center to average of the
assigned points
K-means clustering Example
Iteration 3
Iterative Step 1:
Assign data points to
closest cluster center
Repeat until convergence Iterative Step 2:
Change the cluster
center to average of the
assigned points
K-means Clustering – Details
Initial centroids are often chosen randomly.
– Clusters produced vary from one run to another.
The centroid is (typically) the mean of the points in the
cluster.
‘Closeness’ is measured by Euclidean distance, cosine
similarity, correlation, etc.
– K-means will converge for common similarity measures mentioned
above.
K-means Clustering – Details
Most of the convergence happens in the first few
iterations.
– Often the stopping condition is changed to ‘Until relatively few
points change clusters’
Complexity is O( n * K * I )
– n = number of points,
– K = number of clusters,
– I = number of iterations
Two different K-means Clusterings
3
2.5
2
Original Points
1.5
y
1
0.5
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
x
3 3
2.5 2.5
2 2
1.5 1.5
y
y
1 1
0.5 0.5
0 0
-2 -1.5 -1 -0.5 0 0.5 1 2 -2 -1.5 -1 -0.5 0 0.5 1 1.5
1.5 2
x x
Optimal Clustering Sub-optimal Clustering
Importance of Choosing Initial Centroids
Iteration 1 Iteration 2 Iteration 3
3 3 3
2.5 2.5 2.5
2 2 2
1.5 1.5 1.5
y
y
1 1 1
0.5 0.5 0.5
0 0 0
-2 -1.5 -1 -0.5 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0.5 1 1.5 2
0 0 0
x x x
Iteration 4 Iteration 5 Iteration 6
3 3 3
2.5 2.5 2.5
2 2 2
1.5 1.5 1.5
y
y
1 1 1
0.5 0.5 0.5
0 0 0
-2 -1.5 -1 -0.5 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0.5 1 1.5 2
0 0 0
x x x
Importance of Choosing Initial Centroids
Iteration 6
2
3
4
5
1
3
2.5
1.5
1.5
y
y
0.5
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
-
2 x
-
1
.
5
-
1
Importance of Choosing Initial Centroids …
Iteration 1 Iteration 2
3 3
2.5 2.5
2 2
1.5 1.5
y
y
1 1
0.5 0.5
0 0
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2
x x
Iteration 3 Iteration 4 Iteration 5
3 3 3
2.5 2.5 2.5
2 2 2
1.5 1.5 1.5
y
y
1 1 1
0.5 0.5 0.5
0 0 0
-2 -1.5 -1 -0.5 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0.5 1 1.5 2
0 0 0
x x x
Importance of Choosing Initial Centroids …
Iteration 5
1
2
3
4
3
2.5
1.5
1.5
y
y
0.5
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
-
2 x
-
1
.
5
-
1
Evaluating K-means Clusters
Most common measure is Sum of Squared Error (SSE)
– For each point, the error is the distance to the nearest cluster
– To get SSE, we square these errors and sum them.
K
SSE dist 2 (mi ,
i1 xC x) i
x is a data point in cluster Ci and mi is the representative point for cluster Ci
can show that mi corresponds to the center (mean) of the cluster
– Given two clusters, we can choose the one with the smallest error
– One easy way to reduce SSE is to increase K, the number of clusters
A good clustering with smaller K can have a lower SSE than a poor
clustering with higher K
Solutions to Initial Centroids Problem
Multiple runs
– Helps, but probability is not on your side
Sample and use hierarchical clustering to determine initial
centroids
Select more than k initial centroids and then select among
these initial centroids
– Select most widely separated
Postprocessing
Bisecting K-means
– Not as susceptible to initialization issues
Pre-processing and Post-processing
Pre-processing
– Normalize the data
– Eliminate outliers
Post-processing
– Eliminate small clusters that may represent outliers
– Split ‘loose’ clusters, i.e., clusters with relatively high SSE
– Merge clusters that are ‘close’ and that have relatively low SSE
Limitations of K-means
K-means has problems when clusters are of
different
– Sizes
– Densities
– Non-globular shapes
K-means has problems when the data contains
outliers.
Limitations of K-means: Differing Sizes
Original Points K-means (3 Clusters)
Limitations of K-means: Differing Density
Original Points K-means (3 Clusters)
Limitations of K-means: Non-globular Shapes
Original Points K-means (2 Clusters)
Overcoming K-means Limitations
Original Points K-means Clusters
Overcoming K-means Limitations
Original Points K-means Clusters
Density-Based Clustering Methods
Clustering based on density (local cluster criterion), such as density-
connected points
Major features:
– Discover clusters of arbitrary shape
– Handle noise
– One scan
– Need density parameters as termination condition
Several interesting studies:
– DBSCAN: Ester, et al. (KDD’96)
– OPTICS: Ankerst, et al (SIGMOD’99).
– DENCLUE: Hinneburg & D. Keim (KDD’98)
– CLIQUE: Agrawal, et al. (SIGMOD’98) (more grid-based)
132
DBSCAN: Density Based Spatial Clustering of Applications with Noise
Locates region of high density that are separated by regions of low
density.
In center- based approach, density of a point is the number of points
within specified radius, Eps, of that point
A cluster is defined as a maximal set of density-connected points
In center-based approach, we can classify
point as being
1) in the interior of dense region (core)
2) on the edge of a dense region (border)
Outlier
3) in a sparely occupied region (noise)
Border
Eps = 1cm
MinPts = 5
Cor e
DBSCAN
DBSCAN is a density-based algorithm.
– Density = number of points within a specified radius (Eps)
– A point is a core point if it has more than a specified number
of points (MinPts) within Eps
These are points that are at the interior of a cluster
– A border point has fewer than MinPts within Eps, but is in
the
neighborhood of a core point
– A noise point is any point that is not a core point or a border
point.
DBSCAN: Core, Border, and Noise Points
DBSCAN: The Algorithm
– Label all points as core, border or noise.
– Eliminate noise points.
– Put an edge between all core points that are within
Eps of each other.
– Make each group of connected core points into a
separate cluster.
– Assign each border points to one of the clusters of its
associated core points.
138
DBSCAN Algorithm
Eliminate noise points
Perform clustering on the remaining points
DBSCAN Algorithm
Time Complexity
– O(N x time to find points in Eps-neighbourhood)
– where N is the no of points
– Worst case O(N2)
– KD-trees, allow efficient retreivel of all points within
given distance of a specified point in O(N logN)
Space Complexity
– O(N)
DBSCAN: Core, Border and Noise Points
Original Points Point types: core,
border and noise
Eps = 10, MinPts = 4
When DBSCAN Works Well
Original Points Clusters
• Resistant to Noise
• Can handle clusters of different shapes and sizes
DBSCAN: Sensitive to Parameters
DBSCAN online Demo:
http://webdocs.cs.ualberta.ca/~yaling/Cluster/Applet/Code/Cluster.ht
ml
