Data Mining

Cluster Analysis: Basic Concepts

and Algorithms

Lecture Notes for Chapter 8

Introduction to Data Mining

by
Tan, Steinbach, Kumar

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 1

 What is Cluster Analysis?

 Finding groups of objects such that the objects in a group

will be similar (or related) to one another and different
from (or unrelated to) the objects in other groups

Inter-cluster
Intra-cluster distances are
distances are maximized
minimized

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›

 Applications of Cluster Analysis
Discovered Clusters Industry Group
 Understanding Applied-Matl-DOWN,Bay-Network-Down,3-COM-DOWN,

– Group related documents

1 Cabletron-Sys-DOWN,CISCO-DOWN,HP-DOWN,
DSC-Comm-DOWN,INTEL-DOWN,LSI-Logic-DOWN,
Micron-Tech-DOWN,Texas-Inst-Down,Tellabs-Inc-Down,
Technology1-DOWN
Natl-Semiconduct-DOWN,Oracl-DOWN,SGI-DOWN,
for browsing, group genes Sun-DOWN
Apple-Comp-DOWN,Autodesk-DOWN,DEC-DOWN,

and proteins that have 2 ADV-Micro-Device-DOWN,Andrew-Corp-DOWN,

Computer-Assoc-DOWN,Circuit-City-DOWN,
Compaq-DOWN, EMC-Corp-DOWN, Gen-Inst-DOWN,
Technology2-DOWN
similar functionality, or Motorola-DOWN,Microsoft-DOWN,Scientific-Atl-DOWN
Fannie-Mae-DOWN,Fed-Home-Loan-DOWN,
group stocks with similar 3 MBNA-Corp-DOWN,Morgan-Stanley-DOWN Financial-DOWN

price fluctuations Baker-Hughes-UP,Dresser-Inds-UP,Halliburton-HLD-UP,

4 Louisiana-Land-UP,Phillips-Petro-UP,Unocal-UP,
Schlumberger-UP
Oil-UP

 Summarization
– Reduce the size of large
data sets

Clustering precipitation
in Australia

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›

 What is not Cluster Analysis?

 Supervised classification
– Have class label information

 Simple segmentation
– Dividing students into different registration groups
alphabetically, by last name

 Results of a query
– Groupings are a result of an external specification

 Graph partitioning
– Some mutual relevance and synergy, but areas are not
identical

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›

 Notion of a Cluster can be Ambiguous

How many clusters? Six Clusters

Two Clusters Four Clusters

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›

 Types of Clusterings

 A clustering is a set of clusters

 Important distinction between hierarchical and

partitional sets of clusters

 Partitional Clustering
– A division data objects into non-overlapping subsets (clusters)
such that each data object is in exactly one subset

 Hierarchical clustering
– A set of nested clusters organized as a hierarchical tree

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›

 Partitional Clustering

Original Points A Partitional Clustering

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›

 Hierarchical Clustering

p1
p3 p4
p2
p1 p2 p3 p4
Traditional Hierarchical Clustering Traditional Dendrogram

p1
p3 p4
p2
p1 p2 p3 p4
Non-traditional Hierarchical Clustering Non-traditional Dendrogram

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›

 Other Distinctions Between Sets of Clusters

 Exclusive versus non-exclusive

– In non-exclusive clusterings, points may belong to multiple
clusters.
– Can represent multiple classes or ‘border’ points
 Fuzzy versus non-fuzzy
– In fuzzy clustering, a point belongs to every cluster with some
weight between 0 and 1
– Weights must sum to 1
– Probabilistic clustering has similar characteristics
 Partial versus complete
– In some cases, we only want to cluster some of the data
 Heterogeneous versus homogeneous
– Cluster of widely different sizes, shapes, and densities

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›

 Types of Clusters

 Well-separated clusters

 Center-based clusters

 Contiguous clusters

 Density-based clusters

 Property or Conceptual

 Described by an Objective Function

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›
 Types of Clusters: Well-Separated

 Well-Separated Clusters:
– A cluster is a set of points such that any point in a cluster is
closer (or more similar) to every other point in the cluster than
to any point not in the cluster.

3 well-separated clusters

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›

 Types of Clusters: Center-Based

 Center-based
– A cluster is a set of objects such that an object in a cluster is
closer (more similar) to the “center” of a cluster, than to the
center of any other cluster
– The center of a cluster is often a centroid, the average of all
the points in the cluster, or a medoid, the most “representative”
point of a cluster

4 center-based clusters

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›

 Types of Clusters: Contiguity-Based

 Contiguous Cluster (Nearest neighbor or

Transitive)
– A cluster is a set of points such that a point in a cluster is
closer (or more similar) to one or more other points in the
cluster than to any point not in the cluster.

8 contiguous clusters

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›

 Types of Clusters: Density-Based

 Density-based
– A cluster is a dense region of points, which is separated by
low-density regions, from other regions of high density.
– Used when the clusters are irregular or intertwined, and when
noise and outliers are present.

6 density-based clusters

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›

 Types of Clusters: Conceptual Clusters

 Shared Property or Conceptual Clusters

– Finds clusters that share some common property or represent
a particular concept.
.

2 Overlapping Circles

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›

 Types of Clusters: Objective Function

 Clusters Defined by an Objective Function

– Finds clusters that minimize or maximize an objective function.
– Enumerate all possible ways of dividing the points into clusters and
evaluate the `goodness' of each potential set of clusters by using
the given objective function. (NP Hard)
– Can have global or local objectives.
 Hierarchical clustering algorithms typically have local objectives
 Partitional algorithms typically have global objectives
– A variation of the global objective function approach is to fit the
data to a parameterized model.
 Parameters for the model are determined from the data.
 Mixture models assume that the data is a ‘mixture' of a number of
statistical distributions.

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›

 Types of Clusters: Objective Function …

 Map the clustering problem to a different domain

and solve a related problem in that domain
– Proximity matrix defines a weighted graph, where the
nodes are the points being clustered, and the
weighted edges represent the proximities between
points

– Clustering is equivalent to breaking the graph into

connected components, one for each cluster.

– Want to minimize the edge weight between clusters

and maximize the edge weight within clusters

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›

 Characteristics of the Input Data Are Important

 Type of proximity or density measure

– This is a derived measure, but central to clustering
 Sparseness
– Dictates type of similarity
– Adds to efficiency
 Attribute type
– Dictates type of similarity
 Type of Data
– Dictates type of similarity
– Other characteristics, e.g., autocorrelation
 Dimensionality
 Noise and Outliers
 Type of Distribution

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›

 Clustering Algorithms

 K-means and its variants

 Hierarchical clustering

 Density-based clustering

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›

 K-means Clustering

 Partitional clustering approach

 Each cluster is associated with a centroid (center point)
 Each point is assigned to the cluster with the closest
centroid
 Number of clusters, K, must be specified
 The basic algorithm is very simple

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›

 K-means Clustering – Details

 Initial centroids are often chosen randomly.

– Clusters produced vary from one run to another.
 The centroid is (typically) the mean of the points in the
cluster.
 ‘Closeness’ is measured by Euclidean distance, cosine
similarity, correlation, etc.
 K-means will converge for common similarity measures
mentioned above.
 Most of the convergence happens in the first few
iterations.
– Often the stopping condition is changed to ‘Until relatively few
points change clusters’
 Complexity is O( n * K * I * d )
– n = number of points, K = number of clusters,
I = number of iterations, d = number of attributes
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›
 Two different K-means Clusterings
3

2.5

2
Original Points
1.5

y
1

0.5

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2

x

3 3

2.5 2.5

2 2

1.5 1.5
y

y
1 1

0.5 0.5

0 0

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2

x x

Optimal Clustering Sub-optimal Clustering

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›
 Importance of Choosing Initial Centroids

Iteration 6
1
2
3
4
5
3

2.5

1.5
y

0.5

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2

x

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›

 Importance of Choosing Initial Centroids
Iteration 1 Iteration 2 Iteration 3
3 3 3

2.5 2.5 2.5

2 2 2

1.5 1.5 1.5

y

y
1 1 1

0.5 0.5 0.5

0 0 0

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2
x x x

Iteration 4 Iteration 5 Iteration 6

3 3 3

2.5 2.5 2.5

2 2 2

1.5 1.5 1.5

y

y
1 1 1

0.5 0.5 0.5

0 0 0

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2
x x x

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›

 Evaluating K-means Clusters

 Most common measure is Sum of Squared Error (SSE)

– For each point, the error is the distance to the nearest cluster
– To get SSE, we square these errors and sum them.
K
SSE    dist 2 (mi , x )
i 1 xCi

– x is a data point in cluster Ci and mi is the representative point for

cluster Ci
 can show that mi corresponds to the center (mean) of the cluster
– Given two clusters, we can choose the one with the smallest
error
– One easy way to reduce SSE is to increase K, the number of
clusters
 A good clustering with smaller K can have a lower SSE than a poor
clustering with higher K

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›

 Importance of Choosing Initial Centroids …

Iteration 5
1
2
3
4
3

2.5

1.5
y

0.5

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2

x

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›

 Importance of Choosing Initial Centroids …

Iteration 1 Iteration 2
3 3

2.5 2.5

2 2

1.5 1.5
y

y
1 1

0.5 0.5

0 0

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2

x x

Iteration 3 Iteration 4 Iteration 5

3 3 3

2.5 2.5 2.5

2 2 2

1.5 1.5 1.5

y

y
1 1 1

0.5 0.5 0.5

0 0 0

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2
x x x

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›

 Problems with Selecting Initial Points

 If there are K ‘real’ clusters then the chance of selecting

one centroid from each cluster is small.
– Chance is relatively small when K is large
– If clusters are the same size, n, then

– For example, if K = 10, then probability = 10!/1010 = 0.00036

– Sometimes the initial centroids will readjust themselves in
‘right’ way, and sometimes they don’t
– Consider an example of five pairs of clusters

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›

 10 Clusters Example
Iteration 4
1
2
3
8

2
y

-2

-4

-6

0 5 10 15 20
x
Starting with two initial centroids in one cluster of each pair of clusters
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›
 10 Clusters Example
Iteration 1 Iteration 2
8 8

6 6

4 4

2 2
y

y
0 0

-2 -2

-4 -4

-6 -6

0 5 10 15 20 0 5 10 15 20
x x
Iteration 3 Iteration 4
8 8

6 6

4 4

2 2
y

y
0 0

-2 -2

-4 -4

-6 -6

0 5 10 15 20 0 5 10 15 20
x x

Starting with two initial centroids in one cluster of each pair of clusters
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›
 10 Clusters Example
Iteration 4
1
2
3
8

2
y

-2

-4

-6

0 5 10 15 20
x
Starting with some pairs of clusters having three initial centroids, while other have only one.

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›

 10 Clusters Example
Iteration 1 Iteration 2
8 8

6 6

4 4

2 2
y

y
0 0

-2 -2

-4 -4

-6 -6

0 5 10 15 20 0 5 10 15 20
x
Iteration 3 x
Iteration 4
8 8

6 6

4 4

2 2
y

y
0 0

-2 -2

-4 -4

-6 -6

0 5 10 15 20 0 5 10 15 20
x x

Starting with some pairs of clusters having three initial centroids, while other have only one.

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›

 Solutions to Initial Centroids Problem

 Multiple runs
– Helps, but probability is not on your side
 Sample and use hierarchical clustering to
determine initial centroids
 Select more than k initial centroids and then
select among these initial centroids
– Select most widely separated
 Postprocessing
 Bisecting K-means
– Not as susceptible to initialization issues

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›

 Handling Empty Clusters

 Basic K-means algorithm can yield empty

clusters

 Several strategies
– Choose the point that contributes most to SSE
– Choose a point from the cluster with the highest SSE
– If there are several empty clusters, the above can be
repeated several times.

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›

 Updating Centers Incrementally

 In the basic K-means algorithm, centroids are

updated after all points are assigned to a centroid

 An alternative is to update the centroids after

each assignment (incremental approach)
– Each assignment updates zero or two centroids
– More expensive
– Introduces an order dependency
– Never get an empty cluster
– Can use “weights” to change the impact

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›

 Pre-processing and Post-processing

 Pre-processing
– Normalize the data
– Eliminate outliers

 Post-processing
– Eliminate small clusters that may represent outliers
– Split ‘loose’ clusters, i.e., clusters with relatively high
SSE
– Merge clusters that are ‘close’ and that have relatively
low SSE
– Can use these steps during the clustering process
 ISODATA

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›

 Bisecting K-means

 Bisecting K-means algorithm

– Variant of K-means that can produce a partitional or a
hierarchical clustering

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›

 Bisecting K-means Example

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›

 Limitations of K-means

 K-means has problems when clusters are of

differing
– Sizes
– Densities
– Non-globular shapes

 K-means has problems when the data contains

outliers.

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›

 Limitations of K-means: Differing Sizes

Original Points K-means (3 Clusters)

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›

 Limitations of K-means: Differing Density

Original Points K-means (3 Clusters)

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›

Limitations of K-means: Non-globular Shapes

Original Points K-means (2 Clusters)

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›

Overcoming K-means Limitations

Original Points K-means Clusters

One solution is to use many clusters.

Find parts of clusters, but need to put together.
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›
 Overcoming K-means Limitations

Original Points K-means Clusters

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›

Overcoming K-means Limitations

Original Points K-means Clusters

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›

 Hierarchical Clustering

 Produces a set of nested clusters organized as a

hierarchical tree
 Can be visualized as a dendrogram
– A tree like diagram that records the sequences of
merges or splits

6 5
0.2
4
3 4
0.15 2
5
2
0.1

1
0.05
3 1

0
1 3 2 5 4 6

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›

 Strengths of Hierarchical Clustering

 Do not have to assume any particular number of

clusters
– Any desired number of clusters can be obtained by
‘cutting’ the dendogram at the proper level

 They may correspond to meaningful taxonomies

– Example in biological sciences (e.g., animal kingdom,
phylogeny reconstruction, …)

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›

 Hierarchical Clustering

 Two main types of hierarchical clustering

– Agglomerative:
 Start with the points as individual clusters
 At each step, merge the closest pair of clusters until only one cluster
(or k clusters) left

– Divisive:
 Start with one, all-inclusive cluster
 At each step, split a cluster until each cluster contains a point (or
there are k clusters)

 Traditional hierarchical algorithms use a similarity or

distance matrix
– Merge or split one cluster at a time
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 ‹#›