Cluster Analysis: Basic Concepts

and Algorithms

Cluster Analysis: Basic Concepts and Algorithms

https://www.isical.ac.in › ~k.ramachandra
K. Ramachandra Murthy
 Outline
 Introduction
 Types of data in cluster analysis
 Types of clusters
 Clustering Techniques
– Partitional
 K-Means
 Bisecting K-Means
 K-Medoids
 CLARA
 CLARANS
– Hierarchical
 Agglomerative
 Divisive
– Density
 DBSCAN
INTRODUCTION
What is Cluster Analysis?

 Finding groups of objects such that the objects in a group will be similar (or
related) to one another and different from (or unrelated to) the objects in
other groups

Inter-cluster
Intra-cluster distances are
distances are maximized
minimized
Quality: What Is Good Clustering?

 A good clustering method will produce high quality clusters with

– high intra-class similarity

– low inter-class similarity

 The quality of a clustering result depends on both the similarity

measure used by the method and its implementation

 The quality of a clustering method is also measured by its ability to

discover some or all of the hidden patterns

Measure the Quality of Clustering

 Dissimilarity/Similarity metric: Similarity is expressed in terms of a distance

function, typically metric: d(i, j)
 There is a separate “quality” function that measures the “goodness” of a
cluster.
 The definitions of distance functions are usually very different for interval-
scaled, boolean, categorical, ordinal ratio, and vector variables.
 Weights should be associated with different variables based on applications
and data semantics.
 It is hard to define “similar enough” or “good enough”

– the answer is typically highly subjective.

Notion of a Cluster can be Ambiguous

How many clusters? Six Clusters

Two Clusters Four Clusters

TYPES OF DATA IN CLUSTER ANALYSIS
 Data Structures
 Data matrix  x11 ... x1f ... x1p 
 
– (two modes)  ... ... ... ... ... 
x ... x if ... x ip 
 i1 
 ... ... ... ... ... 
x ... x nf ... x np 
 n1 

 Dissimilarity matrix  0 
– (one mode)  d(2,1) 0 
 
 d(3,1) d ( 3,2) 0 
 
 : : : 
 d ( n,1) d ( n,2) ... ... 0
Type of data in clustering analysis

 Interval-scaled variables

 Binary variables

 Nominal, ordinal, and ratio variables

 Variables of mixed types

Interval-valued variables

 Standardize data
– Calculate the mean absolute deviation:
s f 1n (| x1 f  m f |  | x2 f  m f | ... | xnf  m f |)
1
where m f  n (x1 f  x2 f  ...  xnf )
.

– Calculate the standardized measurement (z-score)

xif  m f
zif  s
f
 Using mean absolute deviation is more robust than using standard
deviation
Similarity and Dissimilarity Between
Objects

 Distances are normally used to measure the similarity or dissimilarity

between two data objects
 Some popular ones include: Minkowski distance
d (i, j) q (| x  x |q  | x  x |q ... | x  x |q )
i1 j1 i2 j 2 ip jp
where i = (xi1, xi2, …, xip) and j = (xj1, xj2, …, xjp) are two

p-dimensional data objects, and q is a positive integer

 If q = 1, d is Manhattan distance
d (i, j) | x  x |  | x  x | ... | x  x |
i1 j1 i2 j 2 ip jp
Similarity and Dissimilarity Between
Objects

 If q = 2, d is Euclidean distance
d (i, j)  (| x  x |2  | x  x |2 ... | x  x |2 )
i1 j1 i2 j2 ip jp
– Properties
 d(i,j)  0
 d(i,i) = 0
 d(i,j) = d(j,i)
 d(i,j)  d(i,k) + d(k,j)

 Also, one can use weighted distance, parametric Pearson product

moment correlation, or other dissimilarity measures
Binary Variables
Object j
1 0 sum
 A contingency table for 1 a b a b
Object i
0 c d c d
binary data
sum a  c b  d p
 Distance measure for symmetric binary
d (i, j)  b c
variables: a b c  d
 Distance measure for asymmetric
d (i, j)  b c
binary variables: a b  c
 Jaccard coefficient (similarity measure
simJaccard (i, j)  a
for asymmetric binary variables):
a b  c
Dissimilarity between Binary Variables

 Example
Name Gender Fever Cough Test-1 Test-2 Test-3 Test-4
Jack M Y N P N N N
Mary F Y N P N P N
Jim M Y P N N N N

– gender is a symmetric attribute

– the remaining attributes are asymmetric binary
– let the values Y and P be set to 1, and the value N be set to 0
0 1
d ( jack , mary )  0.33
2  0 1
11
d ( jack , jim )  0.67
111
1 2
d ( jim , mary )  0.75
11 2
Nominal Variables

 A generalization of the binary variable in that it can take more than 2

states, e.g., red, yellow, blue, green
 Method 1: Simple matching

– m: # of matches, p: total # of variables

d (i, j)  p p m

 Method 2: use a large number of binary variables

– creating a new binary variable for each of the M nominal states

Ordinal Variables

 An ordinal variable can be discrete or continuous

 Order is important, e.g., rank
 Can be treated like interval-scaled

– replace xif by their rank rif {1,...,M f }

– map the range of each variable onto [0, 1] by replacing i-th object in
the f-th variable by

rif  1
zif 
Mf 1

– compute the dissimilarity using methods for interval-scaled variables

Ratio-Scaled Variables

 Ratio-scaled variable: a positive measurement on a nonlinear scale,

approximately at exponential scale, such as AeBt or Ae-Bt
 Methods:

– treat them like interval-scaled variables—not a good choice!

(why?—the scale can be distorted)

– apply logarithmic transformation

yif = log(xif)

– treat them as continuous ordinal data treat their rank as interval-

scaled
Variables of Mixed Types
 A database may contain all the six types of variables

– symmetric binary, asymmetric binary, nominal, ordinal, interval and ratio

 One may use a weighted formula to combine their effects

 pf 1 ij( f ) d ij( f )

d (i, j ) 
– f is binary or nominal:
 pf 1 ij( f )
dij(f) = 0 if xif = xjf , or dij(f) = 1 otherwise

– f is interval-based: use the normalized distance

– f is ordinal or ratio-scaled
 compute ranks rif and
zif 
r if  1
 and treat zif as interval-scaled M f  1
Vector Objects

TYPES OF CLUSTERS
Types of Clusters

 Well-separated clusters

 Center-based clusters

 Contiguous clusters

 Density-based clusters

 Property or Conceptual

 Described by an Objective Function

Types of Clusters: Well-Separated

 Well-Separated Clusters:

– A cluster is a set of points such that any point in a cluster is

closer (or more similar) to every other point in the cluster than
to any point not in the cluster.

3 well-separated clusters
Types of Clusters: Center-Based

 Center-based
– A cluster is a set of objects such that an object in a cluster is closer
(more similar) to the “center” of a cluster, than to the center of any
other cluster

– The center of a cluster is often a centroid, the average of all the points
in the cluster, or a medoid, the most “representative” point of a cluster

4 center-based clusters
Types of Clusters: Contiguity-Based

 Contiguous Cluster (Nearest neighbor or Transitive)

– A cluster is a set of points such that a point in a cluster is

closer (or more similar) to one or more other points in the
cluster than to any point not in the cluster.

8 contiguous clusters
Types of Clusters: Density-Based

 Density-based

– A cluster is a dense region of points, which is separated by

low-density regions, from other regions of high density.

– Used when the clusters are irregular or intertwined, and when

noise and outliers are present.

6 density-based clusters
Types of Clusters: Conceptual Clusters

 Shared Property or Conceptual Clusters

– Finds clusters that share some common property or represent
a particular concept.

2 Overlapping Circles
Types of Clusters: Objective Function

 Clusters Defined by an Objective Function

– Finds clusters that minimize or maximize an objective function.

– Enumerate all possible ways of dividing the points into clusters

and evaluate the `goodness' of each potential set of clusters by
using the given objective function. (NP Hard)

– Can have global or local objectives.

 Hierarchical clustering algorithms typically have local objectives
 Partitional algorithms typically have global objectives
 Types of Clusters: Objective Function …

 Map the clustering problem to a different domain and solve a related problem in that domain
– Proximity matrix defines a weighted graph, where the nodes are the points being clustered, and the
weighted edges represent the proximities between points

– Clustering is equivalent to breaking the graph into connected components, one for each cluster

– Want to minimize the edge weight between clusters and maximize the edge weight within clusters
CLUSTERING TECHNIQUES
Types of Clustering

 A clustering is a process to find a set of clusters

 Important distinction between hierarchical and partitional sets of
clusters
 Partitional Clustering
– A division data objects into non-overlapping subsets (clusters) such that each
data object is in exactly one subset

 Hierarchical clustering
– A set of nested clusters organized as a hierarchical tree

 Density based clustering

– Discover clusters of arbitrary shape.

– Clusters dense regions of objects separated by regions of low density

Partitional Clustering

Original Points A Partitional Clustering

Hierarchical Clustering

p1
p3 p4
p2
p1 p2 p3 p4
Traditional Hierarchical Clustering Traditional Dendrogram

p1
p3 p4
p2
p1 p2 p3 p4
Non-traditional Hierarchical Clustering Non-traditional Dendrogram
Density based Clustering

Original Points Cluster

s
Clustering Algorithms

 Partitional Clustering

 Hierarchical clustering

 Density-based clustering
Partitional Clustering
 Partitional Clustering

 Given
– A data set of n objects

– K the number of clusters to form

 Organize the objects into k partitions (k<=n) where each partition represents
a cluster
 The clusters are formed to optimize an objective partitioning criterion
– Objects within a cluster are similar

– Objects of different clusters are dissimilar

K-means Clustering

 The basic algorithm is very simple

 Number of clusters, K, must be specified
 Each cluster is associated with a centroid (mean or center
point)
 Each point is assigned to the cluster with the closest
centroid
K-means Clustering – Details

 Initial centroids are often chosen randomly.

– Clusters produced vary from one run to another.

 The centroid is (typically) the mean of the points in the cluster.

 ‘Closeness’ is measured by Euclidean distance, cosine similarity,
correlation, etc.
 K-means will converge for common similarity measures mentioned
above.
 Most of the convergence happens in the first few iterations.
– Often the stopping condition is changed to ‘Until relatively few points change clusters’
or some measure of clustering doesn’t change.

 Complexity is O( n * K * I * d )
– n = number of points, K = number of clusters,
I = number of iterations, d = number of attributes
 Evaluating K-means Clusters

 Most common measure is Sum of Squared Error (SSE)

– For each point, the error is the distance to the nearest cluster

– To get SSE, we square these errors and sum them.

K
SSE   dist 2 ( mi , x )
i 1 xCi

– x is a data point in cluster Ci and mi is the representative point for cluster Ci

 can show that mi corresponds to the center (mean) of the cluster

– Given two clusters, we can choose the one with the smallest error

 One easy way to reduce SSE is to increase K, i.e. the number of clusters
– A good clustering with smaller K can have a lower SSE than a poor clustering
with higher K
Two different K-means Clusterings
3

2.5

2
Original Points
1.5

y
1

0.5

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2

x

3 3

2.5 2.5

2 2

1.5 1.5
y

y
1 1

0.5 0.5

0 0

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2

x x

Optimal Clustering Sub-optimal Clustering

Importance of Choosing Initial Centroids (Case
i)

Iteration 6
1
2
3
4
5
3

2.5

1.5

y 1

0.5

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2

x
 Importance of Choosing Initial Centroids (Case
i)

Iteration 1 Iteration 2 Iteration 3

3 3 3

2.5 2.5 2.5

2 2 2

1.5 1.5 1.5

y

y
1 1 1

0.5 0.5 0.5

0 0 0

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2
x x x

Iteration 4 Iteration 5 Iteration 6

3 3 3

2.5 2.5 2.5

2 2 2

1.5 1.5 1.5

y

y
1 1 1

0.5 0.5 0.5

0 0 0

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2
x x x
Importance of Choosing Initial Centroids (Case
ii)

Iteration 5
1
2
3
4
3

2.5

1.5

y
1

0.5

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2

x
 Importance of Choosing Initial Centroids (Case
ii)

Iteration 1 Iteration 2
3 3

2.5 2.5

2 2

1.5 1.5

y
1 1

0.5 0.5

0 0

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2

x x

Iteration 3 Iteration 4 Iteration 5

3 3 3

2.5 2.5 2.5

2 2 2

1.5 1.5 1.5

y

y
1 1 1

0.5 0.5 0.5

0 0 0

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2
x x x
Problems with Selecting Initial Points

 If there are K ‘real’ clusters then the chance of selecting one centroid
from each cluster is small.
– Chance is relatively small when K is large

– If clusters are the same size, n, then

– For example, if K = 10, then probability = 10!/10 10 = 0.00036

– Sometimes the initial centroids will readjust themselves in ‘right’ way, and
sometimes they don’t

– Consider an example of five pairs of clusters

 Initial centers from different clusters may produce good clusters

10 Clusters Example (Good Clusters)
Iteration 4
1
2
3
8

y 2

-2

-4

-6

0 5 10 15 20
x
Starting with two initial centroids in one cluster of each pair of clusters
10 Clusters Example (Good Clusters)
Iteration 1 Iteration 2
8 8

6 6

4 4

2 2

y
0 0

-2 -2

-4 -4

-6 -6

0 5 10 15 20 0 5 10 15 20
x x
Iteration 3 Iteration 4
8 8

6 6

4 4

2 2
y

y
0 0

-2 -2

-4 -4

-6 -6

0 5 10 15 20 0 5 10 15 20
x x

Starting with two initial centroids in one cluster of each pair of clusters
10 Clusters Example (Bad Clusters)
Iteration 4
1
2
3
8

y 0

-2

-4

-6

0 5 10 15 20
x
Starting with some pairs of clusters having three initial centroids, while other have only one.
10 Clusters Example (Bad Clusters)
Iteration 1 Iteration 2
8 8

6 6

4 4

2 2

y
0 0

-2 -2

-4 -4

-6 -6

0 5 10 15 20 0 5 10 15 20
Iteration
x 3 Iteration
x 4
8 8

6 6

4 4

2 2
y

y
0 0

-2 -2

-4 -4

-6 -6

0 5 10 15 20 0 5 10 15 20
x x

Starting with some pairs of clusters having three initial centroids, while other have only one.
Solutions to Initial Centroids Problem

 Multiple runs
– Helps, but probability is not on your side

 Sample and use hierarchical clustering to determine initial centroids

 Select more than k initial centroids and then select among these initial centroids
– Select most widely separated

 Post-processing
 Bisecting K-means
– Not as susceptible to initialization issues
Pre-processing and Post-processing

 Pre-processing
– Normalize the data

– Eliminate outliers

 Post-processing
– Eliminate small clusters that may represent outliers

– Split ‘loose’ clusters, i.e., clusters with relatively high SSE

– Merge clusters that are ‘close’ and that have relatively low SSE

– Can use these steps during the clustering process

 ISODATA
Bisecting K-means

 Bisecting K-means algorithm

– Variant of K-means that can produce a partitional or a
hierarchical clustering
Bisecting K-means Example
Limitations of K-means

 K-means has problems when clusters are of differing

– Sizes

– Densities

– Non-globular shapes

 K-means has problems when the data contains outliers.

Limitations of K-means: Differing Sizes

Original Points K-means (3 Clusters)

Limitations of K-means: Differing
Density

Original Points K-means (3 Clusters)

Limitations of K-means: Non-globular
Shapes

Original Points K-means (2 Clusters)

Overcoming K-means Limitations

Original Points K-means Clusters

 One solution is to use many clusters.

– Find parts of clusters, but need to put together.
Overcoming K-means Limitations

Original Points K-means Clusters

 One solution is to use many clusters.

– Find parts of clusters, but need to put together.
Overcoming K-means Limitations

Original Points K-means Clusters

 One solution is to use many clusters.

– Find parts of clusters, but need to put together.
Limitations of K-means: Outlier Problem

 The k-means algorithm is sensitive to outliers !

– Since an object with an extremely large value may substantially distort the

distribution of the data.

 Solution: Instead of taking the mean value of the object in a cluster as a reference

point, medoids can be used, which is the most centrally located object in a cluster.

10 10
9 9
8 8
7 7
6 6
5 5
4 4
3 3
2 2
1 1
0 0
0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10
The K-Medoids Clustering Method

 Find representative objects, called medoids, in clusters

 PAM (Partitioning Around Medoids, 1987)

– starts from an initial set of medoids and iteratively replaces one of the medoids

by one of the non-medoids if it improves the total distance of the resulting

clustering.

– PAM works effectively for small data sets, but does not scale well for large data

sets.

 CLARA (Kaufmann & Rousseeuw, 1990)

 CLARANS (Ng & Han, 1994): Randomized sampling

PAM (Partitioning Around Medoids)

 Use real objects to represent the clusters (called medoids)

1. Select k representative objects arbitrarily

2. For each pair of selected object (i) and non-selected object (h), calculate

the Total swapping Cost (TCih)

3. For each pair of i and h,

1. If TCih < 0, i is replaced by h

2. Then assign each non-selected object to the most similar representative

object

4. repeat steps 2-3 until there is no change in the medoids or in TCih.

 Total swapping Cost (TCih)

Total swapping cost TCih=jCjih

– Where Cjih is the cost of swapping i with h for all non medoid objects j

– Cjih will vary depending upon different cases

Sum of Squared Error (SSE)
𝑘
SSE ( 𝐸 ¿= ∑ ∑ 𝑑 2 ( 𝑗 , 𝑖)
𝑖 =1 𝑗 ∈ 𝐶𝑖

𝑦3
𝑥2 𝑦2
𝑥3
𝑥1 C2

C1
𝑦1
Case (i) : Computation of Cjih

• j currently belongs to the cluster represent by the medoid i

• j is less similar to the medoid t compare to h

10

8
t
7
Therefore, in future, j j
6
belongs to the cluster h
represented by h 5

4
Cjih = d(j, h) - d(j, i)
3
i
2

0
0 1 2 3 4 5 6 7 8 9 10

𝑘
Cjih Can be positive or negative, depending on
𝐸=∑ ∑ 𝑑( 𝑗 ,𝑖) whether j is more similar to h or to i
𝑖=1 𝑗∈𝐶 𝑖
Case (ii) : Computation of Cjih

• j currently belongs to the cluster represent by the medoid i

• j is more similar to the medoid t compare to h

10

8 h
Therefore, in future, j 7 j
belongs to the cluster 6

represented by t 5

4
i t
3

0
0 1 2 3 4 5 6 7 8 9 10

Cjih = d(j, t) - d(j, i)

Nonnegative cost incurred in replacing i with h
Case (iii) : Computation of Cjih

• j currently belongs to the cluster represent by the medoid t

• j is more similar to the medoid t compare to h

10

8
t j
7
Therefore, in future, j
6
belongs to the cluster h
5
represented by t itself
4

3
i
2

0
0 1 2 3 4 5 6 7 8 9 10

Cjih = 0
Case (iv) : Computation of Cjih

• j currently belongs to the cluster represent by the medoid t

• j is less similar to the medoid t compare to h

10

8
t
7
Therefore, in future, j j
6
belongs to the cluster h
5
represented by h
4

3
i
2

0
0 1 2 3 4 5 6 7 8 9 10

Cjih = d(j, h) - d(j, t) Always -ve

PAM or K-Medoids: Example

Data Objects 9
8
3

A1 A2 7
4
6 10
01 2 6 1
5
02 3 4
9

03 4
3 8 2 6 8

04 4 7 3 7
5
05 6 2 2
3 4 5 6 7 8 9
06 6 4
07 7 3
08 Goal: create two
7 4 clusters
09 8 5 Choose randmly two
010 7 6 medoids
08 = (7,4) and 02 = (3,4)
PAM or K-Medoids: Example

Data Objects 9
cluster1
8
3 cluster
A1 A2 7 2
4
6 10
01 2 6 1
5
02 3 4
9

03 4
3 8 2 6 8

04 4 7 3 7
5
05 6 2 2
3 4 5 6 7 8 9
06 6 4
07 7 3 Assign each object to theclosest representative object
08 7 4
09 Using L1 Metric (Manhattan), we form the following clusters
8 5
010 7 6 Cluster1 = {01, 02, 03, 04}

Cluster2 = {05, 06, 07, 08, 09, 010}

PAM or K-Medoids: Example

Data Objects 9
cluster1
8
3 cluster
A1 A2 7 2
4
6 10
01 2 6 1
5
02 3 4
9

03 4
3 8 2 6 8

04 4 7 3 7
5
05 6 2 2
3 4 5 6 7 8 9
06 6 4
07 7 3 Compute the absolute error criterion [for the set
08 7 4 of Medoids (O2,O8)]
09 8 5
𝒌
010 7 6
𝑬=∑ ∑ |𝒑− 𝑶 𝒊|=(|𝑶 𝟏 − 𝑶 𝟐|+|𝑶 𝟑 − 𝑶 𝟐|+|𝑶 𝟒 − 𝑶 𝟐|)+¿ ¿
𝒊=𝟏 𝒑 ∈𝑪 𝒊

(|𝑶𝟓 − 𝑶 𝟖|+|𝑶 𝟔 − 𝑶 𝟖|+|𝑶 𝟕 − 𝑶 𝟖|+|𝑶 𝟗 − 𝑶𝟖|+|𝑶𝟏𝟎 − 𝑶𝟖|)

PAM or K-Medoids: Example

Data Objects 9
cluster1
8
3 cluster
A1 A2 7 2
4
6 10
01 2 6 1
5
02 3 4
9

03 4
3 8 2 6 8

04 4 7 3 7
5
05 6 2 2
3 4 5 6 7 8 9
06 6 4
07 7 3
08 7 4 The absolute error criterion [for the set of Medoids
(O2,O8)]
09 8 5
010 7 6 E = (3+4+4)+(3+1+1+2+2) = 20
PAM or K-Medoids: Example

Data Objects 9
cluster1
8
3 cluster
A1 A2 7 2
4
6 10
01 2 6 1
5
02 3 4
9

03 4
3 8 2 6 8

04 4 7 3 7
5
05 6 2 2
3 4 5 6 7 8 9
06 6 4
07 7 3 • Choose a random object 07
08 7 4
• Swap 08 and 07
09 8 5
010 • Compute the absolute error criterion [for the set of
7 6
Medoids (02,07)

E = (3+4+4)+(2+2+1+3+3) = 22
PAM or K-Medoids: Example

Data Objects 9
cluster1
8
3 cluster
A1 A2 7 2
4
6 10
01 2 6 1
5
02 3 4
9

03 4
3 8 2 6 8

04 4 7 3 7
5
05 6 2 2
3 4 5 6 7 8 9
06 6 4
07 7 3 → Compute the cost function
08 7 4
Absolute error [02 ,07 ] - Absolute error [for 02 ,08 ]
09 8 5
010 7 6
S=22-20
S> 0 => It is a bad idea to replace 08 by 07
PAM or K-Medoids: Example

Data Objects 9
cluster1
8
3 cluster
A1 A2 7 2
4
6 10
01 2 6 1
5
02 3 4
9

03 4
3 8 2 6 8

04 4 7 3 7
5
05 6 2 2
3 4 5 6 7 8 9
06 6 4
07 7 3 C687 = d(07,06) - d(08,06)=2-1=1
08 7 4
C587 = d(07,05) - d(08,05)=2-3=-1 C187=0, C387=0, C487=0
09 8 5
010 C987 = d(07,09) - d(08,09)=3-2=1
7 6
C1087 = d(07,010) - d(08,010)=3-2=1

TCih=jCjih=1-1+1+1=2=S
PAM or K-Medoids : Different View
{03,02}
Initial Medoids
E=20 Dataset=

No. of patterns (n) = 5

No. of clusters (k) =2
PAM or K-Medoids : Different View
{03,02}
Initial Medoids
E=20 Dataset=

No. of patterns (n) = 5

No. of clusters (k) =2

{03,01} {03,04} {03,05} {01,02} {04,02} {05,02}

E=30 E=10 E=12 E=19 E=33 E=14
PAM or K-Medoids : Different View
{03,02}
Initial Medoids
E=20 Dataset=

No. of patterns (n) = 5

No. of clusters (k) =2

{03,01} {03,04} {03,05} {01,02} {04,02} {05,02}

E=30 E=10 E=12 E=19 E=33 E=14
PAM or K-Medoids : Different View
{03,02}
Initial Medoids
E=20 Dataset=

No. of patterns (n) = 5

No. of clusters (k) =2

{03,01} {03,04} {03,05} {01,02} {04,02} {05,02}

E=30 E=10 E=12 E=19 E=33 E=14

{01,04}
E=26 {01,05} {04,05}
E=17 E=9
PAM or K-Medoids : Different View
{03,02}
Initial Medoids
E=20 Dataset=

No. of patterns (n) = 5

No. of clusters (k) =2

{03,01} {03,04} {03,05} {01,02} {04,02} {05,02}

E=30 E=10 E=12 E=19 E=33 E=14

{01,04}
E=26 {01,05} {04,05}
E=17 E=9
PAM or K-Medoids : Different View
{03,02}
Initial Medoids
E=20 Dataset=

No. of patterns (n) = 5

No. of clusters (k) =2

{03,01} {03,04} {03,05} {01,02} {04,02} {05,02}

E=30 E=10 E=12 E=19 E=33 E=14

{01,04}
E=26 {01,05} {04,05}
E=17 E=9
PAM or K-Medoids : Different View
{03,02}
Initial Medoids
E=20 Dataset=

No. of patterns (n) = 5

No. of clusters (k) =2

{03,01} {03,04} {03,05} {01,02} {04,02} {05,02}

E=30 E=10 E=12 E=19 E=33 E=14

{01,04}
E=26 {01,05} {04,05}
E=17 E=9
PAM or K-Medoids : Different View
{03,02}
Initial Medoids
E=20 Dataset=

No. of patterns (n) = 5

No. of clusters (k) =2

{03,01} {03,04} {03,05} {01,02} {04,02} {05,02}

E=30 E=10 E=12 E=19 E=33 E=14

{01,04}
E=26 {01,05} {04,05}
E=17 E=9
PAM or K-Medoids : Different View

• Total possibilities for a node=2*3=k*(n-k)

• Cost to compute E= (n-k)
• Total cost for each node is = k*(n-k)2

Goal is to search in the graph for a node with

minimum cost
What Is the Problem with PAM?

 PAM is more robust than k-means in the presence of noise and outliers

because a medoid is less influenced by outliers or other extreme values than

a mean.

 PAM works efficiently for small data sets but does not scale well for large

data sets.

– O(k(n-k)2 ) for each iteration; where n is # of data, k is # of clusters.

 Sampling based method, CLARA (Clustering LARge Applications)

CLARA (Clustering Large Applications)

 CLARA (Kaufmann and Rousseeuw in 1990)

 It draws multiple samples of the data set, applies PAM on each sample, and

gives the best clustering as the output.

 Strength: deals with larger data sets than PAM.

 Weakness:

– Efficiency depends on the sample size.

– A good clustering based on samples will not necessarily represent a

good clustering of the whole data set if the sample is biased.

CLARA (Clustering Large Applications)

 CLARA draws a sample of the dataset and applies PAM on the sample in

order to find the medoids.

 If the sample is best representative of the entire dataset then the medoids of

the sample should approximate the medoids of the entire dataset.

 Medoids are chosen from the sample.

– Note that the algorithm cannot find the best solution if one of the best k-

medoids is not among the selected sample.

CLARA (Clustering Large Applications)

PAM

sample
CLARA (Clustering Large Applications)
Choose the best
clustering
Cluste Cluste Cluste
 To improve the approximation, multiple rs rs rs
PAM PAM PAM
samples are drawn and the best

…
clustering is returned as the output.
sample1 sample2 samplem
 The clustering accuracy is measured

by the average dissimilarity of all

objects in the entire dataset.

 Experiments show that 5 samples of size

40+2k give satisfactory results

 CLARA (Clustering Large Applications)

 For i= 1 to 5, repeat the following steps

– Draw a sample of 40 + 2k objects randomly from the entire data set, and call Al-

gorithm PAM to find k medoids of the sample.

– For each object in the entire data set, determine which of the k medoids is the most similar to it.

– Calculate the average dissimilarity ON THE ENTIRE DATASET of the clustering obtained in the previous step. If

this value is less than the current minimum, use

this value as the current minimum, and retain the k medoids found in Step (1) as

the best set of medoids obtained so far.

CLARA (Clustering Large Applications)

 Complexity of each Iteration is: 0(k(40+k)2 + k(n-k))

 s: the size of the sample
 k: number of clusters
 n: number of objects

 PAM finds the best k medoids among a given data, and CLARA finds the best k

medoids among the selected samples.

 Problems
 The best k medoids may not be selected during the sampling process, in
this case, CLARA will never find the best clustering.
 If the sampling is biased we cannot have a good clustering.
PAM or K-Medoids : Different View

{02,03} Dataset=
Initial Medoids
E=20
No. of patterns (n) = 5
No. of clusters (k) =2

{03,01} {03,04} {03,05} {01,02} {04,02} {05,02}

E=30 E=10 E=12 E=19 E=33 E=14

{01,04}
E=26 {01,05} {04,05}
E=17 E=21
CLARA: Different View

This graph is a sub-graph of the PAM graph.

Initial Medoids

Dataset= {05,02}
{01,02} {04,02}
Sample= E=19 E=33 E=14

No. of patterns (n) = 5

No. of clusters (k) =2

{01,04}
E=26 {01,05} {04,05}
E=17 E=21
CLARA: Different View

{02,03} Dataset=
Initial Medoids
E=20
Sample=
No. of patterns (n) = 5
No. of clusters (k) =2
{03,01} {03,04} {03,05} {01,02} {04,02} {05,02}
E=30 E=10 E=12 E=19 E=33 E=14

{01,04}
E=26 {01,05} {04,05}
E=17 E=21
CLARANS (“Randomized” CLARA)

 CLARANS (A Clustering Algorithm based on Randomized Search) (Ng and

Han’94).
 CLARANS draws sample of neighbors dynamically.
 The clustering process can be presented as searching a graph where every
node is a potential solution, that is, a set of k medoids.
 If the local optimum is found, CLARANS starts with new randomly selected
node in search for a new local optimum.
 It is more efficient and scalable than both PAM and CLARA.
 Focusing techniques and spatial access structures may further improve its
performance (Ester et al.’95).
CLARA: Different View

This graph is a sub-graph of the PAM graph.

Initial Medoids

Dataset= {05,02}
{01,02} {04,02}
Sample= E=19 E=33 E=14

No. of patterns (n) = 5

No. of clusters (k) =2

{01,04}
E=26 {01,05} {04,05}
E=17 E=9
CLARA: Different View

{02,03} Dataset=
Initial Medoids
E=20
Sample=
No. of patterns (n) = 5
No. of clusters (k) =2
{03,01} {03,04} {03,05} {01,02} {04,02} {05,02}
E=30 E=10 E=12 E=19 E=33 E=14

{01,04}
E=26 {01,05} {04,05}
E=17 E=9
CLARANS

• Does not confine the search to a localized area

• Stops the search when a local minimum is found

• Finds several local optimums and output the clustering with the best local optimum

Goal is to search in the graph for a node

with minimum cost
 CLARANS Properties

 Advantages
– Experiments show that CLARANS is more effective than both PAM and CLARA
– Handles outliers

 Disadvantages
– The computational complexity of CLARANS is O(n2), where n is the number of objects

– The clustering quality depends on the sampling method