AVANTHI INSTITUTE OF ENGINEERING AND TECHNOLOGY

(Approved by A.I.C.T.E., New Delhi & Permanently Affiliated to J.N.T.U-GV, Vizianagaram)

NAAC “B++” Accredited Institute
Cherukupally (Village), Near Tagarapuvalasa Bridge, Vizianagaram (Dist) -531162.
Department of Computer Science and Engineering
Unit IV

Basic Methods in Supervised Learning:

Distance-based methods:
Distance metrics are a key part of several machine learning algorithms. These distance metrics are
used in both supervised and unsupervised learning, generally to calculate the similarity between
data points. An effective distance metric improves the performance of our machine learning model,
whether that’s for classification tasks or clustering.
Types of Distance Metrics in Machine Learning
1. Euclidean Distance
2. Manhattan Distance
3. Minkowski Distance
4. Hamming Distance
Euclidean Distance
Euclidean Distance represents the shortest distance between two vectors.It is the square root of the
sum of squares of differences between corresponding elements.
The Euclidean distance metric corresponds to the L2-norm of a difference between vectors and
vector spaces. The cosine similarity is proportional to the dot product of two vectors and inversely
proportional to the product of their magnitudes. Most machine learning algorithms, including K-
Means use this distance metric to measure the similarity between observations. Let’s say we have
two points, as shown below:

Dr GSN Introduction to AI &ML

 AVANTHI INSTITUTE OF ENGINEERING AND TECHNOLOGY
(Approved by A.I.C.T.E., New Delhi & Permanently Affiliated to J.N.T.U-GV, Vizianagaram)
NAAC “B++” Accredited Institute
Cherukupally (Village), Near Tagarapuvalasa Bridge, Vizianagaram (Dist) -531162.
Department of Computer Science and Engineering

So, the Euclidean Distance between these two points, A and B, will be:

Formula for Euclidean Distance

So, the Euclidean Distance between these two points, A and B, will be

We use this formula when we are dealing with 2 dimensions. We can generalize this for an n-

dimensional space as:

Where , N = number of dimensions,

pi, qi = data points
Manhattan Distance
Manhattan Distance is the sum of absolute differences between points across all the dimensions.
We can represent Manhattan Distance as:

Dr GSN Introduction to AI &ML

 AVANTHI INSTITUTE OF ENGINEERING AND TECHNOLOGY
(Approved by A.I.C.T.E., New Delhi & Permanently Affiliated to J.N.T.U-GV, Vizianagaram)
NAAC “B++” Accredited Institute
Cherukupally (Village), Near Tagarapuvalasa Bridge, Vizianagaram (Dist) -531162.
Department of Computer Science and Engineering
Formula for Manhattan Distance
Since the above representation is 2 dimensional, to calculate Manhattan Distance, we will take the
sum of absolute distances in both the x and y directions. So, the Manhattan distance in a 2-
dimensional space is given as:

And the generalized formula for an n-dimensional space is given as:

Where,
 n = number of dimensions
 pi, qi = data points
Minkowski Distance
Minkowski Distance is the generalized form of Euclidean and Manhattan Distance.
Formula for Minkowski Distance

Hamming Distance
Hamming Distance measures the similarity between two strings of the same length. The Hamming
Distance between two strings of the same length is the number of positions at which the
corresponding characters are different.

Let’s understand the concept using an example. Let’s say we have two strings:

“euclidean” and “manhattan”

Since the length of these strings is equal, we can calculate the Hamming Distance. We will go

character by character and match the strings. The first character of both the strings (e and m,

respectively) is different. Similarly, the second character of both the strings (u and a) is different.

and so on.

Dr GSN Introduction to AI &ML

 AVANTHI INSTITUTE OF ENGINEERING AND TECHNOLOGY
(Approved by A.I.C.T.E., New Delhi & Permanently Affiliated to J.N.T.U-GV, Vizianagaram)
NAAC “B++” Accredited Institute
Cherukupally (Village), Near Tagarapuvalasa Bridge, Vizianagaram (Dist) -531162.
Department of Computer Science and Engineering

Look carefully – seven characters are different, whereas two characters (the last two characters) are

similar:euclidean and manhattan distances

Hence, the Hamming Distance here will be 7. Note that the larger the Hamming Distance between

two strings, the more dissimilar those strings will be (and vice versa).
K-Nearest Neighbor (KNN) Algorithm for Machine Learning
o K-Nearest Neighbour is one of the simplest Machine Learning algorithms based on
Supervised Learning technique.
o K-NN algorithm assumes the similarity between the new case/data and available cases and
put the new case into the category that is most similar to the available categories.
o K-NN algorithm stores all the available data and classifies a new data point based on the
similarity. This means when new data appears then it can be easily classified into a well
suite category by using K- NN algorithm.
o K-NN algorithm can be used for Regression as well as for Classification but mostly it is
used for the Classification problems.
o K-NN is a non-parametric algorithm, which means it does not make any assumption on
underlying data.
Why do we need a K-NN Algorithm?
Suppose there are two categories, i.e., Category A and Category B, and we have a new data point
x1, so this data point will lie in which of these categories. To solve this type of problem, we need a
K-NN algorithm. With the help of K-NN, we can easily identify the category or class of a particular
dataset. Consider the below diagram:

How does K-NN work?

The K-NN working can be explained on the basis of the below algorithm:

Dr GSN Introduction to AI &ML

 AVANTHI INSTITUTE OF ENGINEERING AND TECHNOLOGY
(Approved by A.I.C.T.E., New Delhi & Permanently Affiliated to J.N.T.U-GV, Vizianagaram)
NAAC “B++” Accredited Institute
Cherukupally (Village), Near Tagarapuvalasa Bridge, Vizianagaram (Dist) -531162.
Department of Computer Science and Engineering
o Step-1: Select the number K of the neighbors
o Step-2: Calculate the Euclidean distance of K number of neighbors
o Step-3: Take the K nearest neighbors as per the calculated Euclidean distance.
o Step-4: Among these k neighbors, count the number of the data points in each category.
o Step-5: Assign the new data points to that category for which the number of the neighbor is
maximum.
o Step-6: Our model is ready.

Suppose we have a new data point and we need to put it in the required category. Consider the
below image:

o Firstly, we will choose the number of neighbors, so we will choose the k=5.
o Next, we will calculate the Euclidean distance between the data points. The Euclidean
distance is the distance between two points, which we have already studied in geometry. It
can be calculated as:

o By calculating the Euclidean distance we got the nearest neighbors, as three nearest
neighbors in category A and two nearest neighbors in category B. Consider the below
image:

Dr GSN Introduction to AI &ML

 AVANTHI INSTITUTE OF ENGINEERING AND TECHNOLOGY
(Approved by A.I.C.T.E., New Delhi & Permanently Affiliated to J.N.T.U-GV, Vizianagaram)
NAAC “B++” Accredited Institute
Cherukupally (Village), Near Tagarapuvalasa Bridge, Vizianagaram (Dist) -531162.
Department of Computer Science and Engineering

o As we can see the 3 nearest neighbors are from category A, hence this new data point must
belong to category A.

Decision Tree Classification Algorithm

o Decision Tree is a supervised learning technique that can be used for both classification
and Regression problems, but mostly it is preferred for solving Classification problems. It is
a tree-structured classifier, where internal nodes represent the features of a dataset,
branches represent the decision rules and each leaf node represents the outcome.
o In a Decision tree, there are two nodes, which are the Decision Node and Leaf
Node. Decision nodes are used to make any decision and have multiple branches, whereas
Leaf nodes are the output of those decisions and do not contain any further branches.
o The decisions or the test are performed on the basis of features of the given dataset.
o It is a graphical representation for getting all the possible solutions to a problem/decision
based on given conditions.
o It is called a decision tree because, similar to a tree, it starts with the root node, which
expands on further branches and constructs a tree-like structure.
o In order to build a tree, we use the CART algorithm, which stands for Classification and
Regression Tree algorithm.
o A decision tree simply asks a question, and based on the answer (Yes/No), it further split
the tree into subtrees.

Dr GSN Introduction to AI &ML

 AVANTHI INSTITUTE OF ENGINEERING AND TECHNOLOGY
(Approved by A.I.C.T.E., New Delhi & Permanently Affiliated to J.N.T.U-GV, Vizianagaram)
NAAC “B++” Accredited Institute
Cherukupally (Village), Near Tagarapuvalasa Bridge, Vizianagaram (Dist) -531162.
Department of Computer Science and Engineering

Decision Tree Terminologies

 Root Node: Root node is from where the decision tree starts. It represents the entire dataset,
which further gets divided into two or more homogeneous sets.
 Leaf Node: Leaf nodes are the final output node, and the tree cannot be segregated further
after getting a leaf node.
 Splitting: Splitting is the process of dividing the decision node/root node into sub-nodes
according to the given conditions.
 Branch/Sub Tree: A tree formed by splitting the tree.
 Pruning: Pruning is the process of removing the unwanted branches from the tree.
 Parent/Child node: The root node of the tree is called the parent node, and other nodes are
called the child nodes.

How does the Decision Tree algorithm Work?

In a decision tree, for predicting the class of the given dataset, the algorithm starts from the root
node of the tree. This algorithm compares the values of root attribute with the record (real dataset)
attribute and, based on the comparison, follows the branch and jumps to the next node.

For the next node, the algorithm again compares the attribute value with the other sub-nodes and
move further. It continues the process until it reaches the leaf node of the tree. The complete
process can be better understood using the below algorithm:

o Step-1: Begin the tree with the root node, says S, which contains the complete dataset.
o Step-2: Find the best attribute in the dataset using Attribute Selection Measure (ASM).
o Step-3: Divide the S into subsets that contains possible values for the best attributes.

Dr GSN Introduction to AI &ML

 AVANTHI INSTITUTE OF ENGINEERING AND TECHNOLOGY
(Approved by A.I.C.T.E., New Delhi & Permanently Affiliated to J.N.T.U-GV, Vizianagaram)
NAAC “B++” Accredited Institute
Cherukupally (Village), Near Tagarapuvalasa Bridge, Vizianagaram (Dist) -531162.
Department of Computer Science and Engineering
o Step-4: Generate the decision tree node, which contains the best attribute.
o Step-5: Recursively make new decision trees using the subsets of the dataset created in step
-3. Continue this process until a stage is reached where you cannot further classify the nodes
and called the final node as a leaf node.

Example: Suppose there is a candidate who has a job offer and wants to decide whether he should
accept the offer or Not. So, to solve this problem, the decision tree starts with the root node (Salary
attribute by ASM). The root node splits further into the next decision node (distance from the
office) and one leaf node based on the corresponding labels. The next decision node further gets
split into one decision node (Cab facility) and one leaf node. Finally, the decision node splits into
two leaf nodes (Accepted offers and Declined offer). Consider the below diagram:

Support Vector Machine Algorithm

Support Vector Machine or SVM is one of the most popular Supervised Learning algorithms,
which is used for Classification as well as Regression problems. However, primarily, it is used for
Classification problems in Machine Learning.

The goal of the SVM algorithm is to create the best line or decision boundary that can segregate n-
dimensional space into classes so that we can easily put the new data point in the correct category
in the future. This best decision boundary is called a hyperplane.

Dr GSN Introduction to AI &ML

 AVANTHI INSTITUTE OF ENGINEERING AND TECHNOLOGY
(Approved by A.I.C.T.E., New Delhi & Permanently Affiliated to J.N.T.U-GV, Vizianagaram)
NAAC “B++” Accredited Institute
Cherukupally (Village), Near Tagarapuvalasa Bridge, Vizianagaram (Dist) -531162.
Department of Computer Science and Engineering
SVM chooses the extreme points/vectors that help in creating the hyperplane. These extreme cases
are called as support vectors, and hence algorithm is termed as Support Vector Machine. Consider
the below diagram in which there are two different categories that are classified using a decision
boundary or hyperplane:

Example: SVM can be understood with the example that we have used in the KNN classifier.
Suppose we see a strange cat that also has some features of dogs, so if we want a model that can
accurately identify whether it is a cat or dog, so such a model can be created by using the SVM
algorithm. We will first train our model with lots of images of cats and dogs so that it can learn
about different features of cats and dogs, and then we test it with this strange creature. So as
support vector creates a decision boundary between these two data (cat and dog) and choose
extreme cases (support vectors), it will see the extreme case of cat and dog. On the basis of the
support vectors, it will classify it as a cat. Consider the below diagram:

Dr GSN Introduction to AI &ML

 AVANTHI INSTITUTE OF ENGINEERING AND TECHNOLOGY
(Approved by A.I.C.T.E., New Delhi & Permanently Affiliated to J.N.T.U-GV, Vizianagaram)
NAAC “B++” Accredited Institute
Cherukupally (Village), Near Tagarapuvalasa Bridge, Vizianagaram (Dist) -531162.
Department of Computer Science and Engineering
SVM algorithm can be used for Face detection, image classification, text categorization, etc.

Types of SVM

SVM can be of two types:

o Linear SVM: Linear SVM is used for linearly separable data, which means if a dataset can
be classified into two classes by using a single straight line, then such data is termed as
linearly separable data, and classifier is used called as Linear SVM classifier.
o Non-linear SVM: Non-Linear SVM is used for non-linearly separated data, which means if
a dataset cannot be classified by using a straight line, then such data is termed as non-linear
data and classifier used is called as Non-linear SVM classifier.

Hyperplane and Support Vectors in the SVM algorithm:

Hyperplane: There can be multiple lines/decision boundaries to segregate the classes in n-

dimensional space, but we need to find out the best decision boundary that helps to classify the data
points. This best boundary is known as the hyperplane of SVM.

The dimensions of the hyperplane depend on the features present in the dataset, which means if
there are 2 features (as shown in image), then hyperplane will be a straight line. And if there are 3
features, then hyperplane will be a 2-dimension plane.

We always create a hyperplane that has a maximum margin, which means the maximum distance
between the data points.

Support Vectors:

The data points or vectors that are the closest to the hyperplane and which affect the position of the
hyperplane are termed as Support Vector. Since these vectors support the hyperplane, hence called
a Support vector.

How does SVM works?

Linear SVM:

The working of the SVM algorithm can be understood by using an example. Suppose we have a
dataset that has two tags (green and blue), and the dataset has two features x1 and x2. We want a
classifier that can classify the pair(x1, x2) of coordinates in either green or blue. Consider the
below image:

Dr GSN Introduction to AI &ML

 AVANTHI INSTITUTE OF ENGINEERING AND TECHNOLOGY
(Approved by A.I.C.T.E., New Delhi & Permanently Affiliated to J.N.T.U-GV, Vizianagaram)
NAAC “B++” Accredited Institute
Cherukupally (Village), Near Tagarapuvalasa Bridge, Vizianagaram (Dist) -531162.
Department of Computer Science and Engineering

So as it is 2-d space so by just using a straight line, we can easily separate these two classes. But
there can be multiple lines that can separate these classes. Consider the below image:

Hence, the SVM algorithm helps to find the best line or decision boundary; this best boundary or
region is called as a hyperplane. SVM algorithm finds the closest point of the lines from both the
classes. These points are called support vectors. The distance between the vectors and the
hyperplane is called as margin. And the goal of SVM is to maximize this margin.
The hyperplane with maximum margin is called the optimal hyperplane.

Dr GSN Introduction to AI &ML

 AVANTHI INSTITUTE OF ENGINEERING AND TECHNOLOGY
(Approved by A.I.C.T.E., New Delhi & Permanently Affiliated to J.N.T.U-GV, Vizianagaram)
NAAC “B++” Accredited Institute
Cherukupally (Village), Near Tagarapuvalasa Bridge, Vizianagaram (Dist) -531162.
Department of Computer Science and Engineering
Non-Linear SVM:

If data is linearly arranged, then we can separate it by using a straight line, but for non-linear data,
we cannot draw a single straight line. Consider the below image:

So to separate these data points, we need to add one more dimension. For linear data, we have used
two dimensions x and y, so for non-linear data, we will add a third dimension z. It can be calculated
as:

z=x2 +y2

By adding the third dimension, the sample space will become as below image:

So now, SVM will divide the datasets into classes in the following way. Consider the below image:

Dr GSN Introduction to AI &ML

 AVANTHI INSTITUTE OF ENGINEERING AND TECHNOLOGY
(Approved by A.I.C.T.E., New Delhi & Permanently Affiliated to J.N.T.U-GV, Vizianagaram)
NAAC “B++” Accredited Institute
Cherukupally (Village), Near Tagarapuvalasa Bridge, Vizianagaram (Dist) -531162.
Department of Computer Science and Engineering

Since we are in 3-d Space, hence it is looking like a plane parallel to the x-axis. If we convert it in
2d space with z=1, then it will become as:

Hence we get a circumference of radius 1 in case of non-linear data.

Python Implementation of Support Vector Machine

Now we will implement the SVM algorithm using Python. Here we will use the same
dataset user_data, which we have used in Logistic regression and KNN classification.

Dr GSN Introduction to AI &ML

 AVANTHI INSTITUTE OF ENGINEERING AND TECHNOLOGY
(Approved by A.I.C.T.E., New Delhi & Permanently Affiliated to J.N.T.U-GV, Vizianagaram)
NAAC “B++” Accredited Institute
Cherukupally (Village), Near Tagarapuvalasa Bridge, Vizianagaram (Dist) -531162.
Department of Computer Science and Engineering
Unsupervised learning

Clustering in Machine Learning

Clustering or cluster analysis is a machine learning technique, which groups the unlabelled dataset.
It can be defined as "A way of grouping the data points into different clusters, consisting of similar
data points. The objects with the possible similarities remain in a group that has less or no
similarities with another group."

t does it by finding some similar patterns in the unlabelled dataset such as shape, size, color,
behavior, etc., and divides them as per the presence and absence of those similar patterns.

It is an unsupervised learning method, hence no supervision is provided to the algorithm, and it

deals with the unlabeled dataset.

After applying this clustering technique, each cluster or group is provided with a cluster-ID. ML
system can use this id to simplify the processing of large and complex datasets.

The clustering technique is commonly used for statistical data analysis.

Example: Let's understand the clustering technique with the real-world example of Mall: When we
visit any shopping mall, we can observe that the things with similar usage are grouped together.
Such as the t-shirts are grouped in one section, and trousers are at other sections, similarly, at
vegetable sections, apples, bananas, Mangoes, etc., are grouped in separate sections, so that we can
easily find out the things. The clustering technique also works in the same way. Other examples of
clustering are grouping documents according to the topic.

The clustering technique can be widely used in various tasks. Some most common uses of this
technique are:

o Market Segmentation
o Statistical data analysis
o Social network analysis
o Image segmentation
o Anomaly detection, etc.
Apart from these general usages, it is used by the Amazon in its recommendation system to
provide the recommendations as per the past search of products. Netflix also uses this technique to
recommend the movies and web-series to its users as per the watch history.
The below diagram explains the working of the clustering algorithm. We can see the different fruits
are divided into several groups with similar properties.

Dr GSN Introduction to AI &ML

 AVANTHI INSTITUTE OF ENGINEERING AND TECHNOLOGY
(Approved by A.I.C.T.E., New Delhi & Permanently Affiliated to J.N.T.U-GV, Vizianagaram)
NAAC “B++” Accredited Institute
Cherukupally (Village), Near Tagarapuvalasa Bridge, Vizianagaram (Dist) -531162.
Department of Computer Science and Engineering

Types of Clustering Methods

The clustering methods are broadly divided into Hard clustering (datapoint belongs to only one
group) and Soft Clustering (data points can belong to another group also). But there are also other
various approaches of Clustering exist. Below are the main clustering methods used in Machine
learning:

1. Partitioning Clustering
2. Density-Based Clustering
3. Distribution Model-Based Clustering
4. Hierarchical Clustering
5. Fuzzy Clustering

Partitioning Clustering
It is a type of clustering that divides the data into non-hierarchical groups. It is also known as
the centroid-based method. The most common example of partitioning clustering is the K-Means
Clustering algorithm.
In this type, the dataset is divided into a set of k groups, where K is used to define the number of
pre-defined groups. The cluster center is created in such a way that the distance between the data
points of one cluster is minimum as compared to another cluster centroid.

Dr GSN Introduction to AI &ML

 AVANTHI INSTITUTE OF ENGINEERING AND TECHNOLOGY
(Approved by A.I.C.T.E., New Delhi & Permanently Affiliated to J.N.T.U-GV, Vizianagaram)
NAAC “B++” Accredited Institute
Cherukupally (Village), Near Tagarapuvalasa Bridge, Vizianagaram (Dist) -531162.
Department of Computer Science and Engineering

Density-Based Clustering

The density-based clustering method connects the highly-dense areas into clusters, and the
arbitrarily shaped distributions are formed as long as the dense region can be connected. This
algorithm does it by identifying different clusters in the dataset and connects the areas of high
densities into clusters. The dense areas in data space are divided from each other by sparser areas.

These algorithms can face difficulty in clustering the data points if the dataset has varying densities
and high dimensions.

Distribution Model-Based Clustering

In the distribution model-based clustering method, the data is divided based on the probability of
how a dataset belongs to a particular distribution. The grouping is done by assuming some
distributions commonly Gaussian distribution.

The example of this type is the Expectation-Maximization Clustering algorithm that uses
Gaussian Mixture Models (GMM).

Dr GSN Introduction to AI &ML

 AVANTHI INSTITUTE OF ENGINEERING AND TECHNOLOGY
(Approved by A.I.C.T.E., New Delhi & Permanently Affiliated to J.N.T.U-GV, Vizianagaram)
NAAC “B++” Accredited Institute
Cherukupally (Village), Near Tagarapuvalasa Bridge, Vizianagaram (Dist) -531162.
Department of Computer Science and Engineering

Hierarchical Clustering

Hierarchical clustering can be used as an alternative for the partitioned clustering as there is no
requirement of pre-specifying the number of clusters to be created. In this technique, the dataset is
divided into clusters to create a tree-like structure, which is also called a dendrogram. The
observations or any number of clusters can be selected by cutting the tree at the correct level. The
most common example of this method is the Agglomerative Hierarchical algorithm.

Fuzzy Clustering

Fuzzy clustering is a type of soft method in which a data object may belong to more than one group
or cluster. Each dataset has a set of membership coefficients, which depend on the degree of
membership to be in a cluster. Fuzzy C-means algorithm is the example of this type of clustering;
it is sometimes also known as the Fuzzy k-means algorithm..

K-Means Clustering Algorithm

K-Means Clustering is an unsupervised learning algorithm that is used to solve the clustering
problems in machine learning or data science. In this topic, we will learn what is K-means
clustering algorithm, how the algorithm works, along with the Python implementation of k-means
clustering.

Dr GSN Introduction to AI &ML

 AVANTHI INSTITUTE OF ENGINEERING AND TECHNOLOGY
(Approved by A.I.C.T.E., New Delhi & Permanently Affiliated to J.N.T.U-GV, Vizianagaram)
NAAC “B++” Accredited Institute
Cherukupally (Village), Near Tagarapuvalasa Bridge, Vizianagaram (Dist) -531162.
Department of Computer Science and Engineering
What is K-Means Algorithm?

K-Means Clustering is an Unsupervised Learning algorithm, which groups the unlabeled dataset
into different clusters. Here K defines the number of pre-defined clusters that need to be created in
the process, as if K=2, there will be two clusters, and for K=3, there will be three clusters, and so
on.

It allows us to cluster the data into different groups and a convenient way to discover the categories
of groups in the unlabeled dataset on its own without the need for any training.

It is a centroid-based algorithm, where each cluster is associated with a centroid. The main aim of
this algorithm is to minimize the sum of distances between the data point and their corresponding
clusters.

The algorithm takes the unlabeled dataset as input, divides the dataset into k-number of clusters,
and repeats the process until it does not find the best clusters. The value of k should be
predetermined in this algorithm.

The k-means clustering algorithm mainly performs two tasks:

o Determines the best value for K center points or centroids by an iterative process.
o Assigns each data point to its closest k-center. Those data points which are near to the
particular k-center, create a cluster.

Hence each cluster has datapoints with some commonalities, and it is away from other clusters.

The below diagram explains the working of the K-means Clustering Algorithm:

How does the K-Means Algorithm Work?

The working of the K-Means algorithm is explained in the below steps:

Dr GSN Introduction to AI &ML

 AVANTHI INSTITUTE OF ENGINEERING AND TECHNOLOGY
(Approved by A.I.C.T.E., New Delhi & Permanently Affiliated to J.N.T.U-GV, Vizianagaram)
NAAC “B++” Accredited Institute
Cherukupally (Village), Near Tagarapuvalasa Bridge, Vizianagaram (Dist) -531162.
Department of Computer Science and Engineering
Step-1: Select the number K to decide the number of clusters.

Step-2: Select random K points or centroids. (It can be other from the input dataset).

Step-3: Assign each data point to their closest centroid, which will form the predefined K clusters.

Step-4: Calculate the variance and place a new centroid of each cluster.

Step-5: Repeat the third steps, which means reassign each datapoint to the new closest centroid of
each cluster.

Step-6: If any reassignment occurs, then go to step-4 else go to FINISH.

Step-7: The model is ready.

Let's understand the above steps by considering the visual plots:

Suppose we have two variables M1 and M2. The x-y axis scatter plot of these two variables is
given below:

o Let's take number k of clusters, i.e., K=2, to identify the dataset and to put them into
different clusters. It means here we will try to group these datasets into two different
clusters.
o We need to choose some random k points or centroid to form the cluster. These points can
be either the points from the dataset or any other point. So, here we are selecting the below
two points as k points, which are not the part of our dataset. Consider the below image:

Dr GSN Introduction to AI &ML

 AVANTHI INSTITUTE OF ENGINEERING AND TECHNOLOGY
(Approved by A.I.C.T.E., New Delhi & Permanently Affiliated to J.N.T.U-GV, Vizianagaram)
NAAC “B++” Accredited Institute
Cherukupally (Village), Near Tagarapuvalasa Bridge, Vizianagaram (Dist) -531162.
Department of Computer Science and Engineering
o Now we will assign each data point of the scatter plot to its closest K-point or centroid. We
will compute it by applying some mathematics that we have studied to calculate the distance
between two points. So, we will draw a median between both the centroids.

From the above image, it is clear that points left side of the line is near to the K1 or blue centroid,
and points to the right of the line are close to the yellow centroid. Let's color them as blue and
yellow for clear visualization.

o As we need to find the closest cluster, so we will repeat the process by choosing a new
centroid. To choose the new centroids, we will compute the center of gravity of these

Dr GSN Introduction to AI &ML

 AVANTHI INSTITUTE OF ENGINEERING AND TECHNOLOGY
(Approved by A.I.C.T.E., New Delhi & Permanently Affiliated to J.N.T.U-GV, Vizianagaram)
NAAC “B++” Accredited Institute
Cherukupally (Village), Near Tagarapuvalasa Bridge, Vizianagaram (Dist) -531162.
Department of Computer Science and Engineering
centroids, and will find new centroids as below:

o Next, we will reassign each data point to the new centroid. For this, we will repeat the same
process of finding a median line. The median will be like below image:

o From the above image, we can see, one yellow point is on the left side of the line, and two
blue points are right to the line. So, these three points will be assigned to new centroids.

Dr GSN Introduction to AI &ML

 AVANTHI INSTITUTE OF ENGINEERING AND TECHNOLOGY
(Approved by A.I.C.T.E., New Delhi & Permanently Affiliated to J.N.T.U-GV, Vizianagaram)
NAAC “B++” Accredited Institute
Cherukupally (Village), Near Tagarapuvalasa Bridge, Vizianagaram (Dist) -531162.
Department of Computer Science and Engineering

As reassignment has taken place, so we will again go to the step-4, which is finding new centroids
or K-points.

We will repeat the process by finding the center of gravity of centroids, so the new centroids will
be as shown in the below image:

Dr GSN Introduction to AI &ML

 AVANTHI INSTITUTE OF ENGINEERING AND TECHNOLOGY
(Approved by A.I.C.T.E., New Delhi & Permanently Affiliated to J.N.T.U-GV, Vizianagaram)
NAAC “B++” Accredited Institute
Cherukupally (Village), Near Tagarapuvalasa Bridge, Vizianagaram (Dist) -531162.
Department of Computer Science and Engineering
As we got the new centroids so again will draw the median line and reassign the data points

o We can see in the above image; there are no dissimilar data points on either side of the line,
which means our model is formed. Consider the below image:

As our model is ready, so we can now remove the assumed centroids, and the two final clusters will
be as shown in the below image:

Dr GSN Introduction to AI &ML

 AVANTHI INSTITUTE OF ENGINEERING AND TECHNOLOGY
(Approved by A.I.C.T.E., New Delhi & Permanently Affiliated to J.N.T.U-GV, Vizianagaram)
NAAC “B++” Accredited Institute
Cherukupally (Village), Near Tagarapuvalasa Bridge, Vizianagaram (Dist) -531162.
Department of Computer Science and Engineering

What is Dimensionality Reduction?

The number of input features, variables, or columns present in a given dataset is known as
dimensionality, and the process to reduce these features is called dimensionality reduction.

A dataset contains a huge number of input features in various cases, which makes the predictive
modeling task more complicated. Because it is very difficult to visualize or make predictions for
the training dataset with a high number of features, for such cases, dimensionality reduction
techniques are required to use.

Dimensionality reduction technique can be defined as, "It is a way of converting the higher
dimensions dataset into lesser dimensions dataset ensuring that it provides similar
information." These techniques are widely used in machine learning for obtaining a better fit
predictive model while solving the classification and regression problems.

It is commonly used in the fields that deal with high-dimensional data, such as speech recognition,
signal processing, bioinformatics, etc. It can also be used for data visualization, noise
reduction, cluster analysis, etc.

Dr GSN Introduction to AI &ML

 AVANTHI INSTITUTE OF ENGINEERING AND TECHNOLOGY
(Approved by A.I.C.T.E., New Delhi & Permanently Affiliated to J.N.T.U-GV, Vizianagaram)
NAAC “B++” Accredited Institute
Cherukupally (Village), Near Tagarapuvalasa Bridge, Vizianagaram (Dist) -531162.
Department of Computer Science and Engineering

The Curse of Dimensionality

Handling the high-dimensional data is very difficult in practice, commonly known as the curse of
dimensionality. If the dimensionality of the input dataset increases, any machine learning algorithm
and model becomes more complex. As the number of features increases, the number of samples
also gets increased proportionally, and the chance of overfitting also increases. If the machine
learning model is trained on high-dimensional data, it becomes overfitted and results in poor
performance.

Hence, it is often required to reduce the number of features, which can be done with dimensionality
reduction.

Benefits of applying Dimensionality Reduction

Some benefits of applying dimensionality reduction technique to the given dataset are given below:

o By reducing the dimensions of the features, the space required to store the dataset also gets
reduced.
o Less Computation training time is required for reduced dimensions of features.
o Reduced dimensions of features of the dataset help in visualizing the data quickly.
o It removes the redundant features (if present) by taking care of multicollinearity.

Disadvantages of dimensionality Reduction

Dr GSN Introduction to AI &ML

 AVANTHI INSTITUTE OF ENGINEERING AND TECHNOLOGY
(Approved by A.I.C.T.E., New Delhi & Permanently Affiliated to J.N.T.U-GV, Vizianagaram)
NAAC “B++” Accredited Institute
Cherukupally (Village), Near Tagarapuvalasa Bridge, Vizianagaram (Dist) -531162.
Department of Computer Science and Engineering
There are also some disadvantages of applying the dimensionality reduction, which are given
below:

o Some data may be lost due to dimensionality reduction.

o In the PCA dimensionality reduction technique, sometimes the principal components
required to consider are unknown.

Approaches of Dimension Reduction

There are two ways to apply the dimension reduction technique, which are given below:

Feature Selection

Feature selection is the process of selecting the subset of the relevant features and leaving out the
irrelevant features present in a dataset to build a model of high accuracy. In other words, it is a way
of selecting the optimal features from the input dataset.

Three methods are used for the feature selection:

1. Filters Methods

In this method, the dataset is filtered, and a subset that contains only the relevant features is taken.
Some common techniques of filters method are:

o Correlation
o Chi-Square Test
o ANOVA
o Information Gain, etc.

2. Wrappers Methods

The wrapper method has the same goal as the filter method, but it takes a machine learning model
for its evaluation. In this method, some features are fed to the ML model, and evaluate the
performance. The performance decides whether to add those features or remove to increase the
accuracy of the model. This method is more accurate than the filtering method but complex to
work. Some common techniques of wrapper methods are:

o Forward Selection
o Backward Selection
o Bi-directional Elimination

Dr GSN Introduction to AI &ML

 AVANTHI INSTITUTE OF ENGINEERING AND TECHNOLOGY
(Approved by A.I.C.T.E., New Delhi & Permanently Affiliated to J.N.T.U-GV, Vizianagaram)
NAAC “B++” Accredited Institute
Cherukupally (Village), Near Tagarapuvalasa Bridge, Vizianagaram (Dist) -531162.
Department of Computer Science and Engineering
3. Embedded Methods: Embedded methods check the different training iterations of the machine
learning model and evaluate the importance of each feature. Some common techniques of
Embedded methods are:

o LASSO
o Elastic Net
o Ridge Regression, etc.

Feature Extraction:

Feature extraction is the process of transforming the space containing many dimensions into space
with fewer dimensions. This approach is useful when we want to keep the whole information but
use fewer resources while processing the information.

Some common feature extraction techniques are:

a) Principal Component Analysis

b) Linear Discriminant Analysis
c) Kernel PCA
d) Quadratic Discriminant Analysis

Common techniques of Dimensionality Reduction

a) Principal Component Analysis

b) Backward Elimination
c) Forward Selection
d) Score comparison
e) Missing Value Ratio
f) Low Variance Filter
g) High Correlation Filter
h) Random Forest
i) Factor Analysis
j) Auto-Encoder

Principal Component Analysis (PCA)

Principal Component Analysis is a statistical process that converts the observations of correlated
features into a set of linearly uncorrelated features with the help of orthogonal transformation.

Dr GSN Introduction to AI &ML

 AVANTHI INSTITUTE OF ENGINEERING AND TECHNOLOGY
(Approved by A.I.C.T.E., New Delhi & Permanently Affiliated to J.N.T.U-GV, Vizianagaram)
NAAC “B++” Accredited Institute
Cherukupally (Village), Near Tagarapuvalasa Bridge, Vizianagaram (Dist) -531162.
Department of Computer Science and Engineering
These new transformed features are called the Principal Components. It is one of the popular tools
that is used for exploratory data analysis and predictive modeling.

PCA works by considering the variance of each attribute because the high attribute shows the good
split between the classes, and hence it reduces the dimensionality. Some real-world applications of
PCA are image processing, movie recommendation system, optimizing the power allocation in
various communication channels.

Backward Feature Elimination

The backward feature elimination technique is mainly used while developing Linear Regression or
Logistic Regression model. Below steps are performed in this technique to reduce the
dimensionality or in feature selection:

o In this technique, firstly, all the n variables of the given dataset are taken to train the model.
o The performance of the model is checked.
o Now we will remove one feature each time and train the model on n-1 features for n times,
and will compute the performance of the model.
o We will check the variable that has made the smallest or no change in the performance of
the model, and then we will drop that variable or features; after that, we will be left with n-1
features.
o Repeat the complete process until no feature can be dropped.
In this technique, by selecting the optimum performance of the model and maximum tolerable error
rate, we can define the optimal number of features require for the machine learning algorithms.
Forward Feature Selection
Forward feature selection follows the inverse process of the backward elimination process. It
means, in this technique, we don't eliminate the feature; instead, we will find the best features that
can produce the highest increase in the performance of the model. Below steps are performed in
this technique:
o We start with a single feature only, and progressively we will add each feature at a time.
o Here we will train the model on each feature separately.
o The feature with the best performance is selected.
o The process will be repeated until we get a significant increase in the performance of the
model.
Missing Value Ratio
If a dataset has too many missing values, then we drop those variables as they do not carry much
useful information. To perform this, we can set a threshold level, and if a variable has missing
values more than that threshold, we will drop that variable. The higher the threshold value, the
more efficient the reduction.
Low Variance Filter
As same as missing value ratio technique, data columns with some changes in the data have less
information. Therefore, we need to calculate the variance of each variable, and all data columns

Dr GSN Introduction to AI &ML

 AVANTHI INSTITUTE OF ENGINEERING AND TECHNOLOGY
(Approved by A.I.C.T.E., New Delhi & Permanently Affiliated to J.N.T.U-GV, Vizianagaram)
NAAC “B++” Accredited Institute
Cherukupally (Village), Near Tagarapuvalasa Bridge, Vizianagaram (Dist) -531162.
Department of Computer Science and Engineering
with variance lower than a given threshold are dropped because low variance features will not
affect the target variable.
High Correlation Filter
High Correlation refers to the case when two variables carry approximately similar information.
Due to this factor, the performance of the model can be degraded. This correlation between the
independent numerical variable gives the calculated value of the correlation coefficient. If this
value is higher than the threshold value, we can remove one of the variables from the dataset. We
can consider those variables or features that show a high correlation with the target variable.
Random Forest
Random Forest is a popular and very useful feature selection algorithm in machine learning. This
algorithm contains an in-built feature importance package, so we do not need to program it
separately. In this technique, we need to generate a large set of trees against the target variable, and
with the help of usage statistics of each attribute, we need to find the subset of features.
Random forest algorithm takes only numerical variables, so we need to convert the input data into
numeric data using hot encoding.
Factor Analysis
Factor analysis is a technique in which each variable is kept within a group according to the
correlation with other variables, it means variables within a group can have a high correlation
between themselves, but they have a low correlation with variables of other groups.
We can understand it by an example, such as if we have two variables Income and spend. These
two variables have a high correlation, which means people with high income spend more, and vice
versa. So, such variables are put into a group, and that group is known as the factor. The number of
these factors will be reduced as compared to the original dimension of the dataset.
Auto-encoders
One of the popular methods of dimensionality reduction is auto-encoder, which is a type of ANN
or artificial neural network, and its main aim is to copy the inputs to their outputs. In this, the input
is compressed into latent-space representation, and output is occurred using this representation. It
has mainly two parts:
o Encoder: The function of the encoder is to compress the input to form the latent-space
representation.
o Decoder: The function of the decoder is to recreate the output from the latent-space
representation.

Dr GSN Introduction to AI &ML