Experiment No: 04

Aim:
To implement Support Vector Machine (SVM) for classification.

Theory:
Support vector machines (SVMs) are a powerful set of supervised learning methods
used for various tasks such as classification, regression, and outlier detection. The
primary objective of the support vector machine algorithm is to identify a hyperplane in
an N-dimensional space (where N is the number of features) that distinctly classifies
data points into different categories.

Hyperplanes are essentially decision boundaries that separate data into different
classes. The placement of the hyperplane depends on the features of the input data. For
example, if there are two input features, the hyperplane is a line; with three input features,
it becomes a two-dimensional plane. In higher-dimensional space, the hyperplane
generalizes accordingly.

The classification accuracy of SVMs largely depends on finding the optimal hyperplane
that maximizes the margin between different classes. The support vectors are the data
points that lie closest to the hyperplane. These vectors influence the position and
orientation of the hyperplane and help build the SVM model.

SVMs are known for their versatility because they can work with both linear and non-
linear data. In cases where the data cannot be separated by a straight line or plane,
kernel functions are used. A kernel function transforms the data into a higher-
dimensional space where a linear hyperplane can be created.

There are various kernels that can be used with SVMs, including linear, polynomial, radial
basis function (RBF), and sigmoid. Each kernel has its own use case, with the linear
kernel being efficient for linearly separable data and the RBF kernel for non-linear data.

Mathematical Foundations of SVM:

Where:
w is the weight vector,
b is the bias term,
y_i is the class label,
x_i is the feature vector for the i-th instance.

Advantages of SVM:
1. Effective in high-dimensional spaces.
2. Works well with a clear margin of separation.
3. Efficient for both linear and non-linear data through the use of kernel functions.
4. Robust against overfitting, especially in high-dimensional spaces.

Limitations of SVM:
1. Not suitable for large datasets as the training time can be significantly high.
2. Less effective when the data is overlapping or noisy.
3. Choice of kernel functions can significantly affect performance.

Code Implementation:
The following Python code implements SVM using the scikit-learn library to classify the
famous Iris dataset.

```python
from sklearn.utils import shuffle
from sklearn.model_selection import train_test_split
import numpy as np
from sklearn.metrics import accuracy_score
from sklearn.svm import SVC
# Load and preprocess the data
df = df.drop(['SepalWidthCm', 'PetalWidthCm'], axis=1)
X = df[['SepalLengthCm', 'PetalLengthCm']].values
Y = df['Species'].apply(lambda x: -1 if x == 'Iris-setosa' else 1).values
# Shuffle and split the data
X, Y = shuffle(X, Y)
x_train, x_test, y_train, y_test = train_test_split(X, Y, train_size=0.9)
# Train the model using SVM
clf = SVC(kernel='linear')
clf.fit(x_train, y_train)
# Predict and calculate accuracy
y_pred = clf.predict(x_test)
accuracy = accuracy_score(y_test, y_pred)
print('Accuracy of the SVM model:', accuracy)
```

Conclusion:
In this experiment, we successfully implemented a Support Vector Machine (SVM) to
classify the Iris dataset. The model was able to find the hyperplane that best separates
the classes. The results demonstrate that SVM is a powerful and flexible algorithm for
classification tasks.

The experiment also highlighted the importance of support vectors in determining the
optimal hyperplane. These support vectors directly influence the model's decision
boundary, which makes SVM particularly effective for both linearly separable and non-
linearly separable data.

Additionally, kernel functions provide SVM with the ability to handle complex data
patterns, making it a versatile tool in the field of machine learning. Further exploration
could include experimenting with different kernel functions or applying SVM to more
complex datasets for comparison.