Page 1 of 8

UNIT-2: Data Preparation

Working with Real Data:

Working with real data in data preparation for machine learning involves several steps
to ensure the data is properly formatted, cleaned, and preprocessed for use in training a
machine learning model. Here's a general guide to the data preparation process:

1. Data Collection: Obtain the dataset from reliable sources. This could be from
databases, APIs, CSV files, Excel spreadsheets, or any other structured data
format.
2. Exploratory Data Analysis (EDA): Perform EDA to understand the structure,
distribution, and characteristics of the data. This involves:
 Checking for missing values.
 Summarizing statistics (mean, median, min, max, etc.).
 Visualizations (histograms, box plots, scatter plots, etc.) to understand
relationships and distributions.
 Identify outliers and anomalies.
3. Data Cleaning:
 Handle missing values: Impute missing values (using mean, median, mode,
or more sophisticated methods), or remove rows/columns with missing
data depending on the amount of missingness and the nature of the
problem.
 Deal with outliers: Decide whether to remove outliers or transform them to
mitigate their impact on the model.
 Address inconsistencies and errors in data: This might involve correcting
typos, standardizing formats, or resolving inconsistencies in categorical
variables.
4. Feature Engineering:
 Create new features: Combine existing features or derive new ones that
might be more informative for the model.
 Encode categorical variables: Convert categorical variables into numerical
representations using techniques like one-hot encoding, label encoding,
or embeddings.
 Feature scaling: Scale numerical features to a similar range (e.g., using
min-max scaling or standardization) to prevent features with large values
from dominating the model.
5. Data Transformation:
 Normalize the data: Scale the features to have a mean of 0 and a standard
deviation of 1 to improve convergence during training.
 Page 2 of 8

 Dimensionality reduction: If dealing with high-dimensional data, use

techniques like Principal Component Analysis (PCA) or feature selection to
reduce the number of features while preserving most of the variance.
6. Data Splitting:
 Split the data into training, validation, and test sets to assess model
performance and prevent overfitting.
7. Data Preprocessing Pipeline:
 Create a preprocessing pipeline that encapsulates all the data preparation
steps. This ensures consistency and allows easy application to new data.
8. Iterative Process: Data preparation is often an iterative process. You may need
to revisit previous steps based on insights gained during model training and
evaluation.
9. Documentation: Document all the steps taken during data preparation,
including any assumptions made or decisions taken. This documentation is
crucial for reproducibility and collaboration.

Look at the Big Picture:

Looking at the big picture in data preparation for machine learning involves
understanding the overarching goals, challenges, and best practices that guide the
entire process. Here's an overview:

1. Understanding Business Objectives: Data preparation starts with a clear

understanding of the business problem or objectives that the machine learning
model aims to address. This understanding helps in defining the scope of data
collection, the choice of features, and the evaluation metrics for the model.
2. Data Collection: Acquiring relevant and high-quality data is crucial for the
success of any machine learning project. This involves identifying data sources,
collecting data, and ensuring its integrity, accuracy, and completeness. Data may
come from various internal or external sources, such as databases, APIs, sensor
data, or web scraping.
3. Data Cleaning and Preprocessing: Raw data often contains errors, missing
values, outliers, and inconsistencies that need to be addressed before feeding it
into a machine learning model. Data cleaning involves tasks like imputation of
missing values, handling outliers, removing duplicates, and standardizing formats.
Preprocessing includes feature scaling, encoding categorical variables, and
handling skewness in distributions.
4. Feature Engineering: Feature engineering is the process of creating new features
or transforming existing ones to enhance the performance of machine learning
models. This step requires domain knowledge and creativity to extract
 Page 3 of 8

meaningful information from the data. Feature engineering aims to capture

relevant patterns, reduce dimensionality, and improve the model's ability to
generalize.
5. Exploratory Data Analysis (EDA): EDA is an essential step in understanding the
underlying structure and relationships within the data. It involves visualizations,
statistical summaries, and hypothesis testing to gain insights into the data
distribution, correlations between variables, and potential patterns or trends.
6. Data Transformation and Scaling: Data transformation techniques like
normalization or standardization are applied to ensure that all features have a
similar scale. This prevents features with larger magnitudes from dominating the
model and helps in achieving faster convergence during training.
7. Data Splitting: Before training a machine learning model, the dataset is split into
training, validation, and test sets. This ensures that the model is trained on one
set, validated on another set for hyperparameter tuning, and tested on a separate
set to evaluate its generalization performance.
8. Documentation and Reproducibility: Documenting the entire data preparation
process is crucial for reproducibility and transparency. This includes recording
data sources, preprocessing steps, feature engineering techniques, and any
assumptions or decisions made during the process.
9. Iterative Process: Data preparation is often an iterative process that involves
refining data cleaning procedures, experimenting with different feature
engineering techniques, and optimizing preprocessing steps based on model
performance and feedback.
10. Continuous Monitoring and Maintenance: Once a machine learning model is
deployed, it's essential to monitor its performance over time and update the data
preparation pipeline accordingly. This ensures that the model remains effective in
real-world scenarios and adapts to changing data patterns or business
requirements.

Get the Data:

Discover and Visualize the Data to Gain Insights :

Discovering and visualizing the data is a crucial step in data preparation for machine
learning. Here's a guide on how to perform exploratory data analysis (EDA) to gain
insights:

1. Load the Data: Start by loading your dataset into your preferred data analysis
environment such as Python with libraries like Pandas, NumPy, and
Matplotlib/Seaborn for visualization.
2. Basic Data Exploration:
 Page 4 of 8

 Check the first few rows of the dataset using the .head() function to
understand its structure.
 Check the dimensions of the dataset (number of rows and columns) using
the .shape attribute.
 Use the .info() function to get a concise summary of the dataset, including
data types and missing values.
3. Summary Statistics:
 Compute summary statistics such as mean, median, standard deviation,
minimum, and maximum values for numerical features using the .describe ()
function.
 For categorical features, you can use the .value_ counts () function to get the
frequency distribution of unique values.
4. Data Visualization:
 Histograms: Plot histograms to visualize the distribution of numerical
features. This helps in understanding the range and spread of values and
identifying potential outliers.
 Box plots: Use box plots to visualize the distribution of numerical features,
identify outliers, and compare distributions across different categories.
 Scatter plots: Plot scatter plots to visualize the relationship between pairs
of numerical features. This helps in identifying patterns, correlations, and
potential trends in the data.
 Bar plots: Use bar plots to visualize the frequency distribution of
categorical features. This helps in understanding the distribution of
categories and identifying dominant categories.
 Heatmaps: Plot heatmaps to visualize the correlation matrix between
numerical features. This helps in identifying multicollinearity and
understanding the strength and direction of correlations.
5. Feature Relationships:
 Explore relationships between features using scatter plots, pair plots (for
multiple numerical features), and categorical plots (for categorical
features).
 Look for patterns, trends, and correlations between features, which can
provide valuable insights for feature selection and engineering.
6. Missing Values and Outliers:
 Visualize missing values using heatmaps or bar plots to identify patterns of
missingness across features.
 Plot box plots or scatter plots to identify outliers in numerical features.
Decide whether to remove or impute outliers based on domain knowledge
and the impact on the model.
7. Interactive Visualizations:
 Page 5 of 8

 Consider using interactive visualization libraries like Plotly or Bokeh for

more interactive and dynamic exploration of the data.
8. Iterative Exploration:
 Perform iterative data exploration and visualization based on initial
insights and hypotheses generated. This may involve drilling down into
specific subsets of the data or focusing on particular features of interest.

Prepare the Data for machine Learning Algorithms:

Preparing data for machine learning algorithms involves several steps to ensure that the
dataset is formatted correctly, features are appropriately scaled, and the data is ready to
be used for training a model. Here's a comprehensive guide to preparing data for
machine learning:

1. Handling Missing Values:

 Identify and handle missing values in the dataset. Options include:
 Removing rows or columns with missing values if they are
insignificant.
 Imputing missing values using methods like mean, median, mode,
or more sophisticated techniques such as K-nearest neighbors
(KNN) imputation or predictive modeling.
 For categorical variables, consider adding a new category to
represent missing values if they carry meaningful information.
2. Encoding Categorical Variables:
 Convert categorical variables into a numerical format suitable for machine
learning algorithms. Common techniques include:
 One-hot encoding: Create binary columns for each category, where
1 indicates the presence of the category and 0 indicates absence.
 Label encoding: Map each category to a unique integer. This is
suitable for ordinal categorical variables with a natural order.
 Target encoding: Encode categorical variables based on the target
variable's mean or other aggregated metrics. This can be useful for
high-cardinality categorical variables.
3. Feature Scaling:
 Scale numerical features to a similar range to prevent features with larger
magnitudes from dominating the model. Common scaling techniques
include:
 Min-max scaling (Normalization): Scale features to a range between
0 and 1.
 Page 6 of 8

 Standardization: Transform features to have a mean of 0 and a

standard deviation of 1.
 Robust scaling: Scale features using median and interquartile range
to handle outliers.
4. Feature Engineering:
 Create new features or transform existing ones to capture meaningful
information and improve the model's performance. Feature engineering
techniques include:
 Polynomial features: Generate polynomial combinations of features
to capture nonlinear relationships.
 Interaction terms: Create new features by taking the product or
ratio of existing features.
 Domain-specific transformations: Apply domain knowledge to
create features relevant to the problem.
5. Dimensionality Reduction:
 Reduce the number of features to alleviate the curse of dimensionality and
improve computational efficiency. Techniques include:
 Principal Component Analysis (PCA): Project data onto a lower-
dimensional subspace while preserving the maximum variance.
 Feature selection: Select a subset of relevant features based on
statistical tests, feature importance scores, or domain knowledge.
6. Data Splitting:
 Split the dataset into training, validation, and test sets to evaluate the
model's performance. Common splits include:
 Training set: Used to train the model.
 Validation set: Used to tune hyperparameters and assess model
performance during training.
 Test set: Held out for final evaluation to estimate the model's
generalization performance on unseen data.
7. Data Pipeline:
 Create a data preprocessing pipeline that encapsulates all the data
preparation steps. This ensures consistency and facilitates reproducibility
when applying the preprocessing steps to new data.
8. Documentation and Versioning:
 Document all data preparation steps, including assumptions,
transformations, and preprocessing techniques applied. Version control
the data preprocessing pipeline to track changes and ensure
reproducibility.
Select and Train a Model:
 Page 7 of 8

Selecting and training a model in the data preparation phase of machine learning
involves choosing an appropriate algorithm, training it on the prepared dataset, and
evaluating its performance. Here's a step-by-step guide:

1. Choose a Model:
 Select a machine learning algorithm suitable for your problem based on
factors such as the nature of the data (e.g., classification, regression), size
of the dataset, interpretability requirements, and computational resources.
 Common algorithms include linear regression, logistic regression, decision
trees, random forests, support vector machines (SVM), k-nearest neighbors
(KNN), and neural networks.
2. Prepare the Data:
 Ensure that the dataset is properly cleaned, preprocessed, and split into
training and testing sets as described earlier in the data preparation
process.
3. Train the Model:
 Fit the selected model to the training data using the fit() method or
equivalent in your chosen machine learning library (e.g., scikit-learn in
Python).
 Provide the training features (X_train ) and the corresponding target labels
(y_train ) as input to the fit() method.
4. Model Evaluation:
 Evaluate the trained model's performance using appropriate evaluation
metrics based on the problem type (e.g., accuracy, precision, recall, F1-
score for classification; mean squared error, R-squared for regression).
 Calculate the performance metrics on the test set using the predict( )
method to generate predictions and compare them with the actual target
labels (y_test ).
 Visualize the model's performance using relevant plots such as confusion
matrices, ROC curves (for binary classification), or calibration plots.
5. Hyperparameter Tuning:
 Fine-tune the model's hyperparameters to improve its performance. This
involves searching over a predefined hyperparameter space using
techniques like grid search or random search.
 Use cross-validation to estimate the model's performance on different
subsets of the training data and avoid overfitting.
6. Model Selection:
 Compare the performance of different models using cross-validation or a
separate validation set.
 Page 8 of 8

 Select the model with the best performance based on the evaluation
metrics and your specific requirements (e.g., accuracy, interpretability,
computational efficiency).
7. Training Pipeline:
 Create a training pipeline that encapsulates the data preparation, model
training, and evaluation steps. This ensures reproducibility and facilitates
experimentation with different algorithms and hyperparameters.
8. Documentation and Reporting:
 Document the model selection process, including the chosen algorithm,
hyperparameters, and evaluation results.
 Provide insights into the model's strengths, weaknesses, and potential
areas for improvement.