By: Liana A.

Eich
Publishied by: Amazon
Copyright © 2023
TABLE OF CONTENTS
Introduction to Python and Data Science
Installation and Configuration of the Environment
Overview of Libraries
Basic Programming with Python
Variables and Data
Typing in Python
Operators and Expressions
Arithmetic Operators:
Comparison operators:
Assignment operators:
Logical operators
Member operators
Identity operators
Ternary operators
Conditional assignment operators
Control Structures
If structure
For structure
Try-except structure
While structure
With structure
Functions
Function definition
Default value arguments
Importing and exporting data
Exporting Data
Data cleaning and Pre-processing
Data Wrangling and Transformation
 Data Manipulation in Pandas
Loading data into a DataFrame
Selecting Data
Transforming Dados
Data Cleaning
Treating Missing Values
Treating Outliers
Treating Typing Errors
Conclusion
Exploring Data with the power of exploratory analysis (EDA)
Descriptive Statistics
Measures of central tendency
Measures of dispersion
Measures of shape
Frequencies and proportions
Visualization with Matplotlib
Seaborn and Plotly for Advanced Visualization
Machine Learning with Python
Regression Analysis
Classification Analysis
Clustering Analysis
Time Series Analysis
Natural Language Processing (NLP)
Text Classification and Sentiment Analysis
Deep Learning with TensorFlow
TensorFlow
Introduction to Python and Data Science
Well, well, well, if you're looking for a programming
language for data science, look no further! Python is the
champion of the category and has a lot to offer, my dear
friend.
Where to start? Easy peasy! Python is a high-level language,
which means that even if you don't have Einstein's IQ, you
can understand it! This allows you to focus on solving your
data science problems instead of getting stuck in complex
and hard-to-understand codes.
And there's more: you don't need to worry about
memorizing difficult commands or doing advanced math
calculations. Python has a plethora of libraries and tools
ready to use, which is like having a personal assistant for
data science! It's like the famous saying, "it's paid, it's
solved."
But hold on a second, the best part of using Python for data
science is its community. It's like having a group of friends
who are all experts in data science. If you get stuck on a
problem or have a question, you can count on the
community to help you out. And if you're looking for a
specific library or tool, you'll probably find someone who has
already developed and shared it with the community. Have
you ever thought that even your crush could be there, huh?!
And as if that weren't enough, Python is super versatile. It's
like a Swiss Army knife of programming. You can integrate
Python with other programming languages and tools,
making it the ideal tool for any data science task. It's like a
variety TV show!
And the best part of all: the demand for Python in data
science is growing more and more. Companies in various
sectors are looking for data scientists and analysts who
know how to use Python. This means that by learning
Python, you will be in a privileged position in the job market.
You will be the king or queen of data science! Get ready for
the coronation!

Installation and Configuration of the Environment

Well, well, it seems like you're excited to start learning
Python for data science. But before diving into the data, we
need to set up the working environment. Don't worry, it's
not as scary as it sounds. And what if I told you that we can
turn all of this into a big playtime?
The first step is to install Python on your computer. And hey,
we have two main versions: 2.x and 3.x. Now, while the 2.x
version is still widely used, we recommend that you install
the 3.x version since it's the latest version and has more
features. And speaking of which, the 2.x version is being
discontinued, so don't be that person who gets stuck in the
past, okay? That's not cool at all.
After installing Python, you'll need to install an IDE
(Integrated Development Environment) to write and run
your code. There are several options available, but we
recommend Anaconda. It's easy to use and has a large
support community. Plus, Anaconda already comes with
several data science libraries installed. It's like having a
complete survival kit to explore your data.
Now that you have Python and Anaconda installed, we need
to create a virtual environment. You know, it's like a secret
room to store your programming toys. This way, you can
install specific libraries for each project without worrying
about affecting other projects. And if something goes wrong,
just delete the secret room, and start over. How practical,
isn't it?
Since you have a virtual environment, it's time to install the
data science libraries you need. Some of the most popular
libraries include Pandas, Numpy, Matplotlib, and Scikit-learn.
And remember, the more libraries you install, the more
resources you'll have to solve data science problems. It's
like having a box full of toys! And we love toys, don't we?
Finally, you need to check if everything is working correctly.
You can create a test file and run it in your IDE. If everything
is working as it should, you'll see a simple greeting message
like "Hello, world!". In summary, installing and setting up
the Python environment for data science may seem
daunting, but it's an important task that must be done
before exploring your data. And remember, if you encounter
any problems, just seek help from the Python community.
They are like Mickey's crew, always very friendly and
welcoming!
Now let's go through a step-by-step guide that will make
your life even easier:

1. Make sure you have Python installed on your

computer. If you haven't already, you can download
the latest version of Python at
https://www.python.org/downloads/ and follow the
installation instructions.
2. Choose an IDE (Integrated Development
Environment) to program in Python. Some popular
options include IDLE (standard with Python),
PyCharm, and Jupyter Notebook. Choose the one
you feel most comfortable with and install it.
3. Now, let's install the necessary libraries for data
science. To do this, we need to use the terminal or
command prompt. In Windows, click on the "Start"
button and type "cmd" in the search bar. On Mac,
open the "Terminal" through the "Utilities" folder.
4. Once inside the terminal or command prompt, type
"pip install numpy" (or another library you want to
install, such as pandas or Matplotlib). Wait until the
installation process is complete.
 5. To check if the installation was successful, create a
simple Python file in your IDE and try to import the
library you just installed. If there are no errors, it
means everything is working correctly.
6. And that's it! Now you have access to all the
amazing libraries to help you play with your data
and make your life as a data scientist much more
fun. Have fun!
And now, we're ready to start the real fun!

Overview of Libraries
Python is simply amazing for data science, and do you know
why? Because this language has so many wonderful
libraries that make data analysis a true game. So let's take
a look at the most popular ones.
Numpy is like a magic wand for matrices and arrays.
Whether you want to calculate statistics, perform complex
calculations, or just find the maximum number in a dataset,
Numpy is the answer to all these problems.
Pandas is like the grandma of data science. She has all the
data organization tools you need, from grouping data to
adding new records to a table. And even better, it's super-
fast and efficient, so you can do your work without worrying
about speed.
Matplotlib is the king of charts. With it, you can create all
kinds of charts, from line charts to colorful scatter plots. And
if you're lazy like me, you can also use the "plot" method to
create charts in seconds. Matplotlib is like the best friend
you can have, especially when it comes to charts.
Seaborn is the chart library for people with good taste. It's
built on top of Matplotlib but adds a host of beautiful styles
and extra features. If you want to impress your colleagues
with high-quality charts, Seaborn is the right choice. It's like
the fashion designer of charts!
Scikit-learn is the machine learning library you need. If you
want to perform common tasks like classification,
regression, or clustering, Scikit-learn has everything you
need. And best of all, the documentation is clear and
detailed, so you can start playing with machine learning in
no time. It's like the baby monitor of machine learning.
And finally, we have Tensorflow. This library is the icing on
the cake for anyone who wants to venture into the world of
deep learning. It's used by data scientists and machine
learning engineers around the world to build amazing
models. It's like the supreme wizard of artificial intelligence!
So, if you want to have a powerful arsenal for data science,
choose your best weapons wisely. And with these libraries,
you'll be prepared for any challenge that data analysis may
bring you!
Basic Programming with Python
Are you ready to set sail into the world of Python
programming? Well shiver me timbers, ye better be! Python
is the treasure map to the land of data science, and ye
won't want to miss out on all the bounty it has to offer.
But before ye start digging for gold with all the fancy
libraries, ye need to know the basics of Python
programming. Don't ye worry, ye don't need to be a
mathematical whiz or a computer expert to learn Python.
It's a simple and easy-to-learn language, perfect for
beginners.
Let's start with some basic concepts. In Python, ye can store
information in variables, like numbers or strings. Ye can also
perform basic math operations, like addition, subtraction,
multiplication, and division. And of course, ye can write
conditional statements to control the flow of your code, like
"if this happens, do that."
But the real treasure of Python lies in its data manipulation
capabilities. Ye can work with lists, dictionaries, and other
data structures to organize and manipulate information. And
if ye want to do something really cool, like processing large
amounts of data or creating amazing visualizations, ye can
use libraries like Pandas and Matplotlib.
So, hoist the main sail and let's get started, me hearties!
The treasure trove of Python programming awaits!

Variables and Data

It's like packing a suitcase, but instead of clothes, you're
storing data. You can put all sorts of stuff in your suitcase,
like integers, floating-point numbers, strings, and more. Just
like you choose the right suitcase for your clothes, you also
need to choose the right variable type to store your data.
There are several types of data, but let's focus on three of
them: integers, strings, and booleans. Integers are whole
numbers, like 1, 2, and 3. Strings are sequences of
characters, like "Hello, world!" and "I love data science."
Booleans are true or false values, like True or False.
Now, imagine you have a suitcase to store your t-shirts. But
instead of putting your t-shirts in the suitcase, you put your
shoes. That's what happens when you use the wrong
variable type to store your data. It can cause problems and
confusion in the future, so it's important to choose the right
variable type for each type of data.
In summary, variables are like suitcases for storing your
data and data types are like labels for the suitcases. Choose
the right suitcase for each type of clothing, and you'll be
well on your way to becoming a data science pro.
Declaring variables in Python is very simple and
straightforward. You just need to choose a name for the
variable and assign a value to it using the equals sign (=).
For example:
name_variable = value

Here we go! Let's get some examples of how to declare

variables in Python:
name_variable = value

Some practical examples of declaring variables in Python include:

# integer variable
x = 10
# string variable
name = "John"
# integer variable
age = 30
# floating point variable
height = 1.75
# boolean variable
legal_age = True

You can also assign multiple values to multiple variables at

once, like this:
x, name, age, height, legal_age = 10, "John", 30, 1.75, True

Remember that the name of a variable must begin with a

letter or underscore (_) and cannot contain spaces or special
characters. Additionally, the name of a variable should be
descriptive and meaningful for its contents.

Typing in Python
Well, imagine if you had a friend who always wanted to
know exactly what you were wearing before going out with
him. And if that friend had to ask every time you went out,
even though you had told him thousands of times that you
were wearing a white t-shirt. That would be tiring, right?
That's where strongly typed language comes in. It's like that
curious friend: always wants to know exactly what you're
doing and how you're doing it. This can be useful in some
situations, but in others, it can be a headache.
But Python is different. It is a dynamically typed language,
which means it does not need to know exactly what you are
doing or how you are doing it before going out with you.
Instead, it trusts you and lets you do what you want.
This means that you can change the "clothing" of a variable
as many times as you want, without having to notify Python.
For example, you can declare a variable as an integer, but
then change it to a string or even to a list.
In summary, Python's dynamically typed language allows
you to be more flexible and creative in your programming,
without worrying about the limitations of strongly typed
language. It's like having a friend who trusts you and gives
you the freedom to be yourself!
Operators and Expressions
Operators and expressions in Python are like spices in the
kitchen: without them, the food becomes bland! But don't
worry, even if you're not a chef, you can use arithmetic
operators to add, subtract, multiply, and divide your data
ingredients. Just remember to use "//" for integer division
and avoid making a soup of decimal numbers.
Comparison operators are like a food critic: they evaluate
your data and say whether it's delicious or not. If your data
doesn't measure up, you can filter it with operators like
"==" and ">". And if you're like me, you can make decisions
based on your taste preferences.
Finally, assignment operators are like your cooking hands:
they take the ingredients and put them in a pot. Use the "="
operator to assign values to variables and create a unique
data recipe.
In summary, operators and expressions are the spices that
give flavor to your data analysis. And, as every chef knows,
it's important to use them wisely to create a delicious dish!

Arithmetic Operators:
Subtraction

result = 4 - 2
print(result) # 2

Multiplication

result = 2 * 2
print(result) # 4

Division

result = 4 / 2
print(result) # 2.0

Integer Division

result = 5 // 2
print(result) # 2

Comparison operators:
Equality

result = 2 == 2
print(result) # True

Difference

result = 2 != 2
print(result) # False

Greater than

result = 2 > 1
print(result) # True

Less than

result = 2 < 3
print(result) # True

Assignment operators:
Assigning a value to a variable

age = 10
print(age) # 10

Addition and assignment

age = 10
age += 5
print(age) # 15

Subtraction and assignment

age = 10
age -= 5
print(age) # 5

Examples:
String manipulation is like creating art with letters. In
Python, you can use the addition operator to join two strings
and create something completely new. For example, you
can join "Hello" with "world" to create the famous phrase
"Hello, world!". Or, if you are a data artist, you can join
column names in a dataframe to create a new dataset.
And if you want to repeat the same word multiple times, the
multiplication operator is your best friend. For example, you
can use the "*" operator to repeat the word "data" three
times and create the string "datadatadata". It's like creating
a stencil of letters to print on different parts of your work.
So, next time you are working with strings in Python,
remember that operators are your art tools. Use them
wisely and create something amazing!
See the following example:

# String concatenation
name = "John"
surname = "Walter"
full_name = name + " " + surname
print(full_name) # "John Walter"

# String repetition
line = "-" * 20
print(line) # "--------------------"

In addition, we can also use expressions to dynamically

format strings. For example, we can use the following
syntax:
name = "John"
age = 30
sentence = "My name is {name} and I am {age} years old."
print(sentence) # "My name is John and I am 30 years old."

Logical operators
Logical operators: and, or, not. They are used to combine
logical expressions and determine whether an expression is
true or false. For example:
age = 30
legal_age = age > 16
has_DriverLicence = True
can_drive = legal_age and has_DriverLicence
print(can_drive) # True

Member operators
Operators of membership: in and not in. They are used to
check if a certain element is present in a list or other data
structure. For example:

names = ["John", "Mary", "Peter"]

name = "John"
is_present = name in names
print(is_present) # True

Identity operators
Operators of identity: is and is not. They are used to check if
two variables point to the same object in memory. For
example:
list1 = [1, 2, 3]
list2 = [1, 2, 3]
are_equal = list1 is list2
print(are_equal) # False

Ternary operators
Ternary operators: x if c else y. They are used to define a
value for a variable based on a condition. For exemple:
age = 30
message = " You are of legal age." if age >= 21 else "You are a minor."
print(message) # " You are of legal age."
Conditional assignment operators
Conditional assignment operators: x += y, x -= y, and so
on. They are a combination of assignment and arithmetic
operation. For example:
age = 30
age += 5
print(age) # 35

Here's an example of Python code that illustrates the use of

various logical operators in data analysis, using School
scores:

import pandas as pd

# Loading School grade data

data = pd.read_csv("school.csv")

# Selecting only the known math grades

data = data[data["MATH_GRADES"].notnull()]

# Creating a new column to indicate whether the participant passed or not in

the math test.
minimum_grade= 400
data["Passed_Math"] = data["MATH_GRADES"] >= minimum_grade

# Selecting only participants who passed the math test and have a grade in
biology
data = data[(data["Passed_Math"] == True) &
(data["BIOLOGY_GRADES"].notnull())]

# Calculating the average of biology scores for participants who passed the
math test.
biology_average = data ["BIOLOGY_GRADES"].mean()
print("Average of the biology scores of the participants who passed the math
test:", biology_average)

In this example, we used the "notnull" and "&" logical

operators to sift through the data and select only the most
relevant information for our analysis. It's like searching for a
gold nugget in a stream of data.
Furthermore, we used the ">=" comparison operator to
determine if the participants passed the math test or if they
need to study a bit more. It's like having a thermometer to
measure the temperature of each person's knowledge.
And to have a general idea of the participants' performance,
we calculated the average of the essay scores. It's like doing
a math calculation to find the answer to the question "how
good are these students?".
In summary, operators and expressions are the key to
unlocking the true potential of data. So, use them wisely
and make incredible discoveries! Note that we used Pandas
to import the data. Pandas is an open-source library in
Python that provides data structures and data analysis
tools. It is widely used in data science applications such as
exploratory analysis, data processing, data visualization,
and data modeling. Don't worry about libraries for now, we'll
talk more about this topic later on.
Control Structures
The if structure is like the Cupid of programming, it allows
you to make decisions based on certain conditions. If you
want to make a program that only greets people named
"João," for example, you can use the if structure to check if
the person's name is "João" before greeting them. And if the
name is not "João," the program simply ignores that person.
Seems like an excellent way to avoid handshakes in vain!
The for structure is like a magician who performs a
sequence of tricks. With it, you can execute a set of
commands several times, without having to write
everything multiple times. For example, if you want to
calculate the average of a list of numbers, you can use the
for structure to loop through the list and add up all the
numbers, then divide the result by the number of elements.
Easy as magic!
Finally, the while structure is like an endless road trip in
programming. With it, you can execute a set of commands
repeatedly, while a certain condition is true. This can be
useful when you need to perform a task until a certain
condition is met. For example, you can use the while
structure to iterate through a list until a certain value is
found. It's like an adventure race, you never know where it's
going to end up!
In summary, control structures are like magical tools that
allow us to create intelligent and automated programs. With
them, we can do everything from greeting people with
specific names to looping through infinite lists. So, when it
comes to programming, always remember that you are the
magician, and the control structures are your magic wands!

If structure
The if structure is used to perform a specific action if a
certain condition is true. The basic syntax is as follows:
if condition:
# Code to be executed if the condition is true

For example, imagine that you have an age column in a

dataset and want to create a column that indicates whether
the person is over 18 or not. You can use the if structure to
do this:

import pandas as pd

# Loading a dataset of Titanic passengers

data = pd.read_csv("titanic.csv")

# Creating a new column to indicate if the passenger is an adult

data["Legal_Age"] = False
for i in range(len(data)):
if data.loc[i, "Age"] >= 18:
data.loc[i, "Legal_Age"] = True

In this example, we used the if structure to check if each

passenger's age is greater than or equal to 18. If the
condition is true, we set the new column "Legal_Age" to
True, otherwise we keep the default value as False.
For structure
The for loop is used to repeat an action a determined
number of times. The basic syntax is as follows:

for item in object:

# Code to be executed for each item

For example, imagine that you want to calculate the

average math scores for each Brazilian state. You can use
the for loop structure to do that:

import pandas as pd
# Loading School grades data
data = pd.read_csv("school.csv")

# Selecting only the math scores with known value

data = data[data["MATH_GRADES"

Try-except structure
Structure try-except: it is used to handle errors and
exceptions in your code. The basic syntax is as follows:
try:
# Code that may raise an exception
except Exception as and:
# Code to be executed if an exception occurs

For example, imagine that you are trying to open a file and
want to handle the case of the file not existing. You can use
the try-except structure to do this:
try:
with open("data.csv", "r") as file:
data = file.read()
except FileNotFoundError:
print("File not found.")

While structure
While loop: is used to repeat an action while a certain
condition is true. The basic syntax is as follows:

while condition:
# The code to be executed while the condition is true

For example, imagine that you want to read data from a

source until there is no more data to be read. You can use
the while structure to do this:
data = []
line = input("Enter a data line (or press Enter to finish):")
while line:
data.append(line)
line = input("Enter a data line (or press Enter to finish):")

With structure
The "with" statement is used to ensure that resources, such
as files, are properly released regardless of exceptions or
errors. The basic syntax is as follows:

with object as variable:

# Code to be executed with the object

For example, imagine that you want to open a file for writing
and ensure that it is closed properly, regardless of
exceptions or errors. You can use the with structure to do
this:
data = ["Line 1", "Line 2", "Line 3"]
with open("data.txt", "w") as file:
for data in data:
file.write(data + "\n")
Functions
Functions, also known as the unsung heroes of
programming, are like personal assistants that you can call
on to help with specific tasks. They receive clear
instructions on what to do, work diligently in the
background, and return with a successful result. And the
best part is, you can call them again and again, without
needing to repeat the code.
Creating functions is like assembling your own custom
programming toolkit. You can create functions to handle
complex calculations, data manipulation, chart generation,
and more. And, just like any toolkit, you can add or remove
tools as your needs change.
In addition, by creating functions, you are following a good
programming practice. This means that you are creating
cleaner, organized, and easier-to-understand code. Your
code becomes more efficient, allowing you to focus on more
important tasks and let the functions work hard for you.
So, don't underestimate the power of functions in Python.
They are your friends and allies in the programming journey.
And, like any friendship, the more you invest in it, the more
it will grow and become a fundamental part of your
programming process.

Function definition
The basic syntax for defining a function in Python is as
follows:

def name_function(arguments):
# Code to be executed in the function
return value
For example, imagine that you want to calculate the
average of three numbers. You can create a function to do
so:

def average(a, b, c):

return (a + b + c) / 3

# Using the function to calculate the average of three numbers

result = average(2, 4, 6)
print(result) # Result: 4.0

In this example, we defined a function called "mean" that

accepts three arguments (a, b, and c) and returns the mean
of the three numbers. Then, we used the function to
calculate the mean of 2, 4, and 6 and stored the result in a
variable.

Default value arguments

You can specify default values for the arguments of a
function in case they are not provided. The syntax is as
follows:

def name_function(argument1=default_value1, argument2=default_value2):

# Code to be executed in the function
return value

For example, imagine that you want to calculate the

average of three numbers, but want to specify a default
value for the third number if it's not provided. You can do
this in the following way:

def average(a, b, c=1):

return (a + b + c) / 3

# Using the function with three

Here's another example of a function performing a fictional
analysis of School exam results:

import pandas as pd

# Loading School data

data = pd.read_csv("school.csv")

# Definition of the function to calculate the final grade

def calculate_grade(line):
weight = [0.3, 0.3, 0.2, 0.1, 0.1]
grade = [line["BIOLOGY_GRADES"], line["HISTORY_GRADES"],
line["ENGLISH_GRADES"], line["MATH_GRADES"], line["ESSAY_GRADES"]]
final_grade = sum([grade[i] * weight [i] for i in range(len(grade)) if not
pd.isna(grade[i])])
return final_grade

# Applying the function to all rows of the dataset

data["FINAL_GRADE"] = data.apply(calculate_grade, axis=1)

# Displaying the first rows of the dataset with the final grade
print(data.head())

That's right! This School database example is huge and can

be difficult to analyze in tools like Excel. That's where
Python and its powerful data analysis libraries like Pandas
come in. With Pandas, we can efficiently load, manipulate,
and analyze large datasets without complications.
Additionally, with the help of Python's functions and control
structures, we can automate complex tasks and gain
valuable insights from the data.
Importing and exporting data
Ah, importing data... What would we, data scientists, do
without this skill? It doesn't matter if the data comes from a
CSV, Excel or JSON file, we always need to deal with this
ungrateful task. But with Python, everything becomes
easier! With the Pandas library, importing CSV files is as
easy as eating a slice of pizza (and that's already pretty
easy). Just take a look at this code here:

import pandas as pd
data = pd.read_csv("data.csv")

Excel is a popular tool for working with data, and Python is

not far behind when it comes to importing data from Excel
spreadsheets. With the Pandas library, you can easily import
Excel files. To import an Excel worksheet into a Pandas
DataFrame, simply use the "read_excel" function. For
example, the following code imports a worksheet called
"data" from an Excel file called "data.xlsx" into a Pandas
DataFrame:

import pandas as pd
data = pd.read_excel("data.xlsx", sheet_name="dados")

You can use the "sheet_name" argument to specify which

Excel worksheet you want to import. If you omit this
argument, the Pandas library will import the first worksheet
by default. Additionally, you can use other arguments such
as "header" to specify which row should be used as the
column header and "usecols" to specify which columns to
import. With these features, you can customize the data
import to meet your needs.

JSON (JavaScript Object Notation): You can import JSON files

using the Pandas library. For example, the following code
imports a JSON file named "data.json" into a Pandas
DataFrame:

import pandas as pd
data = pd.read_json("data.json")
Exporting Data
Hey, did you know that you can use the power of Python to
export your data and impress the world with your analysis?
That's right, and it's not as hard as it seems. To export data
in CSV format, you just need to use the Pandas library and
its "to_csv" method. For example, with the code below, you
can export a Pandas DataFrame to a CSV file named
"exported_data.csv":

import pandas as pd
data.to_csv("exported_data.csv", index=False)

Excel: You can export data to an Excel file using the Pandas
library. For example, the following code exports a Pandas
DataFrame to a sheet named "exported _data" in an Excel
file called " exported _data.xlsx":

import pandas as pd
data.to_excel("exported_data.xlsx", sheet_name=" exported_data",
index=False)

JSON: You can export data to a JSON file using the Pandas
library. For example, the following code exports a Pandas
DataFrame to a JSON file named "exported_data.json":

import pandas as pd
data.to_json("exported_data.json")

Nice! Data science doesn't have to be boring, and we can

have a lot of fun with Python libraries. With the Pandas
library, importing and exporting data becomes as easy as
eating carrot cake, but without the calories, of course!
And believe me, importing and exporting data is the
foundation for many other fun activities we can do with
Python. In this example below, let's analyze fictional crime
data in Brazil, because who doesn't love a good crime story?
First, we import the data from a CSV file and transform it
into a DataFrame. Then, we eliminate rows that are not
relevant to our analysis and calculate the average crime
index in each Brazilian state.
But it doesn't stop there! Afterwards, we select only the
states with a crime index above the average and sort them
in descending order. To wrap it up, we export the relevant
information to an Excel file called "crime_index.xlsx". And
voilà, we have an organized file with interesting information
about crime in Brazil!

import pandas as pd
import matplotlib.pyplot as plt

# Importing data from CSV file

data = pd.read_csv("crime_rate.csv")

# Displaying general information about the data

print(data.info())

# Removing missing values

data.dropna(inplace=True)

# Grouping data by state and calculating the average crime rate

grouped_data = data.groupby("state").mean()

# Sorting the grouped data by crime rate

grouped_data.sort_values("crime_rate", ascending=False, inplace=True)

# Displaying the 5 states with the highest crime rate

print(grouped_data.head(5))

# Creating a bar chart to visualize the data

grouped_data.plot(kind="bar", y="crime_rate", figsize=(12, 6))
plt.xlabel("State")
plt.ylabel("Crime rate")
plt.title("Crime rate by state")
plt.show()

# Exporting data to an Excel file

Grouped_data.to_excel("treated_crime_rate.xlsx",
sheet_name="processed_data")

This code is like the Batman of Data Science, fighting

against the crime of disorganized and crazy data. It uses
advanced analysis and manipulation techniques to tame
these wild data and produce useful and interesting
information. The code starts by importing data from a CSV
file, removing missing values, and grouping the data by
state. Then, it performs a calculation to find the average
crime index and sorts the data in descending order so that
we can find the most dangerous states in Brazil. But the
fight is not over yet! To better visualize the data, the code
creates a bar chart using the Matplotlib library. And, finally,
the result of this epic battle is exported to an Excel file, now
organized and much easier to understand. That's how you
win the battle of data, my friend!
Data cleaning and Pre-processing
This step is like cleaning the house before receiving visitors.
You need to sweep the floor, dust off, and tidy things up.
Similarly, before starting to analyze the data, it is necessary
to do some cleaning and get everything organized.
In the Data Cleaning and Preprocessing process, several
tasks may be necessary, such as handling missing values,
removing duplicates, correcting typos, standardizing
formats, normalizing data, and much more. It is a
meticulous job, but essential to ensure that the analysis
results are accurate and reliable.
Fortunately, Python offers several libraries to facilitate the
task of cleaning and pre-processing data, such as Pandas,
Numpy, and Scikit-learn. These tools allow for importing
data in various formats, applying filters, standardizing
formats, detecting outliers, normalizing data, among other
tasks.
In summary, Data Cleaning and Preprocessing is an
essential step in data analysis and should be executed with
care and attention to detail. With the right tools, such as
Python libraries, this step can be much easier and efficient,
allowing you to spend more time on data analysis and
interpretation.

Some common tasks in data cleaning and preprocessing

include:

1. Removal of missing values: It is common for raw

data to contain missing values, which can hinder
future analysis. Therefore, it is necessary to identify
and remove these values.
2. Treatment of duplicate data: It is important to check
for and remove any duplicate data, as it can hinder
analysis.
 3. Treatment of inconsistent data: It is important to
check for data consistency and correct any errors or
inconsistent values.
4. Data type conversion: It is important to verify that
all data types are correct and, if necessary, convert
them to the correct type.
5. Data normalization: It is important to normalize the
data to avoid analysis being affected by scale
differences.
6. Data grouping: It is common to group data by
categories or other criteria to facilitate future
analysis.

These are just a few examples of how Pandas can be used in

data cleaning and preprocessing. It is important to
remember that each dataset may have its own
particularities and, therefore, may require different cleaning
and preprocessing steps. It is important to be familiar with
the different tools available in Pandas to be able to choose
the most appropriate ones for each situation.

Removal of missing values:

import pandas as pd

data = pd.read_csv("raw_data.csv")
data.dropna(inplace=True)

Duplicate data handling:

import pandas as pd

data = pd.read_csv("raw_data.csv")
data.drop_duplicates(inplace=True)

Data type conversion:

import pandas as pd
data = pd.read_csv("raw_data.csv")
data["column_1"] = data["column_1"].

Here is a complete example of data preprocessing on a

fictional dataset of student registration in a school using the
Pandas library in Python:

import pandas as pd

# Loading the database

data = pd.read_csv("student_registration.csv")

# Checking for missing values

print("Missing values before cleaning:")
print(data.isnull().sum())

# Removing missing values

data.dropna(inplace=True)

# Checking for missing values again

print("Missing values after cleaning:")
print(data.isnull().sum())

# Checking for duplicated data

print("Duplicated data before cleaning:")
print(data.duplicated().sum())

# Removing duplicated data

data.drop_duplicates(inplace=True)

# Checking for duplicated data again

print("Duplicated data after cleaning:")
print(data.duplicated().sum())

# Checking data types

print("Data types before conversion:")
print(data.dtypes)

# Converting columns to the correct data type

data["age"] = data["age"].astype(int)
data["enrolled"] = data["enrolled"].astype(bool)

# Checking data types again

print("Data types after conversion:")
print(data.dtypes)

# Saving the cleaned and pre-processed database

data.to_csv("student_registration_clean.csv", index=False)
Data Wrangling and Transformation
Welcome to the Data Wrangling and Transformation chapter!
Here, we will show you how to prepare your raw data for
analysis in a fun and entertaining way (or at least try to).
Data preparation may seem boring, but it is essential to
ensure that your analyses are based on accurate and
reliable information. After all, we do not want to rely on data
that was collected suspiciously, or on information that is
incomplete or inaccurate.
Fortunately, with the help of Python and the Pandas library,
we can transform raw data into useful information easily
and efficiently. We will show you some fun examples of how
to do this!
But before we dive into the data wrangling and
transformation techniques, it is important to remember that
this step is critical to any data science project. So, take a
deep breath and let's get started!

Data Manipulation in Pandas

The DataFrame and Series are like inseparable siblings in
the Pandas library. The DataFrame is the older brother,
responsible for managing all information in the form of an
organized and elegant table, while the Series is the younger
brother, a smaller and simpler version that has the main
function of storing a single column of data.
But don't be fooled, even though it's smaller, the Series is
very powerful! It can be used to store data of different
types, such as numbers, strings, dates, and more. And with
the Pandas functions, we can easily manipulate this data in
various ways, such as filtering, sorting, aggregating, and
visualizing.
The DataFrame, on the other hand, is the boss of the gang!
With it, we can store and manage large amounts of data in
an organized and efficient way. In addition, with the Pandas
functions, we can easily transform and manipulate the data
according to our needs, allowing us to perform complex
analyses with ease.
In summary, the DataFrame and Series are the stars of the
Pandas library and form a dynamic duo in the manipulation
and transformation of data in Python.

Loading data into a DataFrame

The Pandas library's read_csv() function is a powerful tool
for importing data in CSV format into a DataFrame. And if
you're wondering what a CSV is, imagine a simple text file
that stores data in a table format, where each row
represents a new row in the table and each column is
separated by a specific character, usually a comma. From
this file, Pandas creates a DataFrame that can be easily
manipulated and analyzed. And best of all, the library offers
a range of parameters and options to customize the reading
process, including specifying the separator character,
automatic header detection, and more.

import pandas as pd
df = pd.read_csv('file.csv')

This loads the data from the CSV file into a DataFrame. If
the CSV file does not have a header, we can specify this
using the header=None parameter. If there are missing
values in the CSV file, we can specify how they are
represented using the na_values parameter.

Selecting Data
To select a specific column, we can use bracket notation
with the column name. For example, if we have a
DataFrame called "data" and we want to select the "age"
column, we can do:
data["age"]

This will return a Series containing all values in the "age"

column. We can also select multiple columns by passing a
list of column names, for example:

data[["age", "gender"]]

This will return a new DataFrame containing only the "age"

and "gender" columns.

To select specific rows, we use the loc[] or iloc[] functions.

The loc[] function allows us to select rows by label, while the
iloc[] function allows us to select rows by index:

df.loc[0] # select the first row

df.iloc[0] # select the row with index 0

We can also select rows based on a condition:

df[df['column'] > 10]

This returns all rows where the column value is greater than
10.
Transforming Dados
Pandas offers a wide range of functions for transforming
data in a DataFrame. Some examples include:
Adding a column: we can add a new column to the
DataFrame using bracket notation:
df['total_value'] = df['price'] * df['quantity']

This adds a new column to the DataFrame with the name

"total_value" and the values calculated from the "price" and
"quantity" columns.

df['column'].apply(lambda x: x.upper())

This applies the upper() function to each value in the

"column" column.
Grouping data: we can group data in a DataFrame by one or
more columns using the groupby() function. For example, to
calculate the mean of a column grouped by another column,
we can do the following:

df.groupby('column1')['column2'].mean()

This groups the rows in the DataFrame by unique values in

column column1, calculates the mean of column column2
for each group, and returns the results in a new Series.

Data Cleaning
When it comes to data science, data cleaning and
preprocessing are like that pre-party cleaning: no one wants
to do it, but it needs to be done. After all, who wants to have
a great party in a messy and dirty house, right? The same
goes for data! And that's where the "data wrangling" comes
in, which is a crucial part of any data science project.
And how do we "wrangle" data? With the help of the Pandas
library in Python, of course! With it, we can organize, clean,
and transform the data into something that is suitable for
analysis. And for that, we have to be familiar with the two
main data structures in Pandas: the DataFrame and the
Series.
The DataFrame is like an elegant dining table, all organized
and with defined places. Each column is a Series, which is
like the dishes you would serve on that table. And just as
you can add or remove a dish from the table, Pandas allows
us to add or remove columns from the DataFrame.
In addition, we can select and filter the data using Pandas
functions. It's like choosing which guest will sit in which
place at the table, you know? You can choose a specific
column or filter the data according to certain conditions,
such as "I only want guests who ate vegetables."
But what if one of your guests doesn't show up? Or if
someone introduces themselves as someone else? That's
where data cleaning comes in, which is the least fun part of
the party. Here, we need to identify and deal with missing
values, duplicates, and even typos. Fortunately, Pandas has
many functions that help us deal with these issues.
In short, we can say that "data wrangling" is like that
tedious party preparation, but it is essential for it to be a
success. And with Pandas in Python, we can do this
preparation efficiently and even in a fun way (or at least less
boring).

Treating Missing Values

Missing values are a common problem in real-world data.
They can occur for various reasons, such as measurement
errors, registration failures, or data deletion. Pandas offers
several functions to identify and handle missing values. To
identify missing values in a DataFrame, we can use the
function isnull():
df.isnull()

This returns a boolean DataFrame with the same labels and

indices as the original DataFrame, indicating where the
missing values are present.

We can handle missing values in various ways, such as

removing rows or columns that contain missing values,
filling the missing values with a default value, or
interpolating missing values based on existing data. Some
useful functions for handling missing values are:

dropna(): remove rows or columns that contain missing

values.
fillna(): fills missing values with a default value.
interpolate(): interpolate missing values based on existing
data.

Treating Duplicate Values

Duplicate values are another common issue in data. They
can occur when data is collected from various sources or
due to data entry errors. Pandas offers a function to identify
and remove duplicate values in a DataFrame:
df.drop_duplicates()

This removes all duplicated rows in the DataFrame and

returns a new DataFrame without duplicates.

Treating Outliers
Outliers are like that person who always stands out in the
crowd but for some reason doesn't quite fit in with the rest
of the group. They can cause a stir in your data analysis and
mess everything up! It's like a unicorn appearing in the
middle of a flock of sheep - it gets attention, but it doesn't
make much sense.
But don't worry, identifying outliers is easy! Just use math
and calculate the mean and standard deviation of the
column in question. Then, simply remove those values that
are too far from the mean, like that annoying friend who
insists on disagreeing with everything and everyone just to
get attention.

mean = df['column'].mean()
std = df['column'].std()
df = df[(df['column'] > mean - std) & (df['column'] < mean + std)]

This removes all rows in which the column value is more

than one standard deviation from the mean.

Treating Typing Errors

Have you heard the saying "to err is human, but to persist in
error is foolishness"? In the world of data, this is very true!
Typos are one of the most common problems and,
unfortunately, they can greatly interfere with data analysis.
But don't worry, Pandas is here to save us!
To identify where these errors are, we can use the function
value_counts(), which is like a detective specialized in
counting the number of times each value appears in the
column. But just identifying the error is not enough, we need
to correct it! And for that, we have the replace() function,
which is like a spell checker for data, replacing all the wrong
values with the correct form. It seems like magic, right? But
it's just the power of Pandas!

df['column'].value_counts()
This returns a Series that contains the number of times each
value appears in the column. We can correct typing errors
by replacing incorrect values with correct values using the
replace() function:

df['column'].replace({'wrong_value': 'correct_value'})

This replaces all occurrences of the wrong value with the

string 'correct_value'.

Conclusion
Congratulations, you have reached the end of this chapter!
Now you are ready to take on the challenge of preparing
and transforming raw data into clean and organized data for
analysis. Remember that although data manipulation can be
a tedious and time-consuming task, it is a crucial step in
ensuring accurate and reliable results in your data science
projects.
And if you thought data wrangling is just a tedious and
monotonous activity, think again! With a little creativity and
a sense of humor, you can turn this task into something fun
and interesting. And if you need help, don't forget that
Pandas is your best friend!
We hope this chapter has been helpful and informative.
Good luck on your data wrangling and data transformation
adventures!
Exploring Data with the power of
exploratory analysis (EDA)
Exploratory Data Analysis (EDA) is an important process for
gaining a better understanding of your data before diving
into modeling and analysis. And guess what? Python has a
powerful library called Pandas that makes exploratory data
analysis a lot easier. So, if you're looking for an exciting
adventure in data exploration, come join us on this
incredible journey!
Venturing into EDA:
First of all, we need to have an overview of the data. This
includes checking the first and last lines of the data, the
data types in each column, the presence of missing values,
and the overall distribution of the data. It sounds like a lot,
but don't worry! Pandas makes it easier than convincing
your friend to go on a diet.
Now that we have a general idea of the data, we can dive
deeper and explore the relationships between variables. We
can use graphs and descriptive statistics to understand
trends, patterns, and relationships between variables. And
this can be as fun as bowling with your friends! (Of course, if
you're the nerdy type who finds these things fun. We have
no prejudices here!)
In addition, we can use preprocessing techniques, such as
data standardization or normalization, to ensure that
variables are on comparable scales. This can help avoid bias
in analysis results. And let's be honest, avoiding bias is as
important as avoiding olives on pizza (which, let's agree, is
awful).
But wait, there's more! We can also use principal component
analysis (PCA) to reduce the dimensionality of the data and
visualize the data in a lower-dimensional space. This can be
useful for finding hidden patterns in the data and for
visualizing the data in scatter plots or heat maps. This is as
fun as building sandcastles at the beach!
In summary, exploratory data analysis is an essential
process in data science that helps gain a better
understanding of the data and prepare it for more advanced
analysis. Python and the Pandas library make exploratory
data analysis easy and fun (well, at least more fun than a
boring business meeting). With the right techniques and
tools, you can explore your data like never before and find
valuable insights. So, what are you waiting for? Grab your
explorer hat and start exploring your data today!
Descriptive Statistics
Ah, data! It can be fascinating... or incredibly boring. That's
where descriptive statistics come in, a way to talk about
your data in an interesting and useful way.
Let's start at the beginning: what are descriptive statistics?
Well, basically they are techniques that help to describe and
summarize a set of data. In other words, they turn that
jumble of numbers into a story that you can understand and
tell others.

There are various ways to calculate descriptive statistics,

but here are some of the most common.:

Measures of central tendency

These measures indicate where the "center" of your data is.
The most common is the mean, which is the sum of all
values divided by the number of values. But be careful: if
you have some extreme values, the mean can be
misleading. In this case, the median (the value that is right
in the middle of your data) may be a better option.

Measures of dispersion
These measures indicate how "spread out" your data is. The
most common one is the standard deviation, which indicates
how far each value is from the mean. The larger the
standard deviation, the more "spread out" your data is.
Another measure of dispersion is the range, which is the
difference between the largest and smallest value.

Measures of shape
These measures indicate the shape of the distribution of
your data. The most common one is kurtosis, which
indicates how "flat" or "peaked" the distribution is. If the
kurtosis is high, it means that the distribution is more
"peaked", with more values concentrated near the mean. If
the kurtosis is low, it means that the distribution is more
"flat", with fewer values near the mean.

Frequencies and proportions

These measures indicate how many times each value
appears in your data. The most common one is frequency,
which is simply the number of times a value appears. But
it's also possible to calculate proportions, which indicate the
percentage of times a value appears relative to the total
number of values.
Now that you know a little about the most common
measures, let's see how to calculate these descriptive
statistics in Python using the NumPy library.

To calculate the mean, we can use the mean() function from

NumPy:
import numpy as np

values = [10, 20, 30, 40, 50]

average = np.mean(values)
print(average)

This code returns the mean of the values (which is 30 in this

case).

To calculate the standard deviation, we can use the std()

function from NumPy:
import numpy as np

values = [10, 20, 30, 40, 50]

standard_deviation = np.std(valores)
print(standard_deviation)

This code returns the standard deviation.

Before we move on to advanced data analysis techniques,
it's important to understand the basics of descriptive
statistics. Descriptive statistics are used to summarize and
describe data in a concise and understandable way. These
statistics can include measures of central tendency, such as
the mean and median, as well as measures of dispersion,
such as the standard deviation and interquartile range.
But let's be honest, descriptive statistics are not always the
most exciting thing in the world. It's like looking at a black
and white painting - you know it's important, but it can be a
bit monotonous. So let's spice things up and add a bit of
humor to the mix.
For example, let's talk about the mean. The mean is the
value obtained by adding up all the values in the data set
and dividing by the total number of values. It's basically the
central point of a data set. But how can we make it more
interesting? Well, it's like the filling in a sandwich. You have
a slice of bread on top, a slice of bread on the bottom, and a
delicious filling in the middle. The mean is the filling that
holds all the data together. And who doesn't love a good
sandwich, right?
And how about the median? The median is the middle value
in a data set when they are arranged in ascending or
descending order. It represents the value that divides the
data set in two equal parts. But we can make it more fun.
It's like an obstacle course - you have some high data points
and some low data points, but the median is there in the
middle, jumping over all the obstacles and crossing the
finish line.
Well, these are just some fun ways to think about basic
descriptive statistics. Of course, they are much more than
that, but it's always good to add a bit of humor to the mix.
In summary, descriptive statistics are an important part of
data analysis, allowing us to summarize and describe data
in a concise and understandable way. Pandas offers a wide
range of functions to calculate these statistics, making the
process quick and easy.

Visualization with Matplotlib

Ah, we have finally reached the chapter on visualizations
with Matplotlib! That means we can finally stop looking at a
boring table of numbers and start bringing data to life with
colorful and animated charts.
Matplotlib is a Python visualization library that allows you to
create various types of charts, from simple line charts to
super elaborate bar and pie charts. In addition, Matplotlib
offers a wide range of customization options to make your
charts even more impressive.
One of the simplest visualizations you can create with
Matplotlib is a line chart. You can create a simple line chart
using the "plot" function and passing in the data you want
to plot. Let's take a look at an example:
import matplotlib.pyplot as plt
x = [1, 2, 3, 4, 5]
y = [2, 4, 6, 8, 10]
plt.plot(x, y)
plt.show()
In this example, we defined two lists, one for the x values
and another for the y values. Then, we used the Matplotlib
"plot" function to plot the x and y values on a line graph.
Finally, we used the "show" function to display the graph.
This is just the beginning. With Matplotlib, you can create
bar charts, pie charts, scatter plots, and many other types
of visualizations. You can also customize the appearance of
the graphs by adding titles, axis labels, and legends.
For example, here is an example of a simple bar chart that
shows the monthly sales of a clothing store:

In this example, we used the "bar" function to create a bar

chart, passing the lists of months and sales as arguments.
Then, we added a title to the chart, as well as axis labels to
make it more informative.

But it doesn't stop there! Matplotlib also allows you to

create more complex graphs and even animations. With the
mpl_toolkits library, you can create 3D graphs, while the
Matplotlib animation library allows you to create impressive
animations of moving data.
In summary, Matplotlib is a powerful tool for creating
visualizations in Python. With it, you can transform your
data into interesting and informative graphs, which can help
you better understand your data and communicate your
findings in a clear and impactful way.
In the next example, we'll create a graph so complex (just
kidding, it's not that complex) that even the boldest
mathematicians will be impressed. We will plot the famous
quadratic function f(x) = x^2, with defined limits, and its
derivative f'(x) = 2x. It's not easy to be a respectable graph,
but we can do it.
First, we use Matplotlib's plot() function to draw the
quadratic function and make it shine like the sun in the sky.
Then, we add vertical and horizontal limits using the
axvline() and axhline() functions, because who said limits
only exist in relationships?
But we don't stop there, oh no. To show the function's
derivative, we'll calculate f'(x) = 2x using our mathematical
knowledge, and plot the curved line on the same graph,
because after all, what's a straight line compared to a
wonderful curve?
The end result is a graph that not only visually shows the
quadratic function and its limits, but also gives us a glimpse
of its derivative. And believe us, this is just the tip of the
iceberg of how fun and surprising Matplotlib can be.

import numpy as np
import matplotlib.pyplot as plt

# Creating the array x

x = np.linspace(-10, 10, 1000)

# Function f(x) = x^2

y = x**2

# Limits
x_lim = 2
y_lim = x_lim**2

# Derivative
dy_dx = 2*x

# Plotting the Graph

fig, ax = plt.subplots()

ax.plot(x, y, label='f(x) = x^2')

ax.axvline(x=x_lim, color='red', linestyle='--', label=f'x = {x_lim}')
ax.axhline(y=y_lim, color='green', linestyle='--', label=f'y = {y_lim}')
ax.plot(x, dy_dx, label='f\'(x) = 2x')

ax.set_xlabel('x')
ax.set_ylabel('y')
ax.set_title('Quadratic Function')
ax.legend()

plt.show()
Seaborn and Plotly for Advanced
Visualization
Ah, we've finally arrived at the fun part of data science! In
this chapter, we'll talk about the Seaborn and Plotly
libraries, which will take your graphics to the next level. It's
time to put our creativity to work and build amazing
visualizations!
Seaborn is a data visualization library based on Matplotlib,
but with much more style. If you want to create elegant and
attractive graphics, this is the right library for you. In
addition, Seaborn also supports descriptive statistics,
regression analysis, and categorical data visualization.
Plotly, on the other hand, is an interactive visualization
library that allows for the creation of animated and
interactive charts. With Plotly, it's possible to create 3D
charts, bubble charts, heatmaps, line charts, among others.
In addition, you can create interactive visualizations that
allow users to explore the data on their own.
Let's start with Seaborn. An example of a graph we can
create with Seaborn is a scatter plot with a linear regression.
This graph is useful for showing the relationship between
two variables and the overall trend of the data. We can
create a scatter plot with a linear regression using Seaborn's
lmplot() function:

sns.lmplot(x='variavel_x', y='variavel_y', data=df)

This function creates a scatter plot with a linear regression

line for the data. We can customize the plot by adding titles,
axis labels, colors, and line styles.
With Plotly, we can create incredibly interactive graphs. One
example is the heatmap graph. This graph is useful for
showing the distribution of a variable in different categories.
We can create a heatmap graph with Plotly using the
function heatmap():

import seaborn as sns

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

# Generating a random correlation matrix

np.random.seed(0)
corr = np.corrcoef(np.random.randn(10, 200))

# Converting the matrix to a DataFrame

df = pd.DataFrame(corr)

# Creating the heatmap using Seaborn

sns.heatmap(df, cmap='coolwarm', annot=True)

plt.title('Random Correlation Matrix')

plt.show()

In this example, we created a random correlation matrix

with 10 variables and 200 observations and turned it into a
Pandas DataFrame. Then, we created a heatmap using the
Seaborn function heatmap(), specifying the coolwarm color
scheme and the annot=True option to display the matrix
values on the heatmap.
The result is a colorful heatmap that visualizes the
correlation matrix, allowing for a quick analysis of the
relationships between variables.
With Seaborn and Plotly, the possibilities are endless.
Combining these libraries with the data cleaning,
manipulation, and analysis techniques we learned in
previous chapters, we can create amazing visualizations
that help us understand and communicate our data
effectively and in a fun way.
Let's move on to another cool example that showcases the
use of Seaborn combined with Plotly. Suppose you have the
results of a game and want to visualize the number of wins
and losses for each team. We can create a stacked bar chart
with Seaborn and Plotly to show these results.

import seaborn as sns

import plotly.graph_objs as go
import pandas as pd

# Creating a dictionary with the results

results = {'Team 1': {'Wins': 10, 'Losses': 5},
'Team 2': {'Wins': 8, 'Losses': 7},
'Team 3': {'Wins': 12, 'Losses': 3}}

# Converting the dictionary to a Pandas DataFrame

df = pd.DataFrame.from_dict(results, orient='index')

# Creating a stacked bar chart with Seaborn

sns.set(style="whitegrid")
ax = sns.barplot(x=df.index, y='Wins', data=df, color='#2ecc71')
ax = sns.barplot(x=df.index, y='Losses', data=df, color='#e74c3c')
ax.set_title('Game results')
ax.set_xlabel('Team')
ax.set_ylabel('Number of games')

# Creating a stacked bar chart with Plotly

fig = go.Figure(data=[
go.Bar(name='Wins', x=df.index, y=df['Wins'], marker_color='#2ecc71'),
go.Bar(name='Losses', x=df.index, y=df['Losses'], marker_color='#e74c3c')
])
fig.update_layout(barmode='stack', title='Game Results', xaxis_title='Team',
yaxis_title='Number of games')
fig.show()

In this example, we used a dictionary to store the results

and turned it into a Pandas DataFrame. Then, we used
Seaborn to create a stacked bar chart and Plotly to create
another stacked bar chart with the same information. Using
Plotly, we can interact with the chart and see the exact
number of wins and losses for each team.
Machine Learning with Python
If you have ever watched the movie "The Terminator", you
know that machines are unstoppable. But what if I told you
that there are machines that can help predict future events?
And even better, they are so easy to use that even your
grandma could use them.
I'm talking about Machine Learning machines. And when it
comes to Machine Learning in Python, one of the most
powerful and easy-to-use libraries is Seaborn.
With Seaborn, we can create elegant and informative
graphics that help us visualize patterns and trends in data.
And when we combine Seaborn with Machine Learning
algorithms, we can create models that can predict future
events with impressive accuracy.
For example, imagine you're a store manager and want to
predict future sales based on sales history. Using Seaborn
and Machine Learning algorithms, you can create a model
that predicts sales with very high accuracy. And so, you can
make informed decisions about how to manage your
inventory and increase your sales.
But don't worry, you don't have to be a math genius or a
programming expert to use Seaborn and Machine Learning
in Python. With a little practice, anyone can master these
powerful tools and turn data into valuable insights.
So get ready to enter the exciting world of Machine Learning
with Seaborn and Python. And remember, machines can be
unstoppable, but they still need humans to guide them in
the right direction.
In the example below we're going to play fortune teller!
We're creating a little game with a super simple data set
that includes some cards and their respective values. Then,
we use the "LinearLearner" from the "LearningKit" (aka
Scikit-Learn) to train a mind reading model with this data.
Finally, we make a prediction for a new card and print the
prediction on the screen. But be careful, okay? If you try to
use this power for evil purposes, you'll end up becoming a
villain in a superhero movie!
import numpy as np
from sklearn.linear_model import LinearRegression

# Creating the dataset

X = np.array([1, 2, 3, 4, 5]).reshape(-1, 1)
y = np.array([2, 4, 6, 8, 10])

# Training the linear regression model

reg = LinearRegression().fit(X, y)

# Making a prediction for a new value of X

new_X = np.array([6]).reshape(-1, 1)
prediction = reg.predict(new_X)

# Printing the prediction

print(prediction)

And the predicted result by the machine will be: [12.]

Let's go for another cool example. Suppose you want to

create a machine learning model to predict whether a day
will be sunny or cloudy based on the temperature and
relative humidity. You can create a simple dataset, with
invented values for the independent variables (temperature
and humidity) and the dependent variable (sunny/cloudy).
# Creating the dataset

temperature = [25, 26, 27, 28, 29, 30, 31, 32, 33, 34]
humidity = [50, 55, 60, 65, 70, 75, 80, 85, 90, 95]
sunny_cloudy = ['sunny', 'sunny', 'cloudy', 'cloudy', 'cloudy', 'sunny', 'sunny',
'cloudy', 'cloudy', 'cloudy']

data = {'Temperature': temperature, 'Humidity': humidity, 'Sunny/Cloudy':

sunny_cloudy}
# Converting to a Pandas DataFrame

import pandas as pd

df = pd.DataFrame(data)

# Splitting the dataset into training and testing sets

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(df[['Temperature', 'Humidity']],

df['Sunny/Cloudy'], test_size=0.2)

# Training the model

from sklearn.tree import DecisionTreeClassifier

model = DecisionTreeClassifier()

model.fit(X_train, y_train)

# Making predictions

predictions = model.predict(X_test)

# Evaluating the model

from sklearn.metrics import accuracy_score

print('Accuracy: ', accuracy_score(y_test, predictions))

Ah, the wonderful world of machine learning! In this

example, we're using a decision tree model to solve a
mystery: will the day be sunny or cloudy based on the
temperature and relative humidity? We create a made-up
dataset with this information and split it into train and test
sets.
Next, we put our model to work! The DecisionTreeClassifier
model from Scikit-Learn is our private detective and starts
analyzing the data to help us solve this mystery. With its
sharp decision tree skills, it makes predictions for the test
set.
But the detective's job is not done yet. We need to evaluate
its efficiency! We use accuracy, which is the proportion of
correct predictions to the total number of predictions, to
evaluate our model's performance. And voilà, we have the
answer to our mystery.
Although this example is simple, it fun and lightheartedly
illustrates the basic process of building a machine learning
model using Python. Now, how about putting on your
detective hat and start building your own models?
Regression Analysis
Ah, regression analysis, the darling of all data scientists who
love to predict the future! In this chapter, we will dive into
the fascinating world of regression analysis with Python, but
first, put on your fortune teller outfit, grab your crystal ball
and get ready to see what the future holds!
Let's start with a simple example of linear regression
analysis. Let's say you have a dataset with the height and
weight of some people, and you want to predict weight
based on height. Using the Pandas library, we can load the
data into a DataFrame and then plot a scatter plot to see if
there is any apparent relationship between the two
variables. Then, we use the Statsmodels library to fit a
linear regression model and finally use the trained model to
make predictions on new data.
But regression analysis doesn't stop there. There are many
other types of regression analysis, such as polynomial
regression, logistic regression, and Ridge regression. Each
type of regression is suitable for different types of problems
and requires different modeling techniques. For example,
polynomial regression is useful when there is a nonlinear
relationship between variables, while logistic regression is
suitable for classification problems.
But don't worry, you don't need to be a fortune teller to
understand regression analysis. With Python and the
Pandas, Statsmodels, and Scikit-Learn libraries, we can
create accurate and useful models to predict the future. So,
put on your fortune teller outfit, grab your crystal ball, and
let's explore the amazing world of regression analysis!
To create a simple example of linear regression without
importing data, we will create a set of fictitious values for
the independent and dependent variables and use them to
fit a linear regression model.
First, let's import the necessary libraries:
import pandas as pd
import statsmodels.api as sm
from sklearn.linear_model import LinearRegression
import matplotlib.pyplot as plt

Now, let's create a fictional dataset with two independent

variables, x1 and x2, and one dependent variable, y:
x1 = [1, 2, 3, 4, 5]
x2 = [4, 6, 9, 18, 20]
y = [6, 6, 7, 12, 15]

We can then combine these variables into a Pandas

DataFrame:
data = pd.DataFrame({'x1': x1, 'x2': x2, 'y': y})

Next, we will fit a linear regression model using the

Statsmodels library:
X = sm.add_constant(data[['x1', 'x2']])
model = sm.OLS(data['y'], X).fit()

We can view the summary of the fitted model with the

following code:
print(model.summary())

To make predictions with the model, we can use the Scikit-

Learn library:
model_sk = LinearRegression().fit(X, data['y'])
y_pred = model_sk.predict(X)

Finally, we can create a plot to visualize the original data

and the model predictions:
plt.scatter(x1, y, label='y')
plt.scatter(x1, y_pred, label='y_pred')
plt.legend()
plt.show()

The resulting plot will show the points corresponding to the

original data and the predictions made by the linear
regression model..
Classification Analysis
Have you ever felt like a detective trying to figure out
whether a person is guilty or innocent? Well, that's basically
the idea behind classification analysis. The goal is to
determine which category an observation belongs to, like a
suspect in a crime being categorized as guilty or innocent.
There are many classification algorithms that can be used to
solve this type of problem. Some of them are so good at
classifying things that they could be used as jurors in a
court (although we're not sure if they're good at following
jury rules!).
A common example of classification is spam detection in
emails. The classification algorithm is trained to identify the
most common features of spam and then classify new
emails as spam or not spam based on those features.
To implement classification analysis in Python, we use
libraries like Scikit-Learn. With Scikit-Learn, we can create
classification models, train them on training datasets, and
test them on test datasets to evaluate their accuracy.
For example, we can create a classification model to predict
whether a person has diabetes based on features such as
age, BMI, and blood glucose level. We can train the model
on a training dataset and test it on a test dataset to see how
well it performs.
Some of the most common classification algorithms include
Naive Bayes, decision trees, and logistic regression. Each
algorithm has its own advantages and disadvantages, so it's
important to try out different algorithms to find the best one
for your dataset.
But, as always, don't forget that classification analysis is not
perfect and there is always the possibility of errors. After all,
even the best detective can make mistakes and wrongly
classify a person as guilty or innocent.
In this example, we will create a classification model to
predict whether a person has diabetes or not based on three
features: age, BMI, and blood glucose level. Remember, this
is an analysis for educational purposes with very few input
variables and data points, so don't use this example to
diagnose people, okay?

from sklearn.naive_bayes import GaussianNB

# Creating an example dataset

age = [25, 45, 28, 50, 32, 23, 28, 40, 35, 30]
bmi = [18.5, 25.3, 23.1, 27.8, 21.9, 20.1, 22.5, 26.7, 28.3, 24.7]
glucose = [70, 110, 85, 120, 100, 75, 90, 115, 130, 95]
diabetes = ['N', 'P', 'N', 'P', 'N', 'N', 'N', 'P', 'P', 'N']

# Creating a dataframe with the data

df = pd.DataFrame({'Age': age, 'BMI': bmi, 'Glucose': glucose, 'Diabetes':
diabetes})

# Separating the data into independent and dependent variables

X = df[['Age', 'BMI', 'Glucose']]
y = df['Diabetes']

# Splitting the dataset into training and testing sets

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3,
random_state=42)

# Training a Naive Bayes model with the training data

clf = GaussianNB()
clf.fit(X_train, y_train)

# Making predictions on the testing set

y_pred = clf.predict(X_test)

# Evaluating the model's performance

acc = accuracy_score(y_test, y_pred)
print('Accuracy: {:.2f}'.format(acc))

For example, suppose you want to test a patient named

William who is 35 years old, has a BMI of 27.5, and a
glucose level of 120. You can create a dataset for this
patient as follows:

new_patient = pd.DataFrame({'Age': [35], 'BMI': [27.5], 'Glucose': [120]})

Then, you can use the predict() method of the trained model
to make a prediction for the new patient:
prediction = clf.predict(new_patient)
print(prediction)

Applying our model to the fictional patient above, our

program returned "Positive", so it predicts that William
needs to quit soda and coconut cake and run to a doctor.
Accuracy: 1.00
['P']

In this example, we decided to create a random dataset with

values that are so made up that they seem to have come
out of the mind of a surrealist artist. Then, we turned this
mess into a tidy DataFrame with the help of Pandas,
because organization is everything, right?
With that in hand, we split our dataset into training and test
sets using the train_test_split() function from Scikit-Learn.
After all, we don't want to be caught off guard by
unexpected data that we don't know how to handle, do we?
Now it's time to put the Naive Bayes model from Scikit-
Learn to work and train it with the training data. Then, we
put the cherry on the cake (except for William) and make
predictions for the test set. And we couldn't forget to check
the model's performance using accuracy to see if we got
more right than wrong.
Remember, classification is like choosing the perfect match:
you need to know what you're doing and choose the right
model, as well as have quality data to get the best results.
But if all else fails, you can always turn to Tinder.
Clustering Analysis
Clustering Analysis is a data analysis technique that groups
objects based on their common characteristics. It's like
making friends with people who have the same interests as
you, but in the world of data.
There are different types of clustering algorithms, such as K-
Means, DBSCAN, and Hierarchical Clustering. Each of them
has its own advantages and disadvantages, just like the
people you meet in life.
With the help of Python and libraries like Scikit-Learn and
Pandas, it is possible to perform clustering analysis on data
sets of various sizes and complexities.
For example, imagine that you have a data set with
information about different types of flowers, such as the size
of petals and sepals, and you would like to group them into
different species. Using the K-Means algorithm, you can
group the flowers based on their common characteristics
and discover which species are more like each other. It's like
organizing a botanical garden.
Clustering Analysis is also useful for grouping customers
based on their buying behaviors, identifying usage patterns
in social media data, or even grouping different types of fish
based on their physical characteristics. It's like creating an
underwater zoo of data.
But just like in real life, choosing the right clustering
algorithm depends heavily on the data set and the analysis
objective. It's like choosing the right type of friends to go on
an adventure with you.
Therefore, it is important to know the different clustering
techniques and test them on your own data sets to discover
which one is best for your needs.
Let's say you're trying to discover different groups of people
in your city based on their interests. For example, you want
to know if there are groups of people who like to watch
action movies, groups who like romance books, or groups
who love extreme sports. For this, you can use the
clustering analysis technique.
Let's look at an example Python code using the Pandas,
Scikit-Learn, and Matplotlib libraries:

import pandas as pd
from sklearn.cluster import KMeans
import matplotlib.pyplot as plt

# Creating a fictional dataset based on people's interests

action_movies = [3, 2, 5, 8, 7, 6, 9, 10, 8, 7, 6, 4]
romance_books = [8, 10, 9, 7, 3, 2, 1, 3, 4, 5, 6, 7]
extreme_sports = [9, 8, 10, 7, 6, 3, 2, 1, 5, 4, 5, 6]

# Creating a dataframe with the data

df = pd.DataFrame({'Action Movies': action_movies, 'Romance Books':
romance_books, 'Extreme Sports': extreme_sports})

# Training the clustering model with 3 clusters

kmeans = KMeans(n_clusters=3, random_state=42).fit(df)

# Adding a column to the dataframe with the cluster classification

df['Cluster'] = kmeans.labels_

# Plotting the result in a scatter plot

plt.scatter(df['Action Movies'], df['Romance Books'], c=df['Cluster'])
plt.xlabel('Action Movies')
plt.ylabel('Romance Books')
plt.title('People Interests')
plt.show()
In this example, we created a fictitious data set based on
people's interests in action movies, romance books, and
extreme sports. Then, we trained a clustering model with 3
clusters using the K-means algorithm from Scikit-Learn.
Next, we added a column to the dataframe with the cluster
classification and plotted the result in a scatter plot.
This is just a simple example of how to use clustering
analysis in Python. You can apply this technique to various
types of problems, from customer segmentation to species
grouping in ecology. The important thing is to choose the
correct algorithm and adjust the parameters to obtain the
best results.
Let's move on to another example. Suppose you have a list
of songs that you want to group based on their
characteristics, such as duration, popularity, and
"danceability". For this, we can create a simple data set with
this fictitious information:
# Creating a fictional dataset
duration = [3.5, 4.2, 2.9, 3.1, 4.8, 2.6, 3.9, 2.8, 4.1, 3.7, 4.6, 3.0]
popularity = [7, 8, 5, 6, 9, 4, 7, 3, 8, 6, 9, 5]
danceability = [8, 9, 5, 6, 7, 4, 9, 3, 7, 6, 8, 5]

# Transforming the data into a Pandas DataFrame

df = pd.DataFrame({'Duration': duration, 'Popularity': popularity, 'Danceability':
danceability})

Next, we can normalize these features to ensure that they

are all on the same scale:
# Normalizing the features of the data
scaler = StandardScaler()
df_norm = pd.DataFrame(scaler.fit_transform(df), columns=df.columns)

Now that we have our normalized data, we can use the K-

means algorithm from Scikit-Learn to group them into
clusters. Let's suppose we want to group our songs into 3
clusters:
# Creating a K-means model and fitting to the data
kmeans = KMeans(n_clusters=3, random_state=42)
kmeans.fit(df_norm)

With the trained model, we can predict the clusters for each
song in our dataset:
# Predicting the clusters for each song
clusters = kmeans.predict(df_norm)

Finally, we can plot a scatter plot to visualize how our songs

grouped together in each cluster:
# Plotting a scatter plot of the clusters
plt.scatter(df_norm['Duration'], df_norm['Popularity'], c=clusters)
plt.xlabel('Duration (normalized)')
plt.ylabel('Popularity (normalized)')
plt.show()

And there you have it! Now we have our songs grouped into
clusters based on their characteristics, and we can analyze
these clusters to uncover interesting patterns.
Time Series Analysis
Ah, time series analysis! Where would we be, as humans,
without the ability to look back in time and predict the
future? With Python, we can use incredible tools to
understand and predict trends over time, from stock prices
to weather.
But before we dive into time series analysis, let's take a look
at time itself. Time is a funny thing, isn't it? It goes by
quickly when we're having fun and slowly when we're
waiting for something. Sometimes, it seems like time just
disappears!
But thankfully, with time series analysis, we can capture
time and understand its patterns. We can look at the past
and identify trends, seasonality, and irregular patterns that
help us predict the future.
Let's take a look at some practical examples. Let's say
you're a store manager and want to predict the number of
sales in the next six months. With time series analysis, we
can identify historical trends in sales, as well as seasonality
such as higher sales on holidays or weekends. We can then
use this information to make accurate predictions and help
your company plan better.
But time series analysis isn't just for business. We can also
use it to predict the weather by analyzing historical data on
temperature, humidity, and precipitation. And who knows,
maybe even predict the end of the world by looking at the
past and finding apocalyptic patterns.
Now, let's talk about how we can do time series analysis
with Python. One of the most popular libraries is Pandas,
which has already been covered in the book. But in addition
to that, we can use other amazing libraries like NumPy,
Matplotlib, and Statsmodels.
Let's take a look at a practical example. Let's say you have
historical data on Tesla's stock price for the past three
years. We can use the Statsmodels library to do a time
series analysis of the stock price and predict its future price.
With Statsmodels, we can use models like ARIMA
(AutoRegressive Integrated Moving Average), which is a
powerful technique for making accurate time series
predictions. We can then visualize our predictions using
Matplotlib and share our insights with our investor friends.
In short, time series analysis is a powerful and fun tool for
understanding and predicting the world around us. With
Python, we can make accurate predictions about sales,
weather, stock prices, and even the apocalypse. So let's
capture time and make accurate predictions with time series
analysis!
In the next example with fictitious data, we created a made-
up time series for the stock price of a company over the
past 15 months. We used the powerful ARIMA model (not
the little animal, but Autoregressive Integrated Moving
Average) to make predictions for the next 6 months.
The ARIMA model is like a magician, but instead of a rabbit
hat, it uses order parameters (p,d,q) to predict the future. In
our example, we used an order of (1,1,1), which means
we're taking a first difference of the data (d=1), we have an
autoregressive model of order 1 (p=1), and a moving
average model of order 1 (q=1).
After training the model, we used the forecast() method to
make predictions for the next 6 months. Finally, we
combined the original time series with the predictions to see
how they compared.
But it's important to remember that, in real life, we need to
use real data to train the model and make accurate
predictions. So, no fiction over there!
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from statsmodels.tsa.arima.model import ARIMA

# Creating fictitious data for a stock price

stock_price = [10, 12, 15, 18, 20, 22, 25, 28, 30, 35, 40, 45, 50, 55, 60]

# Transforming the data into a Pandas time series

dates = pd.date_range('2022-01-01', periods=len(stock_price), freq='M')
prices = pd.Series(stock_price, index=dates)

# Plotting the time series

plt.plot(prices)
plt.title('Fictitious Stock Price')
plt.xlabel('Date')
plt.ylabel('Price')
plt.show()

# Training the ARIMA model with the data

model = ARIMA(prices, order=(1,1,1))
result = modelo.fit()

# Making predictions for the next 6 months

next_months = 6
next_dates = pd.date_range('2022-07-01', periods=next_months, freq='M')
next_predictions = result.forecast(steps=next_months)

# Plotting the predictions

plt.plot(prices)
plt.plot(next_predictions, label='Predictions')
plt.title('Fictitious Stock Price')
plt.xlabel('Date')
plt.ylabel('Price')
plt.legend()
plt.show()
Natural Language Processing (NLP)
Now, let's explore a very interesting and fascinating field:
Natural Language Processing, or NLP for short. As humans,
we are endowed with a great power of communication. And
what happens when we want to train our machines to have
a similar power of communication to ours? That's where NLP
comes in.
Let's start with something simple: counting words. But don't
think that this task is easy. Words are slippery, they change
meaning according to context, and sometimes have
completely different meanings depending on how they are
used. But fortunately, we have some powerful tools at our
disposal: Python and its incredible libraries.
Let's start by creating a small sentence with a few words. "
When in Rome, do as the Romans do." Now, let's use Python
and its NLTK (Natural Language Toolkit) library to count the
number of occurrences of each word in the sentence. And of
course, let's present the results in an elegant bar chart.
Because we are not only data scientists, we are also artists!
With the help of Python code and NLTK, we will create an
interesting and fun visualization of the data. Who knows, we
might even discover which words are most used by our
rodent friends. Let's go, NLPers!
But don't think that NLP is limited to counting words. No, no,
no! Natural language processing is a vast and multifaceted
field. We can use NLP to do sentiment analysis in texts,
automatic translation, speech recognition, and much more.
It's a field full of challenges and possibilities.
So, if you're interested in helping machines communicate
better with humans, NLP is the field for you. And of course,
Python is the perfect tool to master this incredible field.
First, let's start with text tokenization. Tokenization is the
process of dividing a text into smaller words or phrases,
which are called tokens. We can use the NLTK library to
perform this process. Here's an example of code:
import nltk

nltk.download('punkt') # Download the tokenizer

text = " When in Rome, do as the Romans do."

tokens = nltk.word_tokenize(text)

print(tokens)

This code will split the string " When in Rome, do as the
Romans do." into a list of words:
['When', 'in', 'Rome', ',', 'do', 'as', 'the', 'Romans', 'do', '.']

Now, let's move on to the next step in natural language

processing, which is removing stop words. Stop words are
very common words in language, such as "the", "a", "of",
etc., that usually do not add meaning to the text. We can
easily remove them using the NLTK library. Here's an
example of code:

import nltk

nltk.download('stopwords') # Download the stopwords

from nltk.corpus import stopwords

stop_words = set(stopwords.words('english'))

text = "When in Rome, do as the Romans do."

tokens = nltk.word_tokenize(text)

no_stopwords = [word for word in tokens if not lower.word() in stop_words]

print(no_stopwords)
This code will remove the stop words from the list of tokens,
resulting in a new list without the unnecessary words:
['Rome', ',', 'Romans', '.']

Finally, let's talk about the step of lemmatization.

Lemmatization is the process of reducing a word to its basic
form, called a lemma. For example, the word "running"
would be reduced to its lemma "run". We can use the NLTK
library again to perform this process. Here's an example of
code:

import nltk

nltk.download('wordnet') # WordNet download

from nltk.stem import WordNetLemmatizer

lemmatizer = WordNetLemmatizer()

text = "When in Rome, do as the Romans do."

tokens = nltk.word_tokenize(texto)

lemmas = [lemmatizer.lemmatize(palavra) for palavra in tokens]

print(lemmas)

This code will reduce the words to their corresponding

lemmas:
['When', 'in', 'Rome', ',', 'do', 'a', 'the', 'Romans', 'do', '.']

Note that some words have been lemmatized to their base

form, such as "do" and "be", while others, such as "a" and
"the", have not, as they are not inflected forms of a base
word. Additionally, the lemmatization process requires part-
of-speech information to disambiguate between different
possible lemmas for a given word, but this example does
not include that step.
Text Classification and Sentiment Analysis
Hey there, buddy, or pal, ready to unravel more mysteries of
NLP? Well, in addition to the incredible natural language
processing techniques like tokenization, stemming, and
lemmatization, there are other sensational areas of NLP that
can be of great use in the real world. And here we are going
to talk about two of these areas: text classification and
sentiment analysis.
Text classification aims to categorize a text into one or more
pre-defined classes. It's like that friend of yours who can tell
exactly what movie you're describing even before you finish
telling the synopsis. To do this, we can use machine learning
algorithms such as Naive Bayes, SVM, and Random Forest.
In sentiment analysis, the goal is to determine the polarity
of a text, that is, whether it is positive, negative, or neutral.
It's like the machine can sense the emotions of the text,
something very similar to what your mom does when she
reads your text messages. To do this, we can use rule-based
approaches or machine learning algorithms such as logistic
regression and neural networks.
But let's get to the point, because nothing is better than an
example to clarify the mind, right? So, let's look at an
amazing example of text classification using the Naive
Bayes algorithm. In this example, we'll use the IMDB movie
review dataset, which is commonly used for text
classification tasks. And get ready because it's going to be a
hair-raising example!
import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.metrics import accuracy_score

# Reading the dataset

df = pd.read_csv('IMDB_reviews.csv')
# Splitting the data into training and testing sets
train_size = int(0.8 * df.shape[0])
X_train = df['review'][:train_size]
y_train = df['sentiment'][:train_size]
X_test = df['review'][train_size:]
y_test = df['sentiment'][train_size:]

# Vectorizing the text data

cv = CountVectorizer(stop_words='english')
X_train_cv = cv.fit_transform(X_train)
X_test_cv = cv.transform(X_test)

# Training a Naive Bayes model

nb = MultinomialNB()
nb.fit(X_train_cv, y_train)

# Making predictions on the test set

y_pred = nb.predict(X_test_cv)

# Evaluating the model's performance

acc = accuracy_score(y_test, y_pred)
print('Accuracy: {:.2f}'.format(acc))

For sentiment analysis, we can use similar approaches, but

with labeled datasets with polarity (positive, negative, or
neutral). Let's take a look at an example code for sentiment
analysis using the movie review dataset as well:
import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

# Reading the dataset

df = pd.read_csv('IMDB_reviews.csv')

# Preparing the data

df['sentiment'] = df['sentiment'].map({'positive': 1, 'negative': 0})
X = df['review']
y = df['sentiment']
# Vectorizing the text data
cv = CountVectorizer(stop_words='english')
X_cv = cv.fit_transform(X)

# Splitting the data into training and test sets

train_size = int(0.8 * df.shape[0])
X_train_cv = X_cv[:train_size]
y_train = y[:train_size]
X

This code is a lesson on how to train a machine to be a

better movie critic than Roger Ebert. Basically, it reads a
dataset of movie reviews from IMDB, splits the reviews into
"positive" and "negative" and teaches a machine to identify
whether a review is good or bad. Then, it puts that machine
to work, reading a set of movie reviews it has never seen
before, and making predictions about whether they are
positive or negative. In the end, the program even evaluates
the machine's performance, giving it an "A" grade for
accuracy. Now, if we could only teach the machine how to
make popcorn, we would be ready for a perfect day at the
movies!
Deep Learning with TensorFlow
Have you ever heard of artificial intelligence? It's where
machines learn like humans! And how do they learn?
Through Deep Learning! And how do you do Deep Learning?
With TensorFlow, of course!
But before diving into programming, let's first understand
what TensorFlow is. Imagine a big bag full of nodes and
wires that represent the mathematical operations machines
need to perform to learn. Now imagine that bag being
thrown into a boxing ring, fighting against huge amounts of
data and trying to find patterns. That boxing ring is
TensorFlow!
Now imagine that you are a boxing coach and you need to
teach your machine to fight. You need to tell it what to do at
each step, how to adjust its weights, and what to do if it
makes a mistake. This is what we call "training".
But don't worry, TensorFlow has a team of engineers
working hard to make this process as easy as possible.
They've created a library with a wide variety of pre-built
layers for you to choose from, such as dense layer, pooling
layer, convolutional layer, and more!
And now comes the really fun part: examples! Have you
heard of image recognition? Imagine that you want to teach
your machine to identify cats in photos. First, you need to
collect thousands of photos of cats and not-cats, and label
them as such. Then, you can build a neural network with
convolutional and pooling layers, and train it with those
photos. After a few hours of training, your machine will be
ready to identify cats like a pro!
But that's just the beginning. You can also use TensorFlow
for things like natural language processing, sentiment
analysis, time series prediction, and much more. Artificial
intelligence is the future, and TensorFlow is the key to
unlocking it.
In summary, TensorFlow is your passport to the exciting
journey of artificial intelligence. Plus, there are many
resources available, such as tutorials, documentation, and
ready-to-use examples, to help you overcome any obstacle.
And don't forget to have fun! TensorFlow is incredibly
powerful, but it's also incredibly fun to use. Try different
things, play with the data, and see what your machine is
capable of. Who knows, you might even discover some
surprising things about artificial intelligence along the way.
So, grab your pen and paper, and get ready to enter the
world of Deep Learning with TensorFlow. The adventure is
just beginning!

TensorFlow
Sure, you want to know how TensorFlow works, right? Well,
it's a bit like magic, but with a dash of science and
technology. Imagine you have a bag full of data and you
want to do something useful with it. With TensorFlow, you
can build a neural network, which is basically a group of
virtual neurons that work together to learn from the data.
But what makes TensorFlow so amazing is its ability to
adjust the weights of each neuron over time based on what
it learns. It's like each neuron is a little sponge soaking up
knowledge, and TensorFlow helps regulate that process to
ensure that your neural network is the best it can be.
And if you think that's amazing, wait until you see what your
neural network is capable of! It can predict the future,
classify images, translate languages, and even play games.
All thanks to TensorFlow's incredible ability to learn and
adapt to new data.
But don't worry if all of this sounds a little complicated.
TensorFlow is incredibly easy to use, and there are many
resources available to help you understand how it all works.
Well, dear readers, let's liven up the example below of
image classification with TensorFlow!
First, let's create some training data, but don't worry, it's
just for fun. We generated random images with numpy, but
you can imagine they are your favorite photos from
Instagram.
Then, we built a crazy neural network! We have a
convolutional layer, to give the images a "massage," a
pooling layer to pick out the best features, a flattening layer
to flatten them like pancakes, and a fully connected layer
with softmax activation, to make them an even tastier
dessert.
Next, we compiled the model with the Adam optimizer,
which is as smart as a genie in a lamp, and the sparse
categorical cross-entropy loss function, which is as cruel as
a wrestling match.
But don't worry, we trained our model using the fit method,
as if it were being prepared for the craziest obstacle race of
all. Finally, we evaluated the model using the evaluate
method to make sure it's ready to take on the world!
Remember, this is just an example and the results may vary
due to the randomly generated data, but we're sure it will
be funny, at least.

import tensorflow as tf
import numpy as np

# Generating simulated training data

images = np.random.rand(100, 32, 32, 3).astype(np.float32)
labels = np.random.randint(0, 10, 100).astype(np.int32)

# Define the model

model = tf.keras.Sequential()
model.add(tf.keras.layers.Conv2D(32, 3, activation='relu', input_shape=(32, 32,
3)))
model.add(tf.keras.layers.MaxPooling2D(2))
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(10, activation='softmax'))

# Compile the model

model.compile(optimizer='adam',
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=Tru
e),
metrics=['accuracy'])

# Train the model

history = model.fit(images, labels, epochs=10)

# Evaluate the model

test_loss, test_acc = model.evaluate(images, labels, verbose=2)
print('Test accuracy:', test_acc)

Let's go for another interesting example of TensorFlow

application. Imagine you are the owner of a pizzeria and
want to create an app to identify if the pizza you just made
is perfect or not. With TensorFlow, you can train a neural
network to classify whether a pizza is good or bad.
Let's say you have a thousand photos of perfect pizzas and
a thousand photos of bad pizzas. You will use these images
to train the neural network to recognize the difference
between a good pizza and a bad pizza.
The neural network will have several layers, each
responsible for analyzing a feature of the pizza. The first
layer can identify the color of the pizza, the second layer
can identify if the crust is crispy, and so on. The last layer,
called the output layer, will classify the pizza as good or
bad. When the neural network has been trained, you can
use it to analyze a new pizza image, and it will classify it as
good or bad based on the characteristics identified in the
previous layers.
Isn't it amazing? Now you have a perfect pizza app! And the
best part is that you don't even need to be a Deep Learning
expert to create it. TensorFlow takes care of everything for
you.
And you can apply this same concept to other things as
well, such as classifying cats and dogs, speech recognition,
sales forecasting, among other things. The sky's the limit
with TensorFlow!