Data Mining and Predictive

Modeling
Btech-Tech VIth Semester
Contents: Unit 2 Lecture 1
 Data Objects and Attributes Types
 Basic Statistical Description of Data
 Data Visualization
 Measuring of Data Similarity and Dissimilarity
 Summary

Prof. Moumita Pal 2

 Types of Data Sets

timeout

season
coach

game
score
pla y
team

wi n
ball

lost
 Record m-
 Relational records
 Data matrix, e.g., numerical matrix, crosstabs
Document 1 3 0 5 0 2 6 0 2 0 2
 Document data: text documents: term-frequency vector
 Transaction data Document 2 0 7 0 2 1 0 0 3 0 0
 Graph and network
Document 3 0 1 0 0 1 2 2 0 3 0
 World Wide Web
 Social or information networks
 Molecular Structures
 Ordered
 Video data: sequence of images
 Temporal data: time-series TID Items
Sequential Data: transaction sequences
1 Bread, Coke, Milk


 Genetic sequence data

 Spatial, image and multimedia: 2 Beer, Bread
 Spatial data: maps 3 Beer, Coke, Diaper, Milk
 Image data
4 Beer, Bread, Diaper, Milk
 Video data
5 Coke, Diaper, Milk
Prof. Moumita Pal 3
 Data Objects
 Data sets are made up of data objects.
 A data object represents an entity.
 Examples:
 sales database: customers, store items, sales
 medical database: patients, treatments
 university database: students, professors, courses
Attributes
 Also called samples , examples, instances, data points, objects, tuples.
 Data objects are described by attributes.
 A
Database rows -> data objects; columns ->attributes. B C D E F

Data Object

Prof. Moumita Pal 4

 Attributes

 Attribute (or dimensions, features, variables): a data field,

representing a characteristic or feature of a data object.
 E.g., customer _ID, name, address

 Types:
 Nominal

 Binary

 Numeric: quantitative

 Interval-scaled

 Ratio-scaled

Prof. Moumita Pal 5

 Attribute Types
 Nominal: categories, states, or “names of things”
 Hair_color = {auburn, black, blond, brown, grey, red, white}
 marital status, occupation, ID numbers, zip codes

 Binary
 Nominal attribute with only 2 states (0 and 1)
 Symmetric binary: both outcomes equally important
 e.g., gender
 Asymmetric binary: outcomes not equally important.
 e.g., medical test (positive vs. negative)
 Convention: assign 1 to most important outcome (e.g., HIV positive)
 Ordinal
 Values have a meaningful order (ranking) but magnitude between
successive values is not known.
 Size = {small, medium, large}, grades, army rankings
Prof. Moumita Pal 6
 Numeric Attribute Types
 Quantity (integer or real-valued)
 Interval
 Measured on a scale of equal-sized units
 Values have order
 E.g., temperature in C˚or F˚, calendar dates
 No true zero-point
 Ratio
 Inherent zero-point
 We can speak of values as being an order of magnitude larger
than the unit of measurement (10 K˚ is twice as high as 5 K˚).
 e.g., temperature in Kelvin, length, counts,
monetary quantities
Prof. Moumita Pal 7
 Discrete vs. Continuous Attributes
 Discrete Attribute
 Has only a finite or countably infinite set of values
 E.g., zip codes, profession, or the set of words in a collection

of documents
 Sometimes, represented as integer variables
 Note: Binary attributes are a special case of discrete attributes

 Continuous Attribute
 Has real numbers as attribute values
 E.g., temperature, height, or weight

 Practically, real values can only be measured and represented

using a finite number of digits
 Continuous attributes are typically represented as floating- point
variables
Prof. Moumita Pal 8
 Contents: Unit 2 Lecture 1

 Data Objects and Attributes Types

 Basic Statistical Description of Data
 Data Visualization
 Measuring of Data Similarity and Dissimilarity
 Summary

Prof. Moumita Pal 9

 Basic Statistical Descriptions of Data
 Motivation
 To better understand the data: central tendency, variation and
spread
 Data dispersion characteristics
 median, max, min, quantiles, outliers, variance, etc.
 Numerical dimensions correspond to sorted intervals
 Data dispersion: analyzed with multiple granularities of
precision
 Boxplot or quantile analysis on sorted intervals
 Dispersion analysis on computed measures
 Folding measures into numerical dimensions
 Boxplot or quantile analysis on the transformed cube
Prof. Moumita Pal 10
 Measuring the Central Tendency
 Mean (algebraic measure) (sample vs. population):
Note: n is sample size and N is population size.
 Weighted arithmetic mean:
 Trimmed mean: chopping extreme values
 Median:
Middle value if odd number of values, or average of the middle two values otherwise
Estimated by interpolation (for grouped data):

 Mode:
 Value that occurs most frequently in the data
 Unimodal, bimodal, trimodal
 Empirical formula:
Prof. Moumita Pal 11
 Symmetric vs Skewed Data
 Median, mean and mode of symmetric, positively and negatively skewed
data

Prof. Moumita Pal 12

 Measuring the Dispersion of Data
 Quartiles, outliers and boxplots
 Quartiles: Q1 (25th percentile), Q3 (75th percentile)
 Inter-quartile range: IQR = Q3 – Q1
 Five number summary: min, Q1, median, Q3, max
 Boxplot: ends of the box are the quartiles; median is marked; add whiskers, and plot outliers
individually
 Outlier: usually, a value higher/lower than 1.5 x IQR
 Variance and standard deviation (sample: s, population: σ)
 Variance: (algebraic, scalable computation)

 Standard deviation s (or σ) is the square root of variance s2 (or σ2)

Prof. Moumita Pal 13
 Boxplot Analysis
 Five-number summary of a distribution
 Minimum, Q1, Median, Q3, Maximum
 Boxplot
 Data is represented with a box
The ends of the box are at the first and third
 quartiles, i.e., the height of the box is IQR

 The median is marked by a line within the box

Whiskers: two lines outside the box extended
 to Minimum and Maximum

 Outliers: points beyond a specified outlier threshold, plotted

individually
Prof. Moumita Pal 14
 Visualization of Data Dispersion

Prof. Moumita Pal 15

 Properties of Normal Distribution Curve
 The normal (distribution) curve
 From μ–σ to μ+σ: contains about 68% of the measurements

(μ: mean, σ: standard deviation)

 From μ–2σ to μ+2σ: contains about 95% of it

 From μ–3σ to μ+3σ: contains about 99.7% of it

Prof. Moumita Pal 16

 Graphic Displays of Basic Statistical
Descriptions
 Boxplot: graphic display of five-number summary

 Histogram: x-axis are values, y-axis represents frequencies

 Quantile plot: each value xi is paired with fi indicating that
approximately 100 fi % of data are  xi
 Quantile-quantile (q-q) plot: graphs the quantiles of one univariant distribution
against the corresponding quantiles of another
 Scatter plot: each pair of values is a pair of coordinates and plotted as points in
the plane

Prof. Moumita Pal 17

 Histogram Analysis
 Histogram: Graph display of tabulated frequencies, shown as bars
 It shows what proportion of cases fall into each of several categories.
 Differs from a bar chart in that it is the area of the bar that denotes the value, not the
height as in bar charts, a crucial distinction when the categories are not of uniform
width.
 The categories are usually specified as non-overlapping intervals of some variable. The
categories (bars) must be adjacent. 4
0
3
5
3
0
2
2
50
15

1
0
5
Prof. Moumita Pal 18

0
1000 3000 5000 7000 9000
 Histograms Often Tell More than
Boxplots
 The two histograms
shown in the left may
have the same boxplot
representation
 The same values for:
min, Q1, median, Q3,
max
 But they have rather
different data
distributions
Prof. Moumita Pal 19
 Quantile Plot
 Displays all of the data (allowing the user to assess both the
overall behavior and unusual occurrences)
 Plots quantile information
 For a data x data sorted in increasing order, f indicates that
i i
approximately 100 fi% of the data are below or equal to the
value xi

Prof. Moumita Pal 20

 Quantile-Quantile (Q-Q) Plot
 Graphs the quantiles of one univariate distribution against the corresponding
quantiles of another
 View: Is there is a shift in going from one distribution to another?
 Example shows unit price of items sold at Branch 1 vs. Branch 2 for each quantile. Unit
prices of items sold at Branch 1 tend to be lower than those at Branch 2.

Prof. Moumita Pal 21

Prof. Moumita Pal 22
 Scatter plot
 Provides a first look at bivariate data to see clusters of points, outliers, etc
 Each pair of values is treated as a pair of coordinates and plotted as points in
the plane

Prof. Moumita Pal 23

 Positively and Negatively Correlated
Data

 The left half fragment is positively

correlated
 The right half is negative correlated
Prof. Moumita Pal 24
 Uncorrelated Data

Prof. Moumita Pal 25

 Any Questions?

Prof. Moumita Pal 26