THE 3V’S OF BIG DATA: VARIETY, VELOCITY, and VOLUME

THE 3V’S OF BIG DATA: VARIETY, VELOCIT
and VOLUME
SPEAKER: Nick Weir
CEO
ChoozON

Friday, July 27, 2012

Mining
Big
Data
and
the
3V’s:
The
Business
and
Science
of
Big
Data
Nick
Weir,
Ph.D.

CEO
ChoozOn
Corpora,on
Twi$er:
@usamaf

March
21,
2012
GigaOM
Structure:
Data
Conference
New
York,
NY


Outline
• Big
Data
all
around
us
• The
role
of
Data
Mining
and
Predic6ve

Analy6cs
• On-‐line
data
and
facts
• Case
studies
from
mul6ple
ver6cals:
1. Yahoo!
Big
Data
2. Social
Network
Data
3. ChoozOn
Big
Data
from
oﬀers


What
MaAers
in
the
Age
of
Big
Data?

1.Being
able
to
exploit
all
the
data
that
is
available

• not
just
what
you've
got
available

• what
you
can
acquire
and
use
to
enhance
your
acFons

2.
Prolifera:ng
analy:cs
throughout
your
organiza:on
• make
every
part
of
your
business
smarter

3.
Driving
signiﬁcant
business
value

• embedding
analyFcs
into
every
area
of
your
business

can
help
you
drive
top
line
revenues
and/or
boJom
line

cost
eﬃciencies


Why
Big
Data?


Why
Big
Data?
A
new
term,
with
associated
“Data
Scien:st”
posi:ons:


Why
Big
Data?
A
new
term,
with
associated
“Data
Scien:st”
posi:ons:
• Big
Data:
is
a
mix
of
structured,
semi-‐structured,
and

unstructured
data:
–Typically
breaks
barriers
for
tradi:onal
RDB
storage
–Typically
breaks
limits
of
indexing
by
“rows”
–Typically
requires
intensive
pre-‐processing
before
each

query
to
extract
“some
structure”
–
usually
using
Map-‐
Reduce
type
opera:ons


Why
Big
Data?
A
new
term,
with
associated
“Data
Scien:st”
posi:ons:
• Big
Data:
is
a
mix
of
structured,
semi-‐structured,
and

unstructured
data:
–Typically
breaks
barriers
for
tradi:onal
RDB
storage
–Typically
breaks
limits
of
indexing
by
“rows”
–Typically
requires
intensive
pre-‐processing
before
each

query
to
extract
“some
structure”
–
usually
using
Map-‐
Reduce
type
opera:ons
• Above
leads
to
“messy”
situa6ons
with
no
standard

recipes
or
architecture:
hence
the
need
for
“data

scien6sts”

–Conduct
“Data
Expedi:ons”


What
Makes
Data
“Big
Data”?


What
Makes
Data
“Big
Data”?
• Big
Data
is
Characterized
by
the
3-‐V’s:
–Volume:
larger
than
“normal”
–
challenging
to
load
and

process
• Expensive
to
do
ETL
• Expensive
to
ﬁgure
out
how
to
index
and
retrieve
• MulQple
dimensions
that
are
“key”
–Velocity:
Rate
of
arrival
poses
real-‐,me
constraints
on
what

are
typically
“batch
ETL”
opera,ons
• If
you
fall
behind
catching
up
is
extremely
expensive
(replicate
very
expensive

systems)
• Must
keep
up
with
rate
and
service
queries
on-‐the-‐ﬂy
–Variety:
Mix
of
data
types
and
varying
degrees
of
structure
• Non-‐standard
schema
• Lots
of
BLOB’s
and
CLOB’s
• DB
queries
don’t
know
what
to
do
with
semi-‐structured
and
unstructured


The
DisQncQon
between
“Data”
and
“Big

Data”
is
fast
disappearing
• Most
real
data
sets
nowadays
come
with
a
serious
mix
of
semi-‐
structured
and
unstructured
components:
– Images
– Video
– Text
descrip:ons
and
news,
blogs,
etc…
– User
and
customer
commentary
– Reac:ons
on
social
media:
e.g.
TwiTer
is
a
mix
of
data
anyway
• Using
standard
transforms,
enQty
extracQon,
and
new

generaQon
tools
to
transform
unstructured
raw
data
into
semi-‐
structured
analyzable
data

• Hadoop
vs.
Not
Hadoop
-‐

when
to
use
what
kind
of
techniques

requiring
Map-‐Reduce
and
grid
compuQng


Text
Data:
The
Big
Driver

• While
we
speak
of
“big
data”
and
the
“Variety”
in
3-‐
V’s
• Reality:
biggest
driver
of
growth
of
Big
Data
has
been

text
data
• In
fact
Map-‐Reduce
became
popularized
by
Google
to

address
the
problem
of
processing
large
amounts
of

text
data:

– Indexing
a
full
copy
of
the
web
– Frequent
re-‐indexing
– Many
operaQons
with
each
being
a
simple
operaQon
but


To
Hadoop
or
not
to
Hadoop?
When
to
use
techniques
requiring
Map-‐Reduce
and
grid
compu?ng?
• Typically
organizaFons
try
to
use
Map-‐Reduce
for
everything
to
do
with

Big
Data
– Inefficient
and
oIen
irra6onal
– Certain
opera6ons
require
specialized
storage
• UpdaFng
segment
memberships
over
large
numbers
of
users
• Defining
new
segments
on
user
or
usage
data
• Map-‐Reduce
is
useful
when
a
very
simple
operaFon
is
to
be
applied
on

a
large
body
of
unstructured
data
– Typically
this
is
during
en6ty
and
aAribute
extrac6on
– S6ll
need
Big
Data
analysis
post-‐Hadoop
• Map-‐Reduce
is
not
efficient
or
effecFve
for
tasks
involving
deeper

staFsFcal
modeling
–
Good
for
gathering
counts
and
simple
(sufficient)
sta6s6cs
• E.g.
how
many
Fmes
a
keyword
occurs,
quick
aggregaFon
of
simple
facts
in
unstructured

data,
esFmates
of
variances,
density,
etc…
– Mostly
pre-‐processing
for
Data
Mining

Hadoop
Use
Cases
by
Data
Type


Big
Data
Applica:ons
and
Uses

100%
Capture
IT
Log
&
Security

Forensics
&
AnalyFcs Find
New
Signal Predict
Events

Product
Planning
Automated
Device
Data
Failure
Analysis
AnalyFcs
ProacFve
Fixes
RecommendaQon
AdverFsing SegmentaQon
AnalyFcs Social
Media

Hadoop
+

MPP
+
EDW
Big
Data Cost
ReducQon Ad
Hoc
Insight
Warehouse
AnalyFcs PredicQve
AnalyQcs


So
this
Internet
thing
is
going
to

be
big!
Big
Opportunity,
Big
Data,
Big

Challenges!


Stats
about
on-‐line
usage


Stats
about
on-‐line
usage
• How
many
people
are
on-‐line
today?

–2.1
Billion
(per
Comscore
es:mates)
–30%
of
world
Popula:on


Stats
about
on-‐line
usage
• How
many
people
are
on-‐line
today?

–2.1
Billion
(per
Comscore
es:mates)
–30%
of
world
Popula:on

• How
much
Fme
is
spent
on-‐line
per
month
by
the

whole
world?
–4M
person-‐years
per
month


Stats
about
on-‐line
usage
• How
many
people
are
on-‐line
today?

–2.1
Billion
(per
Comscore
es:mates)
–30%
of
world
Popula:on

• How
much
Fme
is
spent
on-‐line
per
month
by
the

whole
world?
–4M
person-‐years
per
month

• How
many
hours
per
month
per
Internet
User?
–16
hours
(global
average)
–32
hours
(U.S.
Average)


Stats
about
on-‐line
usage
• How
many
people
are
on-‐line
today?

–2.1
Billion
(per
Comscore
es:mates)
–30%
of
world
Popula:on

• How
much
Fme
is
spent
on-‐line
per
month
by
the

whole
world?
–4M
person-‐years
per
month

• How
many
hours
per
month
per
Internet
User?
–16
hours
(global
average)
–32
hours
(U.S.
Average)
*Sources: Feb.2012 - from Go-Gulf.com compiled from Comscoredatamine.com, Nielsen.com, thisDigitalLife.com,
PewInternet.org


Volume,
Velocity,
Variety


Volume,
Velocity,
Variety
• Google:

How
many
queries
per
day?


Volume,
Velocity,
Variety
• Google:

How
many
queries
per
day?
– More
than
1
Billion


Volume,
Velocity,
Variety
• Google:

How
many
queries
per
day?
– More
than
1
Billion
• Twi$er:
How
many
Tweets/day?


Volume,
Velocity,
Variety
• Google:

How
many
queries
per
day?
– More
than
1
Billion
• Twi$er:
How
many
Tweets/day?
– More
than
250M


Volume,
Velocity,
Variety
• Google:

How
many
queries
per
day?
– More
than
1
Billion
• Twi$er:
How
many
Tweets/day?
– More
than
250M
• Facebook:
Updates
per
day?


Volume,
Velocity,
Variety
• Google:

How
many
queries
per
day?
– More
than
1
Billion
• Twi$er:
How
many
Tweets/day?
– More
than
250M
• Facebook:
Updates
per
day?
– More
than
800M


Volume,
Velocity,
Variety
• Google:

How
many
queries
per
day?
– More
than
1
Billion
• Twi$er:
How
many
Tweets/day?
– More
than
250M
• Facebook:
Updates
per
day?
– More
than
800M
• YouTube:
Views/day


Volume,
Velocity,
Variety
• Google:

How
many
queries
per
day?
– More
than
1
Billion
• Twi$er:
How
many
Tweets/day?
– More
than
250M
• Facebook:
Updates
per
day?
– More
than
800M
• YouTube:
Views/day
– 4
Billion
views
– 60
hours
of
video
uploaded
every
minute!


Volume,
Velocity,
Variety
• Google:

How
many
queries
per
day?
– More
than
1
Billion
• Twi$er:
How
many
Tweets/day?
– More
than
250M
• Facebook:
Updates
per
day?
– More
than
800M
• YouTube:
Views/day
– 4
Billion
views
– 60
hours
of
video
uploaded
every
minute!
• Social
Networks:
users
who
have
used
sites
for
spying

on
their
partners?
–56%


Volume,
Velocity,
Variety
• Google:

How
many
queries
per
day?
– More
than
1
Billion
• Twi$er:
How
many
Tweets/day?
– More
than
250M
• Facebook:
Updates
per
day?
– More
than
800M
• YouTube:
Views/day
– 4
Billion
views
– 60
hours
of
video
uploaded
every
minute!
• Social
Networks:
users
who
have
used
sites
for
spying

on
their
partners?
–56%
*Sources: Feb.2012 - from Go-Gulf.com compiled from Comscoredatamine.com, Nielsen.com,
thisDigitalLife.com, PewInternet.org


Turning
the
three
Vs
of
Big
Data
into

Value


Turning
the
three
Vs
of
Big
Data
into

Value
Do
we
understand
what
each
individual
is
trying
to

achieve?


Turning
the
three
Vs
of
Big
Data
into

Value
Do
we
understand
what
each
individual
is
trying
to

achieve?
• What
is
user
intent?


Turning
the
three
Vs
of
Big
Data
into

Value
Do
we
understand
what
each
individual
is
trying
to

achieve?
• What
is
user
intent?
• Cri,cal
in
mone,za,on,
adver,sing,
etc…


Turning
the
three
Vs
of
Big
Data
into

Value
Do
we
understand
what
each
individual
is
trying
to

achieve?
• What
is
user
intent?
• Cri,cal
in
mone,za,on,
adver,sing,
etc…
Do
we
understand
what
a
community’s
sen:ment
is?


Turning
the
three
Vs
of
Big
Data
into

Value
Do
we
understand
what
each
individual
is
trying
to

achieve?
• What
is
user
intent?
• Cri,cal
in
mone,za,on,
adver,sing,
etc…
Do
we
understand
what
a
community’s
sen:ment
is?
•
What
is
the
emoFon?


Turning
the
three
Vs
of
Big
Data
into

Value
Do
we
understand
what
each
individual
is
trying
to

achieve?
• What
is
user
intent?
• Cri,cal
in
mone,za,on,
adver,sing,
etc…
Do
we
understand
what
a
community’s
sen:ment
is?
•
What
is
the
emoFon?
•
Is
it
negaFve
or
posiFve?


Turning
the
three
Vs
of
Big
Data
into

Value
Do
we
understand
what
each
individual
is
trying
to

achieve?
• What
is
user
intent?
• Cri,cal
in
mone,za,on,
adver,sing,
etc…
Do
we
understand
what
a
community’s
sen:ment
is?
•
What
is
the
emoFon?
•
Is
it
negaFve
or
posiFve?
•
What
is
the
health
of
my
brand
online?


Turning
the
three
Vs
of
Big
Data
into

Value
Do
we
understand
what
each
individual
is
trying
to

achieve?
• What
is
user
intent?
• Cri,cal
in
mone,za,on,
adver,sing,
etc…
Do
we
understand
what
a
community’s
sen:ment
is?
•
What
is
the
emoFon?
•
Is
it
negaFve
or
posiFve?
•
What
is
the
health
of
my
brand
online?
Do
we
understand
context
and
content?


Turning
the
three
Vs
of
Big
Data
into

Value
Do
we
understand
what
each
individual
is
trying
to

achieve?
• What
is
user
intent?
• Cri,cal
in
mone,za,on,
adver,sing,
etc…
Do
we
understand
what
a
community’s
sen:ment
is?
•
What
is
the
emoFon?
•
Is
it
negaFve
or
posiFve?
•
What
is
the
health
of
my
brand
online?
Do
we
understand
context
and
content?
•
What
are
appropriate
ads?


Turning
the
three
Vs
of
Big
Data
into

Value
Do
we
understand
what
each
individual
is
trying
to

achieve?
• What
is
user
intent?
• Cri,cal
in
mone,za,on,
adver,sing,
etc…
Do
we
understand
what
a
community’s
sen:ment
is?
•
What
is
the
emoFon?
•
Is
it
negaFve
or
posiFve?
•
What
is
the
health
of
my
brand
online?
Do
we
understand
context
and
content?
•
What
are
appropriate
ads?
•
Is
it
Ok
to
associate
my
brand
with
this
content?


Turning
the
three
Vs
of
Big
Data
into

Value
Do
we
understand
what
each
individual
is
trying
to

achieve?
• What
is
user
intent?
• Cri,cal
in
mone,za,on,
adver,sing,
etc…
Do
we
understand
what
a
community’s
sen:ment
is?
•
What
is
the
emoFon?
•
Is
it
negaFve
or
posiFve?
•
What
is
the
health
of
my
brand
online?
Do
we
understand
context
and
content?
•
What
are
appropriate
ads?
•
Is
it
Ok
to
associate
my
brand
with
this
content?
• Is
content
sad?,
happy?,
serious?,
informaFve?

Case
Studies
Yahoo!
Big
Data

18

Yahoo!
–
One
of
Largest
Des:na:ons

on
the
Web
More people visited
Yahoo! in the past
month than:
80%
of
the
U.S.
Internet
popula4on
uses
Yahoo!
• Use coupons
–
Over
600
million
users
per
month
globally! • Vote
• Global
network
of
content,
commerce,
media,
search
and
access
• Recycle
products • Exercise regularly
• 100+
properFes
including
mail,
TV,
news,
shopping,
ﬁnance,
autos,
• Have children
travel,
games,
movies,
health,
etc. living at home
• Wear sunscreen
• 25+
terabytes
of
data
collected
each
day regularly
• Represen:ng
1000’s
of
cataloged
consumer
behaviors

Data is used to develop content, consumer, category and campaign insights for
our key content partners and large advertisers

Sources: Mediamark Research, Spring 2004 and comScore Media Metrix, February 2005.


Yahoo!
Big
Data
–
A
league
of
its
own…

GRAND CHALLENGE PROBLEMS OF DATA PROCESSING
TRAVEL, CREDIT CARD PROCESSING, STOCK EXCHANGE, RETAIL, INTERNET

Y! Data Challenge Exceeds others by 2 orders of magnitude


Behavioral
Targe:ng
(BT)


Behavioral
Targe:ng
(BT)

Targe:ng
your
ads
to
consumers

whose
recent
behaviors
online

indicate
that
your
product

category
is
relevant
to
them


Yahoo!
User
DNA

• On a per consumer basis: maintain a
behavioral/interests profile and profitability (user
value and LTV) metrics

How
it
works
|
Network
+
Interests
+
Modelling


How
it
works
|
Network
+
Interests
+
Modelling
Analyze predictive patterns for purchase
cycles in over 100 product categories
Varying Product Purchase Cycles


How
it
works
|
Network
+
Interests
+
Modelling
Match Users to the Models

In each category, build models to describe
behaviour most likely to lead to an ad
response (i.e. click).


How
it
works
|
Network
+
Interests
+
Modelling
Rewarding Good Behaviour


Score each user for fit with every category…
daily.


How
it
works
|
Network
+
Interests
+
Modelling
Identify Most Relevant Users


Score each user for fit with every category…
daily.

Target ads to users who get highest
‘relevance’ scores in the targeting categories


Recency
MaTers,
So
Does
Intensity

Active now… …and with feeling


DiﬀerenQaQon
|
Category
speciﬁc
modelling


Automobile
Purchase
Intender
Example

• A
test
ad-‐campaign
with
a
major
Euro
automobile

manufacturer
– Designed
a
test
that
served
the
same
ad
creaQve
to
test
and

control
groups
on
Yahoo
– Success
metric:
performing
speciﬁc
acQons
on
Jaguar
website

• Test
results:
900%
conversion
lij
vs.
control
group
– Purchase
Intenders
were
9
Qmes
more
likely
to
conﬁgure
a

vehicle,
request
a
price
quote
or
locate
a
dealer
than
consumers

in
the
control
group
– ~3x
higher
click
through
rates
vs.
control
group


Mortgage
Intender
Example
Results from a client campaign on Yahoo!

Example: Mortgages Network

+626%
Conv Lift

We found: +122%
CTR Lift
1,900,000 people
Y! Finance Audience
Mortgage Loans Target
Mortgage Loans Target
looking for
mortgage loans. Example search terms qualified for this target:

Mortgages Home Loans Refinancing Ditech

Example Yahoo! Pages visited:
Financing section in Real Estate
Mortgage Loans area in Finance
Real Estate section in Yellow Pages

Source: Campaign Click thru Rate lift is determined by Yahoo! Internal research.
Conversion is the number of qualified leads from clicks over number of
impressions served. Audience size represents the audience within this
behavioral interest category that has the highest propensity to engage
Date: March 2006 with a brand or product and to click on an offer.


Experience summary at Yahoo!
• Dealing with the largest data source in the
world (25 Terabyte per day)
• BT business was grown from $20M to about
$500M in 3 years of investment!
• Building the largest database systems:
– World’s largest Oracle data warehouse
– World’s largest single DB
– Over 300 data mart data
– Analytics with thousands of KPI’s
– Over 5000 users of reports
– Largest targeting system in the world
• Big demands for grid computing (Hadoop)

Social
Network
Social
Network
Marke:ng
Understanding
Context
for

Ads


Case Study: TWITTER Social Marketing?
Viacom’s VH1 Twitter campaign on ANVIL (the movie)
(week of May 14th , 2009 – see AdAge article)

Marketing ANVIL movie
Very niche audience, how do you reach them?




Diffusion

Give 20 free DVD’s to major related artists/groups, ask them
To notify Twitter groups – reached over 2M people




Diffusion

Give 20 free DVD’s to major related artists/groups, ask them
To notify Twitter groups – reached over 2M people
Social Identity: power of word of mouth...
What is the Cost to VH-1?
Compare with traditional approach: TV commercials to promote a documentary
ﬁlm?


The
Display
Ads
Challenge
Today
Damaging to Brand Irrelevant and Damaging to Brand

Mismatch

Completely Irrelevant


NetSeer:
Intent
for
Display
• And
Currently
Processing
4
Billion
Impressions
per

Day


Problem:
Hard
to
Understand
User
Intent
Contextual
Ad
served
by
Google


Problem:
Hard
to
Understand
User
Intent
Contextual
Ad
served
by
Google What
NetSeer
Sees:


Specialized
Search
through
Big
Data

AnalyFcs
over
the
Oﬀers
Universe


Chaos
for

Consumers

Consumer


Chaos
for

Consumers

Deep
Discount
Sites Coupon
Flash
Sites
Sale
Sites
Loyalty
Program
s
Daily
Deals
Sites
Consumer Online
Loyalty
Program
Social s
Search
Network
Engines
s


The
Business
Model
behind
ChoozOn’s
Use
of
Big

Data


The
Business
Model
behind
ChoozOn’s
Use
of
Big

Data

Consumers
• Value from brands they love
• Tame the deal chaos


The
Business
Model
behind
ChoozOn’s
Use
of
Big

Data

Consumers
• Value from brands they love
• Tame the deal chaos

Marketers
• Reach targeted consumers
• Build loyalty
• Create brand evangelists


Carol’s
Consumer
Network


Carol’s
Consumer
Network

“Chozen”
Brands


Carol’s
Consumer
Network

“Chozen”
Brands Interests
(Intent)


Carol’s
Consumer
Network

“Chozen”
Brands Interests
(Intent)

Shopping
Pals


Carol’s
Consumer
Network

“Chozen”
Brands Interests
(Intent)

Shopping
Pals
Deal
Clubs


What
Carol
Gets

Carol’s
Consumer
Network


What
Carol
Gets

Carol’s
Consumer
Network The
Universe
of
Deals
Aﬃliate
Deals
Loyalty
Programs
Daily
Deals
Flash
Sales

Intelligent Matching

A
Personal
Shopper
for
Deals


What
Carol
Gets

Carol’s
Consumer
Network The
Universe
of
Deals
Aﬃliate
Deals 1,600+
brands
Loyalty
Programs
100,000+
oﬀers
Daily
Deals
Flash
Sales


A
Personal
Shopper
for
Deals


What
Carol
Gets

Carol’s
Consumer
Network The
Universe
of
Deals
Aﬃliate
Deals 1,600+
brands
Loyalty
Programs
100,000+
oﬀers
Daily
Deals
Flash
Sales
PLUS her Inbox™


A
Personal
Shopper
for
Deals


Thank
You!

TwiAer:
@choozon

nick@choozon.net
www.ChoozOn.com


THE 3V’S OF BIG DATA: VARIETY, VELOCITY, and VOLUME

More Related Content

Viewers also liked

Similar to THE 3V’S OF BIG DATA: VARIETY, VELOCITY, and VOLUME

More from Gigaom

Recently uploaded

THE 3V’S OF BIG DATA: VARIETY, VELOCITY, and VOLUME