Tutorial on Online User Engagement: Metrics and Optimization

Tutorial on Online User Engagement:
Metrics and Optimization
Liangjie Hong & Mounia Lalmas
1

Outline
2
Introduction and Scope
Metrics
Optimisation
Concluding Remarks & Future Directions

Who we are
● Mounia Lalmas, Research Director & Head of Tech Research @
Personalization at Spotify, London
○ Research interests: user engagement in areas such as advertising, digital
media, search, and now audio
○ Website: https://mounia-lalmas.blog/
● Liangjie Hong, Director of Engineering - Data Science
and Machine Learning at Etsy, New York City
○ Research interests: search, recommendation, advertising
and now hand-craft goods
○ Website: https://www.hongliangjie.com/
3

Acknowledgements
This tutorial is based on “Tutorial on Metrics of User Engagement:
Applications to News, Search and E-Commerce”, 11th ACM
International Conference on Web Search and Data Mining (WSDM),
Los Angeles, February 2018.
4

Introduction
Deﬁnitions
Scope
Case studies
6

What is user engagement? … Some definitions
User engagement is regarded as a persistent and pervasive cognitive
affective state, not a time-specific state.
Wilmar Schaufeli, Marisa Salanova, Vicente González-romá and Arnold Bakker. The Measurement of Engagement and Burnout: A Two
Sample Confirmatory Factor Analytic Approach. Journal of Happiness Studies, 2002.
7

User engagement refers to the quality of the user experience associated with
the desire to use a technology.
Heather O'Brien and Elaine Toms. What is user engagement? A conceptual framework for deﬁning user engagement with technology.
JASIST, 2008.
8

User engagement is a quality of the user experience that emphasizes the
positive aspects of interaction – in particular the fact of wanting to use the
technology longer and often.
Simon Attﬁeld, Gabriella Kazai, Mounia Lalmas and Benjamin Piwowarski. Towards a science of user engagement (Position Paper). WSDM
Workshop on User Modelling for Web Applications, 2011.
9

Characteristics of user engagement
10
Focused attention
Positive affect
Aesthetics
Endurability
Novelty
Richness and control
Reputation, trust and
expectation
Motivation, interests,
incentives and benefits
[1] Heather O'Brien and Elaine Toms. What is user engagement? A conceptual framework for defining user engagement with technology.
JASIST 2008.
[2] Heather O’Brien. Defining and Measuring Engagement in User Experiences with Technology. Doctoral thesis, Dalhousie University, 2008.
[3] Simon Attfield, Gabriella Kazai, Mounia Lalmas and Benjamin Piwowarski. Towards a science of user engagement (Position Paper). WSDM
Workshop on User Modelling for Web Applications, 2011.

11
Focused attention
Positive aﬀect
Aesthetics
Endurability
Novelty
expectation
Users must be focused to be engaged
Distortions in subjective perception of time used to measure it

12
Focused attention
Positive aﬀect
Aesthetics
Endurability
Novelty
expectation
Sensory, visual appeal of interface stimulates user and promotes focused attention
Perceived usability
Linked to design principles (e.g. symmetry, balance, saliency)

13
Focused attention
Positive aﬀect
Aesthetics
Endurability
Novelty
expectation
Novelty, surprise, unfamiliarity and the unexpected; updates & innovation
Appeal to user curiosity; encourages inquisitive behavior and promotes repeated engagement

14
Focused attention
Positive aﬀect
Aesthetics
Endurability
Novelty
expectation
Trust is a necessary condition for user engagement
Implicit contract among people and entities which is more than technological

15
Focused attention
Positive aﬀect
Aesthetics
Endurability
Novelty
expectation
Emotions experienced by user are intrinsically motivating
Initial aﬀective “hook” can induce a desire for exploration, active discovery or participation

16
Focused attention
Positive aﬀect
Aesthetics
Endurability
Novelty
expectation
People remember enjoyable, useful, engaging experiences and want to repeat them
Repetition of use, recommendation, interactivity, utility

17
Focused attention
Positive aﬀect
Aesthetics
Endurability
Novelty
expectation
Richness captures the growth potential of an activity
Control captures the extent to which a person is able to achieve this growth potential

18
Focused attention
Positive aﬀect
Aesthetics
Endurability
Novelty
expectation
Why should users engage?

19
Focused attention
Positive aﬀect
Aesthetics
Endurability
Novelty
expectation
[1] Heather L. O'Brien Elaine G. Toms. What is user engagement? A conceptual framework for
deﬁning user engagement with technology Journal of the American Society for Information Science
and Technology, Volume 59, Issue 6, February 2008.
Quality of the user experience … endurability
Endurability
People remember “satisfactory” experiences
and want to repeat them
We need metrics to quantify the quality of the
user experience with respect to endurability

Why is it important to engage users?
Users have increasingly enhanced expectations about their interactions with
technology
… resulting in increased competition amongst the providers of (online)
services.
utilitarian factors (e.g. usability) → hedonic and experiential factors of
interaction (e.g. fun, fulﬁllment) → user engagement
Mounia Lalmas, Heather O’Brien and Elad Yom-Tov. Measuring user engagement. Morgan & Claypool Publishers, 2014.
20

The engagement life cycle
21
Point of
engagement
Period of
engagement
Disengagement
Re-engagement
How engagement starts
Aesthetics & novelty in sync with user interests & contexts
Ability to maintain user attention and interests
Main part of engagement and usually the focus of study
Loss of interests lead to passive usage & even stopping usage
Identifying users that are likely to churn often undertaken
Engage again after becoming disengaged
Triggered by relevance, novelty, convenience, remember past positive experience
sometimes as result of campaign strategy

22
New
Users
Acquisition
Active Users
Activation
Disengagement
Dormant Users
Churn
Disengagement Re-engagement
Re-engagement
notiﬁcation, email, oﬀer,
marketing, advertising
Disengagement
churn & retention
The engagement life cycle
Point of engagement
how users arrive
acquisition costs
22

23
New
Users
Acquisition
Active Users
Activation
Disengagement
Dormant Users
Churn
Endurability in the engagement life cycle
Period of engagement
relate to user behaviour
with the product during a
session and across sessions
23
Endurability

Considerations in measuring user engagement
short term ←→ long term
laboratory ←→ “in the wild”
subjective ←→ objective
qualitative ←→ quantitative
large scale ←→ small scale
24

Methods to measuring user engagement
25
User study (lab/online)
mostly qualitative
User study (lab/online)
mostly quantitative,
scalability an issue
Data study (online)
quantitative
large scale
physiological
measurements
objective
online
analytics
objective
Data science
Machine learning
Task-based methods
Questionnaire,
interview, report,
product reaction cards
self-reported
methods
subjective

Scope of this tutorial
Focus on online analytics → online user engagement.
Assume that applications are “properly designed”.
Based on “published” work and our experience.
Focus on applications that users “chose” to engage with, widely used by
“anybody” on a “large-scale” and on a mostly regularly basis.
This tutorial is not an “exhaustive” account of works in this and related areas.
26

Case studies
Search
News
E-commerce
Entertainment
Advertising
27

Search
Search engine
evaluation
● Coverage
● Speed
● Query language
● User interface
Sec. 8.6
29
[1] Ricardo Baeza-Yates and Berthier Ribeiro-Neto. Modern Information Retrieval: The Concepts and Technology behind Search. ACM Press
Books, 2nd Edition, 2011.
[2] Christopher Manning, Prabhakar Raghavan and Hinrich Schütze. Introduction to Information Retrieval. Cambridge University Press, 2008.
User satisfaction
Users ﬁnd what they want and return to the
search engine for their next information need →
user engagement
But let us remember:
In carrying out a search task, search is a means,
not an end

Spotify’s mission is to
unlock the potential of
human creativity — by
giving a million creative
artists the opportunity
to live oﬀ their art and
billions of fans the
opportunity to enjoy
and be inspired by it.
Entertainment

36
Search
Native
Display
Video
Brand
Direct
Response
SUPPLY
(publishers)
DEMAND
(advertisers)
Advertising

37
Native advertising
Visually engaging
Higher user attention
Higher brand lift
Social sharing

Online metrics
Terminology, context &
consideration
Intra-session metrics
Inter-session metrics
Other metrics
39

Measures, metrics & key performance indicators
40
Measurement:
process of obtaining
one or more quantity
values that can
reasonably be
attributed to a quantity
e.g. number of clicks
Metric:
a measure is a number
that is derived from
taking a measurement
… in contrast, a metric
is a calculation
e.g. click-through rate
Key
performance
indicator (KPI):
quantiﬁable measure
demonstrating how
eﬀectively key
business objectives are
being achieved
e.g. conversion rate
https://www.klipfolio.com/blog/kpi-metric-measure
a measure can be used as metric but not all metrics are measures
a KPI is a metric but not all metrics are KPIs

Three levels of metrics
Business metrics -- KPIs
Behavioral metrics -- online metrics, analytics
Optimisation metrics -- metrics used to train machine
learning algorithms
41
Thesethreelevelsareconnected

Why several metrics?
42
Games
Users spend much time
per visit
Search
Users come frequently but do
not stay long
Social media
Users come frequently &
stay long
Niche
Users come on average once a
week
News
Users come periodically,
e.g. morning and evening
Service
Users visit site, when
needed

43
Playlists diﬀer in their
listening patterns.
Genres and moods can be
viewed as sub-hubs, each with
some common engagement
patterns.
Searching has a particular
engagement pattern.
Home can be viewed as a
hub with a “star” style
engagement pattern.
Media type and freshness lead to
diﬀerent engagement patterns.

Leaning backLeaning in Active Occupied
Playlists types
Pure discovery sets
Trending tracks
Fresh Finds
Playlist metrics
Downstreams
Artist discoveries
# or % of tracks sampled
Playlists types
Sleep
Chill at home
Ambient sounds
Playlist metrics
Session time
Playlists types
Workout
Study
Gaming
Playlist metrics
Session time
Skip rate
Playlists types
Hits ﬂagships
Decades
Moods
Playlist metrics
Skip rate
Downstreams
44

45
New
Users
Acquisition
Active Users
Activation
Disengagement
Dormant Users
Churn
Quality of the user experience … endurability
relate to user behaviour
with the product during a
session and across sessions
45
Endurability
Endurability in the engagement life cycle

Three levels of engagement related to endurability
46
Involvement
Interaction
Contribution
Presence of a user
pageview, dwell time, playtime, revisit rate
Action of a user
click-through rate, share, likes, conversion rate, save, click, skip rate
Input of a user
post, comment, create, update, reply, upload
What involvement is in application A may be interaction in application B
Degree of engagement in terms of “intention” increases from involvement → interaction → contribution

From visit to session
47
visit visit visit visit visit visit
session session session
Dwell time = time spent on site (page) during a visit
Session length is amount of time user spends on site within the session
Session frequency shows how often users are coming back (loyalty)
Often 30mn is used as threshold for session boundary (desktop)

From endurability to loyalty
intra-session metrics
● page level or less
● visit level
● session level
48
visit visit visit visit visit
session session session
visit
● return soon
● remain engaged later on
inter-session metrics
long-term value (LTV) metrics

Intra-session (within → endurability) inter-session (across → habit)
Involvement
• Dwell time
• Session duration
• Page view (click depth)
• Revisit rate
• Bounce rate
Interaction
• Click-through rate (CTR)
• Number of shares, likes, saves
• Conversion rate
• Streamed, played
Contribution
• Number of replies
• Number of blog posts
• Number of uploads
Granularity
Module
↓
Viewport
↓
Page
↓
Visit
↓
Session
From one session to the next session (return soon)
• Time between sessions (absence time)
inter-session (across → loyalty)
From one session to a next time period such next week,
or in 2 weeks time (remain engaged later on)
• Number of active days
• Number of sessions
• Total usage time
• Number of clicks
• Number of shares
• Number of thumb ups
Intra- vs inter-sessions metrics
• intra-session engagement measures user activity on the site during the session → endurability
• inter-session engagement measures user habit & loyalty with the site → long-term value

Intra- vs inter-sessions metrics … Granularity
Module → Viewport → Page → Visit → Session
Optimisation mostly with these metrics, with increasing complexity from “Module” to “Session”
Next session → Next Day → Next Week → Next Month, etc.
50

Click-through rate
Dwell time
“Organise” metrics
Revisit rate
Page view
Conversion rate
Social media metrics
51

Click-through rate
Dwell time
Revisit rate
Page view
Conversion rate
52

Ratio of users who click on a speciﬁc link to the number of total
users who view a page, email, or advertisement
Translates to play song/video for music/video sites/formats
● Abandonment rate
● Clickbait
● Site design
● Accidental clicks (mobile)
53
Click-through rates (CTR) … Interaction

Click-through rate:
% of clicks when URL
shown (per query)
Hover rate:
% hover over URL
(per query)
Unclicked hover:
Median time user hovers over
URL but no click (per query)
Max hover time:
Maximum time user hovers
over a result (per SERP)
55
No click … Search
Jeﬀ Huang, Ryen White and Susan Dumais. No clicks, no problem: using cursor movements to understand and improve
search. CHI 2011.

Abandonment is when there is no click on the search result page
User is dissatisﬁed (bad abandonment)
User found result(s) on the search result page (good abandonment)
858 queries (21% good vs. 79% abandonment manually examined)
Cursor trail length
Total distance (pixel) traveled by cursor on SERP
Shorter for good abandonment
Movement time
Total time (second) cursor moved on SERP
Longer when answers in snippet (good abandonment)
56
Cursor speed
Average cursor speed (pixel/second)
Slower when answers in snippet (good abandonment)
No click … Search
Jeﬀ Huang, Ryen White and Susan Dumais. No clicks, no problem: using cursor movements to understand and improve search.
CHI 2011.

The quality of a click on mobile apps … advertising
57
peak on app X
● accidental clicks do not reﬂect
post-click experience
● not all clicks are equal
app X
peak on app Y
dwell time distribution of apps X and Y for given ad
app Y
Gabriele Tolomei, Mounia Lalmas, Ayman Farahat and Andy Haines. Data-driven identiﬁcation of accidental clicks on mobile ads
with applications to advertiser cost discounting and click-through rate prediction. Journal of Data Science and Analytics, 2018.

Ratio of users who click on a speciﬁc item to the number of total users who
“view” that item
What is an item?
● Track
● Artist page
● Album
● Playlist
● ...
The value of a click
→ downstream engagement
58
Click-through rate … Music

Downstream engagement … music
What the user does from a particular click
at “place X” → downstream behaviour:
● Total number of tracks played/saved from artist
contained within X
● Number of visits to album pages/artist pages contained
within X
● Total time spent on album pages/artist pages contained
within X
● Total number of playlists updated/created with tracks
contained within X
● ...
→ building relationships
Rishabh Mehrotra, Mounia Lalmas, Doug Kenney, Tim Lim-Meng and Golli Hashemian. Jointly Leveraging Intent and Interaction
Signals to Predict User Satisfaction with Slate Recommendations. WWW 2019.

Click-through rate
Dwell time
Revisit rate
Page view
Conversion rate
60

The contiguous time spent on a site
or web page
Similar measure is play/streaming
time for video and music streaming
services
● Not clear what user is actually looking at
while on page/site
● Instrumentation issue with last page
viewed and open tabs
61
Dwell time … Involvement
distribution of dwell times on 50 websites

Dwell time varies by site
type: leisure sites tend to
have longer dwell times than
news, e-commerce, etc.
Dwell time has a relatively
large variance even for the
same site
62
Dwell time … Involvement
average and variance of dwell time of 50 sites
[1] Mounia Lalmas, Heather O’Brien and Elad Yom-Tov. Measuring user engagement. Morgan & Claypool Publishers, 2014.
[2] Elad Yom-Tov, Mounia Lalmas, Ricardo Baeza-Yates, Georges Dupret, Janette Lehmann and Pinar Donmez. Measuring Inter-Site Engagement.
BigData 2013.

“reading” cursor heatmap of relevant document vs “scanning” cursor heatmap of non-relevant document
Dwell time … Search
63
Qi Guo and Eugene Agichtein. Beyond dwell time: estimating document relevance from cursor movements and other post-click
searcher behavior. WWW 2012.

“reading” a relevant long document vs “scanning” a long non-relevant document
64
Dwell time … Search
Guo and Eugene Agichtein. Beyond dwell time: estimating document relevance from cursor movements and other post-click
searcher behavior. WWW 2012.

Dwell time … news
65
Dwell time better proxy for user interest on
news article in the context of personalization
Optimizing for dwell time led to increase in
click-through rates
A way to reduce and optimize for click-baits
See section on Oﬄine experiment and evaluation
Xing Yi, Liangjie Hong, Erheng Zhong, Nanthan Nan Liu and Suju Rajan. Beyond Clicks: Dwell Time for Personalization. RecSys 2014.

66
Dwell time as streaming time … music
Aggregate over playlists
Consumption time of leep playlist longer than average playlist consumption time.
Optimizing for mean consumption time led to
+22.24% in predicted stream rate compared
to stream rate (equivalent to click-through
rate) on Spotify Home
Paolo Dragone, Rishabh Mehrotra and Mounia Lalmas. Deriving User- and Content-speciﬁc Rewards for Contextual Bandits. WWW 2019.

Dwell time and ad landing page quality
Dwell time → ad click
Positive post-click experience (“long” clicks) has
an eﬀect on users clicking on ads again (mobile)
User click on an ad → ad landing page
Dwell time is time until user returns to publisher and
used as proxy of quality of landing page
67
Mounia Lalmas, Janette Lehmann, Guy Shaked, Fabrizio Silvestri and Gabriele Tolomei. Promoting Positive Post-click Experience
for In-Stream Yahoo Gemini Users. KDD Industry track 2015.

Click-through rate
Dwell time
Revisit rate
Page view
Conversion rate
68

TYPE/TALK
User
communicates
with us
CONSIDER
User evaluates
what we show
them
DECIDE
User ends the
search session
User journey in search … Music
EFFORT
Users evaluate
their experience
on search based
on two main
factors: success
and eﬀort
SUCCESS

DECIDE TYPE
number of
deletions, ...
CONSIDER
back button
clicks, ﬁrst and
last click
position, ...
Time to success
“Success” metrics “Eﬀort” metrics
stream
LISTEN
Have a listening session
add to a playlist, save
into a collection,
follow an artist,
follow a playlist, ...
ORGANIZE
Curate for future listening
In A/B testing, success rate more sensitive than click-through rate.
Organize metrics … Interaction
Praveen Ravichandran, Jean Garcia-Gathright, Christine Hosey, Brian St. Thomas and Jenn Thom. Developing Evaluation Metrics for Instant
Search Using Mixed Methods. SIGIR 2019.

Click-through rate
Dwell time
Revisit rate
Page view
Conversion rate
71

Number of returns to the website within a
session → deﬁnition of a session?
Common in sites which may be browser
homepages, or contain content of regular
interest to users.
Useful for sites such as news aggregators,
where returns indicate that user believes
there may be more information to glean
from the site
72
Revisit rates … Involvement

Goal-oriented sites (e.g.,
e-commerce) have lower
revisits in a given time
range observed → revisit
horizon should be
adjusted by site
73
Revisit rates … Involvement
Elad Yom-Tov, Mounia Lalmas, Ricardo Baeza-Yates, Georges Dupret, Janette Lehmann and Pinar Donmez. Measuring Inter-Site
Engagement. BigData 2013.

Revisit rate … Session length
2.5M users, 785M page views, 1 month sample
Categorization of the most frequently
accessed sites
11 categories (e.g. news), 33 subcategories
(e.g. news ﬁnance, news society)
60 sites from 70 countries/regions
short sessions: average 3.01 distinct sites visited with revisit rate 10%
long sessions: average 9.62 distinct sites visited with revisit rate 22%
74
Janette Lehmann, Mounia Lalmas, Georges Dupret and Ricardo Baeza-Yates. Online Multitasking and User Engagement. CIKM 2013.

Time between each revisit … online multi-tasking
50% of sites are
revisited after less
than 1 minute
75
Janette Lehmann, Mounia Lalmas, Georges Dupret and Ricardo Baeza-Yates. Online Multitasking and User Engagement. CIKM 2013.

Click-through rate
Dwell time
Revisit rate
Page view
Conversion rate
76

Pageview … Involvement
Page view is request to load a single page
Number of pages viewed (click depth): average number of contiguous pages
viewed during a visit → “user journey” across the application
Reload after reaching the page → counted as additional pageview
If same page viewed more than once → a single unique pageview
77
Can be problematic with
ill-designed site as high click depth
may reﬂect users getting lost and
user frustration.
https://www.slideshare.net/timothylelek/google-analytics-for-dummies

Conversion rate … Interaction
Fraction of sessions which end in a desired user action
particularly relevant to e-commerce (making a purchase) … but also include subscribing, free to
premium user conversion
Online advertising using conversion as cost model to charge
advertisers
Not all sessions are expected to result in a conversion, so this
measure not always informative
dwell time often used as proxy of satisfactory experience as may reﬂect aﬃnity with the brand
78

79
Conversations
#comments, #posts,
#replies, #edits
Applause
#like, #thumbs up or
down, #hearts, +1
Amplification
#share, #mail
…
interaction
…
interaction …
contribution
https://www.kaushik.net/avinash/best-social-media-metrics-conversation-amplification-applause-economic-value/

Some ﬁnal words
What comes next
80

Some ﬁnal words on intra-session metrics
Metrics for smaller granularity levels such as
viewport or speciﬁc section → attention
Metrics for scroll → important for stream and
mobile
81
Whether an intra-session metric belongs to
Involvement, Interaction, or Contribution may depend
on the expected type of engagement of the site
scrollingdown
viewport
[1] Dmitry Lagun and Mounia Lalmas. Understanding and Measuring User Engagement and Attention in Online News Reading. WSDM 2016.
[2] Dmitry Lagun, Chih-Hung Hsieh, Dale Webster and Vidhya Navalpakkam. Towards better measurement of attention and satisfaction in
mobile search. SIGIR 2014.

Inter-session metrics → Loyalty
How many users and how fast they return to the site
Total use measurements → Popularity
Total usage time
Total number of sessions
Total view time (video)
Total number of likes (social networks)
Direct value measurement → Lifetime value
Lifetime value, as measured by ads clicked, monetization, etc.
82
Non intra-session metrics

83
Why inter-session metrics
Relationship to loyalty
Absence time

Intra-session measures can easily
mislead, especially for a short time
Consider a very poor ranking function
introduced into a search engine by
mistake
Therefore, bucket testing may provide
erroneous results if only intra-session
measures are used
84
Why inter-session metrics?
Ron Kohavi, Alex Deng, Brian Frasca, Roger Longbotham, Toby Walker and Ya Xu. Trustworthy online controlled experiments: Five puzzling
outcomes explained. KDD 2012.

85
visit
nextsession
visit
visit
visit
visit
visit
absence time next day, next
week, next
month, etc
Total number of sessions
Total number of days active
Total number of clicks
Total amount of time spent ...
a day, a week, 2 weeks, a month, etc

Total number of visits or sessions
Inter-session metrics … loyalty
86
visit
nextsession
visit
visit
visit
visit
visit
week, next
month, etc
Intra-Session Metrics Inter-Session Metrics
Correlation/Causation
See section on Optimization
long-term engagement
relate to business & KPI metrics

Total number of visits or sessions
Inter-session metrics … loyalty
87
visit
nextsession
visit
visit
visit
visit
visit
week, next
month, etc
really mostly about endurability
habit
periodicity
short task/visit
absence time ≠ revisit rate
Cases studies: search and news

Absence time applied to search … Study I
Ranking functions on Yahoo Answer Japan
Two-weeks click data
on Yahoo Answer
Japan search
One millions users
Six ranking functions
Session boundary:
30 minutes of
inactivity
88
Georges Dupret and Mounia Lalmas. Absence time and user engagement: evaluating ranking functions. WSDM 2013.

Examples of metrics for search
(Proxy: relevance of a search result)
Number of clicks
SAT click
Quick-back click
Click at given position
Time to ﬁrst click
Skipping
Abandonment rate
Number of query reformulations
Dwell time (result vs result page)
89

Absence time and survival analysis
SURVIVE
DIE
DIE = RETURN TO SITE
➔ SHORT ABSENCE TIME
hours
90
Odd Aalen, Ornulf Borgan and Hakon Gjessing. Survival and Event History Analysis: A Process Point of View. Statistics for Biology and
Health, 2008.

survival analysis: high hazard rate (die quickly) = short absence
5 clicks
control=noclick
Absence time and number of clicks
3 clicks
No click means a bad user search session … in Yahoo Japan search
Clicking between 3-5 results leads to same user search experience
Clicking on more than 5 results reﬂects poor user search session; users cannot ﬁnd what they
are looking for 91

DCG versus absence time to evaluate ﬁve ranking
functions
DCG@1
Ranking Alg 1
Ranking Alg 2
Ranking Alg 3
Ranking Alg 4
Ranking Alg 5
DCG@5
Ranking Alg 1
Ranking Alg 3
Ranking Alg 2
Ranking Alg 4
Ranking Alg 5
Absence time
Ranking Alg 1
Ranking Alg 2
Ranking Alg 5
Ranking Alg 3
Ranking Alg 4
92

Absence time and search session … What else?
● Clicking lower in the ranking (2nd
, 3rd
) suggests more careful choice from
the user (compared to 1st
)
● Clicking at bottom is a sign of low quality overall ranking
● Users ﬁnding their answers quickly (time to 1st
click) return sooner to
the search application
● Returning to the same search result page is a worse user experience
than reformulating the query
intra-session search metrics → absence time
93
Georges Dupret and Mounia Lalmas. Absence time and user engagement: evaluating ranking functions. WSDM 2013.

Absence time and search experience … Study II
From 21 experiments carried out through A/B testing, using absence time
agrees with 14 of them (which one is better)
Positive
One more query in session
One more click in session
SAT clicks
Query reformulation
Negative
Abandoned session
Quick-back clicks
94
Sunandan Chakraborty, Filip Radlinski, Milad Shokouhi and Paul Baecke. On Correlation of Absence Time and Search Eﬀectiveness. SIGIR 2014.

Absence time and search experience … Studies I & II
Demonstrated that absence time is an appropriate inter-session metric for
search because of the correlation & predictive power of known indicators of
a positive search experience
→ absence time as a metric to compare A/B test in search
These known indicators could act as intra-session metrics, which could be
optimised by the ranking algorithms
They can also be used as features in the ranking algorithms themselves
95

Related oﬀ-site content
Absence time & focused news reading
For 70% of news sites that provide links to oﬀ-site content,
probability that users return within 12 hours increases by 76%
96
Janette Lehmann, Carlos Castillo, Mounia Lalmas and Ricardo Baeza-Yates. Story-focused Reading in Online News and its Potential
for User Engagement. JASIST 2016.

Other metrics
97
● Popularity
● Long-term value (LTV)

Popularity metrics
With respect to users
● MAU (monthly active users), WAU (weekly active users), DAU (daily active users)
● Stickiness (DAU/MAU) measures how much users are engaging with the product
● Segmentation used to dive into demographics, platform, recency, …
With respect to usage
● Absolute value metrics (measures) → aggregates over visits/sessions
total number of clicks; total number of sessions; total number of time spent per day,
month, year
● Usually correlate with number of active users
98

Long-term value (LTV) metrics
99
How valuable diﬀerent users are based on lifetime performance → value that
a user is expected to generate over a given period time, e.g. such as 12
months
● Services relying on advertising for revenue:
○ based on a combination of forecasted average pageviews per user, actual retention &
revenue per pageview
● E-commerce relying on actual purchases:
○ based on total amount of purchases
Help analyzing acquisition strategy
(customer acquisition cost) and estimate
further marketing costs
https://www.apptamin.com/blog/lifetime-value-mobile-customer/

Recap
100
Online engagement & metrics
How it all ﬁts together

Online engagement & metrics … recap
101
User journey
day 1, day 2, … , week 1, ... now
Acquisition→
retainingnewusers
Intra-session
Involvement
Interaction
Contribution
Optimisation
Aggregates → popularity
Intra-session
Involvement
Interaction
Contribution
Optimisation
Aggregates → popularity
inter-session → loyalty
Disengagement?
Re-engagement?
correlation
prediction

102
Metrics to use to
optimize machine
learning algorithms
Key performance
indicators (KPIs)
Long-term value
(LTV) metrics
Popularity metrics
Online engagement & metrics … all together

Optimization
Manual/Semi-Manual Optimization
Automatic Optimization
Combining Two Camps
104

Two Camps of Optimizations
105
● Manual/Semi-Manual Optimization
- e.g. The classic Hypothesis-Experiment-Evaluation Cycle
● Automatic Optimization
- e.g., Online Learning, Multi-armed Bandits, Reinforcement Learning...

Two (Three?) Camps of Optimizations
106
- e.g., Online Learning, Multi-armed Bandits, Reinforcement Learning…
● Combining Two Camps

Manual/Semi-Manual
Optimization
Online Experiments and Evaluation
Oﬄine Experiments and Evaluation
Observational Study
107

Manual/Semi-Manual Optimization
108
Introduced by Jason Gauci from Facebook

109
A/B Tests or Bucket Tests or Online Controlled Experiments

110
● A lot of statistical tools offer measuring the difference between control and treatment
● Link to Average Treatment Effect (ATE) in Causal Inference
● Sometimes the only way to understand causal effects
● Easy to implement and easy to explain
[1] Ben Carterette. Statistical Significance Testing in Information Retrieval: Theory and Practice. SIGIR 2017 Tutorial.
[2] Tetsuya Sakai. Statistical Significance, Power, and Sample Sizes: A Systematic Review of SIGIR and TOIS, 2006-2015. SIGIR 2016.
[3] Tetsuya Sakai. The Probability that Your Hypothesis Is Correct, Credible Intervals, and Effect Sizes for IR Evaluation. SIGIR 2017.
[4] Benjamin A. Carterette. Multiple Testing in Statistical Analysis of Systems-based Information Retrieval Experiments. ACM Trans. Inf. Syst.
30, 1, Article 4, 2012.

111
● A lot of statistical tools offer measuring the difference between control and treatment
● Link to Average Treatment Effect (ATE) in Causal Inference
● Sometimes the only way to understand causal effects
● Easy to implement and easy to explain
● Not well studied in a lot of online settings
● Gold standard for statistical difference
● Weak for practical difference
[1] Ben Carterette. Statistical Significance Testing in Information Retrieval: Theory and Practice. SIGIR 2017 Tutorial.
[2] Tetsuya Sakai. Statistical Significance, Power, and Sample Sizes: A Systematic Review of SIGIR and TOIS, 2006-2015. SIGIR 2016.
[3] Tetsuya Sakai. The Probability that Your Hypothesis Is Correct, Credible Intervals, and Effect Sizes for IR Evaluation. SIGIR 2017.
[4] Benjamin A. Carterette. Multiple Testing in Statistical Analysis of Systems-based Information Retrieval Experiments. ACM Trans. Inf. Syst.
30, 1, Article 4, 2012.

112

113
Xuan Yin and Liangjie Hong. The Identiﬁcation and Estimation of Direct and Indirect Eﬀects in Online A/B Tests through Causal
Mediation Analysis. In KDD 2019.

114
● Online Controlled Experiments and Evaluation
○ Pros:
■ A lot of statistical tools offer measuring the difference between control and treatment
■ Link to Average Treatment Effect (ATE) in Causal Inference
■ Sometimes the only way to understand causal effects
■ Easy to implement and easy to explain
○ Cons:
■ Live traffic is limited (100%)
■ Power differences need time (days to weeks)
■ Cycles and number of innovations are bounded
■ Might hurt user engagement
■ Engineering cost
■ Cannot re-use
■ Nuances to get more accurate insights

115
Metrics for Online Experiments
● Directional
Have correlations with inter-session metrics and KPIs.

116
Metrics for Online Experiments
● Directional
Have correlations with inter-session metrics and KPIs.
● Sensitivity
Easily detect changes.

117
Summary
● Direct and dynamic
● Causality
● Metrics for online experiments
● Impacts (e.g, user engagement, traﬃc, set-up and etc.)
● Cannot re-use
[1] Ron Kohavi, Roger Longbotham, Dan Sommerﬁeld and Randal M. Henne. 2009. Controlled Experiments on the Web: Survey and
Practical Guide. DMKD 18, 1 (February 2009).
[2] Alex Deng and Xiaolin Shi. 2016. Data-Driven Metric Development for Online Controlled Experiments: Seven Lessons Learned. KDD
2016.
[3] Pavel Dmitriev, Somit Gupta, Dong Woo Kim and Garnet Vaz. 2017. A Dirty Dozen: Twelve Common Metric Interpretation Pitfalls in
Online Controlled Experiments. KDD 2017.

Oﬄine Experiment and Evaluation
118
Traditional Oﬄine Dataset/Collection Experiment
● High risk experiments.
It may drive users away.

119
● Learn more insights & highly reusable.
Easy to gather data and easy to compute metrics and compare.

120
● Learn more insights & highly reusable.
Easy to gather data and easy to compute metrics and compare.
● Machine learning theory of generalization.
Textbook scenario.

121
Train Validation Test
Offline Online

122
● Supervised Learning
● Cross-validation
● View online experiments as extension to oﬄine optimization (testset)
Offline Online

123
Optimizing Inter-Session Metrics
If inter-session metrics can be explicitly modeled or write them down in their clear form, you can
use online optimization tools to directly optimize them.

124
Approach I

125
Approach I
● This is usually diﬃcult or impossible because of
○ Complexity of inter-session metrics (you can’t really write them down or hard).
○ You don’t have data.
○ Your have extremely sparse data.
○ Hard to deploy such systems.
...

126
Liang Wu, Diane Hu, Liangjie Hong and Huan Liu. Turning Clicks into Purchases: Revenue Optimization for Product Search in
E-Commerce. SIGIR 2018.

127
Optimizing Gross-Merchandise-Value (GMV) in E-commerce Search
● Expected GMV

128
● Purchase Decision Process

129
● Click Decision(s) from Search-Result-Page (SERP)
● Purchase Decision(s) from Listing Page

130
● Click Decision(s) from Search-Result-Page (SERP)
fc
is learned by a neural-network model through back-prop.

131
● Purchase Decision from Listing Page
Price-Weighted Logistic Regression

132

133

134

135

136

137
● This work is about optimizing GMV in Session
○ How about long-term GMV?
○ How about other discovery?
…
● First step in optimizing user engagements in E-commerce search.

138
Approach II
Optimization

Approach II
1. Intra-Session and Inter-Session Correlation
2. Optimization Intra-Session as Surrogate
3. Finding (Better) Proxy Metrics
Optimization
139

140
Optimization Correlation/Causation

141
Beyond Clicks: Dwell Time in Personalization
Xing Yi, Liangjie Hong, Erheng Zhong, Nanthan Nan Liu and Suju Rajan. Beyond Clicks: Dwell Time for Personalization. RecSys 2014.

142

143

144

145

146

147
● Optimizing Dwell-Time becomes the de-facto method to drive user engagement in Yahoo News
Stream.
● The inter-session user engagement metric is a variant of dwell-time on sessions, considering
the depth of the session.
● They correlate very well in quarterly basis.

148
Summary
● Approach I, Direct Optimization
● Approach II, Correlation and Optimization

149
It doesn’t work or it doesn’t work smoothly.

● Bias
Examples: presentation bias, system bias...
150
Offline Online

● Concept Drifts
Examples: seasonal, interest shift...
151
Offline Online

● Diﬀerent of oﬄine metrics and online metrics
Examples: AUC/nDCG versus DAU...
152
Offline Online

153
● Bias
● Concept Drift
● Diﬀerent of oﬄine metrics and online metrics
Offline Online

154
● Selection/sampling bias
e.g. presentation bias, system bias
● Correlation
e.g. hard to control everything
● Static
e.g., temporal dynamics, lacking “new” user behaviors

155
Summary
● Indirect and can be reused
● Good machine learning theories
● Correlation
● Static
[1] Mark Sanderson. Test Collection Based Evaluation of Information Retrieval Systems. Foundations and Trends® in Information
Retrieval: Vol. 4: No. 4, 2010
[2] Donna Harman. Information Retrieval Evaluation. Synthesis Lectures on Information Concepts, Retrieval, and Services 3:2, 2011.

Oﬄine A/B Experiment and Evaluation
156
Counterfactual Oﬄine Reasoning/Experiment

157
Logging Policy
● Uniform-randomly show items.
● Gather user feedbacks (rewards).
New Policy
● Show items according to a model/algorithm.
● Accumulate rewards if item matches history pattern.
[1] Lihong Li, Wei Chu, John Langford and Xuanhui Wang. Unbiased Online Evaluation of Contextual-bandit-based News Article
Recommendation Algorithms. WSDM 2011.
[2] Alexander Strehl, John Langford, Lihong Li and Sham Kakade. Learning from Logged Implicit Exploration data. NIPS 2010.

158
Lihong Li, Wei Chu, John Langford and Xuanhui Wang. Unbiased Online Evaluation of Contextual-bandit-based News Article

159

160
● Address data bias
● Causality
● Reusable
● Some good theories

161
● Generalization to Non-uniform Logging/Exploration

162
● Generalization to Non-uniform Logging/Exploration

163
● Need logging and an exploration strategy
● In development, emerging topic

164
How to eﬀectively gather data that minimize hurting user engagement metrics?
[1] Liangjie Hong and Adnan Boz. An Unbiased Data Collection and Content Exploitation/Exploration Strategy for Personalization.
CoRR abs/1604.03506, 2016.
[2] Tobias Schnabel, Paul N. Bennett, Susan Dumais and Thorsten Joachims. Short-Term Satisfaction and Long-Term Coverage:
Understanding How Users Tolerate Algorithmic Exploration. WSDM 2018.

165
● Uniform-random greatly hurts user engagement and nobody is doing this.
● Classic Thompson Sampling and Upper-Conﬁdence-Bound would eventually converge.

166
● Uniform-random greatly hurts user engagement and nobody is doing this.
● Classic Thompson Sampling and Upper-Conﬁdence-Bound would eventually converge.
Requirements:
● Provide randomness and do not converge.
● User-friendly.

167

168

169

170

171
Generic Idea:
1. Rewrite the objective function with inverse propensity scoring.
2. Try to optimize or approximate the new objective.
3. Optimization under counterfactual setting, simulating A/B testing
References:
[1] Xuanhui Wang, Michael Bendersky, Donald Metzler and Marc Najork. Learning to Rank with Selection Bias in Personal Search. SIGIR
2016.
[2] Thorsten Joachims, Adith Swaminathan and Tobias Schnabel. Unbiased Learning-to-Rank with Biased Feedback. WSDM 2017.
[3] Thorsten Joachims and Adith Swaminathan. Counterfactual Evaluation and Learning for Search, Recommendation and Ad
Placement. SIGIR 2016 Tutorial.
[4] Adith Swaminathan and Thorsten Joachims. Counterfactual risk minimization: learning from logged bandit feedback. ICML 2015.
[5] Lihong Li, Jinyoung Kim and Imed Zitouni. Toward Predicting the Outcome of an A/B Experiment for Search Relevance. WSDM 2015.
[6] Adith Swaminathan et al. Oﬀ-policy evaluation for slate recommendation. NIPS 2017.
[7] Tobias Schnabel, Adith Swaminathan, Peter Frazier and Thorsten Joachims. Unbiased Comparative Evaluation of Ranking Functions.
ICTIR 2016.
[8] Alexandre Gilotte, Clément Calauzènes, Thomas Nedelec, Alexandre Abraham and Simon Dollé. Oﬄine A/B testing for Recommender
Systems. WSDM 2018.

172
Summary
● Causality
● Reusable
● Need logging and an exploration strategy
● In development, emerging topic
References:
[1] Xuanhui Wang, Michael Bendersky, Donald Metzler and Marc Najork. Learning to Rank with Selection Bias in Personal Search. SIGIR
2016.
[2] Thorsten Joachims, Adith Swaminathan and Tobias Schnabel. Unbiased Learning-to-Rank with Biased Feedback. WSDM 2017.
[3] Thorsten Joachims and Adith Swaminathan. Counterfactual Evaluation and Learning for Search, Recommendation and Ad
Placement. SIGIR 2016 Tutorial.
[4] Adith Swaminathan and Thorsten Joachims. Counterfactual risk minimization: learning from logged bandit feedback. ICML 2015.
[5] Lihong Li, Jinyoung Kim and Imed Zitouni. Toward Predicting the Outcome of an A/B Experiment for Search Relevance. WSDM 2015.
[6] Adith Swaminathan et al. Oﬀ-policy evaluation for slate recommendation. NIPS 2017.
[7] Tobias Schnabel, Adith Swaminathan, Peter Frazier and Thorsten Joachims. Unbiased Comparative Evaluation of Ranking Functions.
ICTIR 2016.
[8] Alexandre Gilotte, Clément Calauzènes, Thomas Nedelec, Alexandre Abraham and Simon Dollé. Oﬄine A/B testing for Recommender
Systems. WSDM 2018.

Observational Study
173
Sometimes, even oﬄine experiments may not be feasible or practical.

Observational Study
174
Sometimes, experiments may not be feasible or practical.
● Example 1:
We want to test which “Add to Cart” button may lead to more Monthly-Active-Users (MAUs).

Observational Study
175
Sometimes, experiments may not be feasible or practical.
● Example 2:
We want to test which search ranking algorithm may lead to higher Year-Over-Year Changes of
user search sessions.

Observational Study
176
Experimentable Non-Experimentable

Causal Inference
177
Statistical Relationship
● Emerging topics between statistics and machine learning
● Well grounded theory for classic cases
● Easy for simple cases
● Not well studied in a lot of online settings
● Diﬃcult for complex scenarios
[1] David Sontag and Uri Shalit. Causal Inference for Observational Studies. ICML 2016 Tutorial.
[3] Lihong Li, Jin Young Kim and Imed Zitouni. Toward Predicting the Outcome of an A/B Experiment for Search Relevance. WSDM 2015.

Experiments v.s. Observational Study
178
Summary
● Run experiments as much as possible.
● Understand experimentable and non-experimentable.

Experiments v.s. Observational Study
179
Summary
● Run experiments as much as possible.
● Understand experimentable and non-experimentable.
● Bias: almost always indicates temporal, spatial and population sampling.
● Conclusions: almost always needs inference.

Metrics, Evaluation and Experiments
180
The relationships between metrics, evaluation and experiments
● Requiring certain user behaviors
○ e.g., NDCG, AUC, Precision, Recall,...

181
● Decomposition assumption
○ e.g., Conversion Rate, Click-Through-Rate,...

182
● Decomposition assumption
○ e.g., Conversion Rate, Click-Through-Rate,...
● Naturally missing/partial data
○ e.g., Dwell-time, View, Scroll,...

Online Learning
Multi-armed Bandits
Reinforcement Learning
183

184
● Have a clear objective/reward/utility/loss
● Emphasize on Maximization/Minimization
● Three classes of Automatic Optimization techniques
○ Online Learning/Optimization
○ Multi-armed Bandit
○ Reinforcement Learning

Online Learning
185
● The learner’s ultimate goal is to minimize the cumulative loss suﬀered
along its run.
● Theoretical analysis is around Regret Minimization.

Online Learning
186
H. Brendan McMahan, Gary Holt, D. Sculley, Michael Young, Dietmar Ebner, Julian Grady, Lan Nie, Todd Phillips, Eugene Davydov, Daniel Golovin, Sharat
Chikkerur, Dan Liu, Martin Wattenberg, Arnar Mar Hrafnkelsson Tom Boulos, and Jeremy Kubica. Ad click prediction: a view from the trenches. KDD 2013.

Online Learning
187
Online Learning
● Easy to understand and implement.
● Do not have a notion of multiple competing hypotheses
● In general, do not know how good/bad
[1] Elad Hazan. Introduction to Online Convex Optimization. Foundations and Trends® in Optimization: Vol. 2:
No. 3-4, 2016.
[2] Shai Shalev-Shwartz. Online Learning and Online Convex Optimization. Foundations and Trends® in
Machine Learning: Vol. 4: No. 2, 2012.

Multi-armed Bandits
188
● The learner’s ultimate goal is to maximize the cumulative reward along its
run.
● Theoretical analysis is around Regret Minimization.

Multi-armed Bandits
189
How to Online Optimize User Intra-Session Metrics and Inter-Session Metrics
Qingyun Wu, Hongning Wang, Liangjie Hong, and Yue Shi. Returning is Believing: Optimizing Long-term User Engagement in
Recommender Systems. In CIKM 2017. ACM, New York, NY, USA, 1927-1936.

Multi-armed Bandits
190
● Most algorithms focus on intra-session eﬀects (e.g., clicks, dwell, etc.).
[1] Abhinandan S. Das, Mayur Datar, Ashutosh Garg, and Shyam Rajaram. Google news personalization: scalable online
collaborative ﬁltering. In WWW 2007. ACM, New York, NY, USA, 271-280.
[2] Yehuda Koren, Robert Bell and Chris Volinsky. Matrix Factorization Techniques for Recommender Systems. Computer
42(8):2009.

Multi-armed Bandits
191
● Most algorithms focus on intra-session eﬀects (e.g., clicks, dwell, etc.).
[1] Abhinandan S. Das, Mayur Datar, Ashutosh Garg, and Shyam Rajaram. Google news personalization: scalable online
collaborative ﬁltering. In WWW 2007. ACM, New York, NY, USA, 271-280.
[2] Yehuda Koren, Robert Bell, and Chris Volinsky. Matrix Factorization Techniques for Recommender Systems. Computer
42(8):2009.
● Users may leave because of boredom from popular items.
Komal Kapoor, Karthik Subbian, Jaideep Srivastava, and Paul Schrater. Just in Time Recommendations: Modeling the Dynamics of
Boredom in Activity Streams. In WSDM 2015. ACM, New York, NY, USA, 233-242.

Multi-armed Bandits
192
● Users may have high immediate rewards but accumate linear regret after they leave.
● Predict a user’s immediate reward, but also project it onto future clicks, making
recommendation decisions dependent over time.
● Rapid change of environment requires this kind of decisions online.

Multi-armed Bandits
193
Some more related work about modeling users’ post-click behaviors:
[1] Nicola Barbieri, Fabrizio Silvestri and Mounia Lalmas. Improving Post-Click User Engagement on Native Ads via Survival Analysis.
WWW 2016.
[2] Mounia Lalmas, Jane.e Lehmann, Guy Shaked, Fabrizio Silvestri and Gabriele Tolomei. Promoting Positive Post-Click Experience for
In-Stream Yahoo Gemini Users. KDD Industry Track 2015.
[3] Nan Du, Yichen Wang, Niao He, Jimeng Sun and Le Song. Time-Sensitive Recommendation From Recurrent User Activities. NIPS
2015.
[4] Komal Kapoor, Mingxuan Sun, Jaideep Srivastava and Tao Ye. A Hazard Based Approach to User Return Time Prediction. KDD 2014.

Multi-armed Bandits
194
Balance between
1. Maximize immediate reward of the recommendation

Multi-armed Bandits
195
Balance between
2. Explore other possibilities to improve model estimation.

Multi-armed Bandits
196
Balance between
2. Explore other possibilities to improve model estimation.
3. Maximize expected future reward by keeping users in the system.
To maximize the cumulative reward over time, the system has to make users click more and return
more often.

Multi-armed Bandits
197
Main Idea

Multi-armed Bandits
198
Main Idea
● Model how likely an item would yield an immediate click:
[1] At iteration i, if we recommend item ai
, how likely it is going to be clicked by user u.

Multi-armed Bandits
199
Main Idea
● Model future visits after seeing this item and their expected clicks:
[2] At iteration i+1, what do we recommend.
[3] How that decision would impact the click behavior at i+1
[4] Future return probability at i+2, and
So on…

Multi-armed Bandits
200
Main Idea
● Model future visits after seeing this item and their expected clicks:
[2] At iteration i+1, what do we recommend.
[3] How that decision would impact the click behavior at i+1
[4] Future return probability at i+2, and
So on…
Can be formulated in a reinforcement learning setting.

Multi-armed Bandits
201
A Major Challenge:
future candidate pool undeﬁned, thus standard reinforcement learning can’t apply.

Multi-armed Bandits
202
A Major Challenge:
future candidate pool undeﬁned, thus standard reinforcement learning can’t apply.
Need approximations.

Multi-armed Bandits
203
Approximations

Multi-armed Bandits
204
Approximations
1. Future clicks depend on users. (Strong? or not)

Multi-armed Bandits
205
Approximations
2. Only model ﬁnite steps in future, or even just one step ahead.

Multi-armed Bandits
206
Approximations
3. Only model whether the user return in a ﬁnite horizon.

Multi-armed Bandits
207
Approximations
3. Only model whether the user return in a ﬁnite horizon.
New Objective:

Multi-armed Bandits
208
Model Summary
1. Use Generalized Linear Model (Bernoulli) to model how likely a user u would click on an item
ai
at iteration i.

Multi-armed Bandits
209
Model Summary
ai
at iteration i.
2. Use Moving Average to model a user u’s marginal click probability.

Multi-armed Bandits
210
Model Summary
ai
at iteration i.
3. Use Generalized Linear Model (Exponential) to model a user u’s return time intervals.

Multi-armed Bandits
211
Model Summary
ai
at iteration i.
4. Use Upper Conﬁdence Bound (UCB) on top of [1-3].

Multi-armed Bandits
212
Model Summary
ai
at iteration i.
4. Use Upper Conﬁdence Bound (UCB) on top of [1-3].
Note that both [1] and [3]’s coeﬃcients are personalized.

Multi-armed Bandits
213

Multi-armed Bandits
214

Multi-armed Bandits
215
Simulations
1. Type 1: items with high click probability but short expected return time;
2. Type 2: items with high click probability but long expected return time;
3. Type 3: items with low click probability but short expected return time;
4. Type 4: items with low click probability and long expected return time.

Multi-armed Bandits
216
Simulations

Multi-armed Bandits
217
Simulations

Multi-armed Bandits
218
Simulations

Multi-armed Bandits
219
Real-World Dataset
● Collect 4 weeks of data from Yahoo news portal.
● Reduce features into 23 by PCA.
● Sessionized the data by 30 mins.
● Return time is computed by time interval between two sessions.
● Total:
-- 18,882 users,
-- 188,384 articles
-- 9,984,879 logged events, and
-- 1,123,583 sessions.

Multi-armed Bandits
220
Real-World Dataset

Multi-armed Bandits
221
Real-World Dataset: Evaluation
● Cumulative clicks over Time
● Click-through Rate (CTR)
● Average Return Time
● Return Rate
● Improved User Ratio
● No return Count

Multi-armed Bandits
222

Multi-armed Bandits
223
Real-World Dataset: Word Cloud

Multi-armed Bandits
224
Multi-armed Bandits
● Easy to understand and implement.
● Challenge to scale to millions/billions.
[1] Lihong Li, Wei Chu, John Langford and Robert Schapire. A contextual Bandit Approach to Personalized News Article
Recommendation. WWW 2010.

226
Early Attempts:
[1] Xiangyu Zhao, Long Xia, Yihong Zhao, Dawei Yin and Jiliang Tang. Model-Based Reinforcement Learning for Whole-Chain
Recommendations. CoRRabs/1902.03987, 2019.
[2] Lixin Zou, Long Xia, Zhuoye Ding, Jiaxing Song, Weidong Liu and Dawei Yin. Reinforcement Learning to Optimize Long-term User
Engagement in Recommender Systems. CoRR abs/1902.05570, 2019.

227
● Intuitive to understand and diﬃcult to implement.
● Challenge to scale to millions/billions.
[1] Xiangyu Zhao, Long Xia, Liang Zhang, Zhuoye Ding, Dawei Yin and Jiliang Tang. Deep Reinforcement Learning for Page-wise
Recommendations. RecSys 2018.
[2] Xiangyu Zhao, Liang Zhang, Zhuoye Ding, Long Xia, Jiliang Tang and Dawei Yin. Recommendations with Negative Feedback via Pairwise
Deep Reinforcement Learning. KDD 2018.
[3] Di Wu, Xiujun Chen, Xun Yang, Hao Wang, Qing Tan, Xiaoxun Zhang, Jian Xu and Kun Gai. Budget Constrained Bidding by Model-free
Reinforcement Learning in Display Advertising. CIKM 2018.

Two Main Camps of Optimization
229
- e.g., Online Learning, Multi-armed Bandits, Reinforcement Learning…

Two Main Camps of Optimization
230
Pros: Have deep roots in Statistics, Economics and etc
Cons: Concerning with ATE (or similar) and slow & costly to operate
Pros: Have deep roots in ML, Control and etc.
Cons: Concerning with maximizing/minimizing rewards/loss
Combining Two Camps
Can we maximize/minimize rewards while concerning ATE?

Combining Two Camps
231
Two Challenges for Standard A/B Testing:
● Time Cost
Product evolution pushes its shareholders to consistently monitor results from online A/B
experiments, which usually invites peeking and altering experimental designs as data collected.
● Opportunity Cost
A static test usually entails a static allocation of users into diﬀerent variants, which prevents an
immediate roll-out of the better version to larger audience or risks of alienating users who may
suﬀer from a bad experience.
Nianqiao Ju, Diane Hu, Adam Henderson and Liangjie Hong. A Sequential Test for Selecting the Better Variant: Online A/B testing,
Adaptive Allocation, and Continuous Monitoring. WSDM 2019.

Combining Two Camps
232
Contributions:
1. Propose an imputed sequential Girshick test for Bernoulli model with a fixed allocation.
2. Use simulations to demonstrate that the test procedure also applies to an adaptive allocation
such as Thompson sampling with a small error inflation.
3. Conduct a regret analysis of A/B tests from the Multi-armed Bandit (MAB) perspective.
4. Conduct extensive studies including simulations as well as experiments on an industry-scale
experiment, demonstrating the effectiveness of the proposed method and offering practical
considerations.

Combining Two Camps
233

Combining Two Camps
234

Combining Two Camps
235
Nianqiao Ju, Diane Hu, Adam Henderson, and Liangjie Hong. A Sequential Test for Selecting the Better Variant: Online A/B testing,
Adaptive Allocation, and Continuous Monitoring. In WSDM 2019.

Combining Two Camps
236

Combining Two Camps
237
● Sequential Test from Statistics + Multi-armed Bandit from ML
● Challenges:
○ Biased v.s. Unbiased
○ Deriving valid p-values
○ Provide practical beneﬁts
● Emerging Topics
[1] Alex Deng. Objective bayesian two sample hypothesis testing for online controlled experiments. WWW 2015.
[2] Alex Deng, Jiannan Lu and Shouyuan Chen. Continuous monitoring of A/B tests without pain: Optional stopping in Bayesian testing.
DSAA 2016.
[3] Ramesh Johari, Pete Koomen, Leonid Pekelis, and David Walsh. Peeking at A/B Tests: Why It Matters, and What to Do About It. KDD
2017.
[4] Steven L Scott. Multi-armed bandit experiments in the online service economy. Applied Stochastic Models in Business and Industry
31, 1:2015.
[5] Minyong R Lee and Milan Shen. Winner’s Curse: Bias Estimation for Total Eﬀects of Features in Online Controlled Experiments. KDD
2018.

Concluding remarks and
future direction
238

Metrics: Concluding Remarks
239
Opportunities:
How to systematically discover new metrics, through for example the
quantiﬁcation of users’ holistic feelings or by learning them.
How to use mixed methods to elicit hypotheses of what engagement means
and inspire metric development.
How to consider non engagement metrics (e.g diversity, revenue) when
measuring online engagement.

Metrics: Concluding Remarks
240
Challenges:
How to account for bias when measuring and optimizing for given metrics.
How to account for intent, segmentation and diversity.
How to incorporate negative signals.

Optimizations: Concluding Remarks
241
Opportunities:
Emerging topics of utilizing and combining techniques, methodologies and
ideas from Machine Learning, Statistics, Economics, Control Theory and more
ﬁelds.

Optimizations: Concluding Remarks
242
Opportunities:
Emerging topics of utilizing and combining techniques, methodologies and
ideas from Machine Learning, Statistics, Economics, Control Theory and more
ﬁelds.
Challenges:
● Still early stage, a lot of heuristics, require more active research
● Costly to practice and involve institution commitments
● Optimizing for multiple (possibly competing) metrics
● Optimize under FATE (Fairness, Accountability, Transparency, and Ethics)

Thank you
Website:https://onlineuserengagement.github.io/
243

Tutorial on Online User Engagement: Metrics and Optimization

More Related Content

Similar to Tutorial on Online User Engagement: Metrics and Optimization (20)

More from Mounia Lalmas-Roelleke (20)

Recently uploaded (20)

Tutorial on Online User Engagement: Metrics and Optimization