Promise of Online Language Learning Programs: Myth or Reality?

Gillian Lord
University of Florida
glord@ufl.edu
The Promise of Online
Language Learning Programs:
Myth or Reality?

• Powerful, omnipresent marketing by
stand-alone online language programs
• Hyperbolic claims to be the
only / fastest / easiest / best/ most ___…
way to learn a language
• K-12 (and higher ed!) programs
feeling threatened…
Why this topic?

Why Rosetta Stone?
SOURCE:“CrowdsourcingDrivesLatestDisruptiveTechin
OnlineLanguageLearning.”Forbes.March24,2015.
“Rosetta Stone … spent $98.5
million on advertising in 2011,
up from $70.5 million in 2010,
according to Kantar Media”
www.nytimes.com/2012/06/20/business/media/rosetta
-stone-ads-emphasize-fun-not-efficiency.html

GOAL OF PRESENTATION:
• Provide linguistic evidence to be able to assess
these kinds of marketing claims
• Offer multiple data sources to consider the
advantages and disadvantages of this kind of
program
• Take as a case study example Rosetta Stone for
learning (Spanish as) a second/foreign language
or

1. Professional assessments
2. (Previous) Empirical studies
3. Student attitudes (a priori)
4. Perceived quality of materials
5. Affective factors
6. Outcomes: (“Does it work?”)
a) Self-perceived communicative abilities
b) Quantitative measures of communicative abilities
c) Qualitative measures of communicative abilities
How to assess the program?

University of Florida students enrolled in Beginning
Spanish 1 (avg. age = 20)
• L1 English
• No other L2 proficiency (beyond h.s. requirement)
• Course designed for those with NO prior Spanish
instruction
3 environments:
• Classroom (C): N=4
• Rosetta Stone (RS): N=4
• Classroom+Rosetta Stone (RS+C): N=4
My data – Participants
Original population had
20-25 participants in
each of the three groups.

Classroom
• In-tact section
• Followed regular
syllabus with
standard materials
My data – 3 environments
Rosetta Stone
• Self-selected (required by
IRB)
• Not required to attend any
regular class
• Used RS package
(“Conversational
Spanish”):
– 16-week course designed
to cover material
comparable to a face-to-
face beginning class
– 6 units of Rosetta Stone®
Version 4 TOTALe®
Spanish, each has 4
lessons [Level 1, half of
Level 2]
– Minimum of 6 Rosetta
StudioTM sessions
– Minimum of 8 hours in
Rosetta WorldTM
– Monitoring of program
access and time on task
RS + Class
• In-tact section of Beginning
Spanish class
• Same instructor as
Classroom group
• Used Rosetta Stone
materials as their textbook
– (including all features
described for RS
group)

Quantitative Data
• Portion of Spanish CLEP test (grammatical competence)
• Versant Proficiency Test in Spanish (oral proficiency)
• Attitude survey (Likert, style, pre- and post)
• Acoustic analysis of vowel production in Spanish
interviews
o 10 tokens of /e/ extracted from each participant’s interview.
 Tokens separated for male and female
 Each formant measured at midpoint using Praat
 F1 and F2 extracted using a script and compared to
standard native values
Data used here (1)

Qualitative Data
• Fluency analysis of Spanish interviews
o Total number of words spoken
o Number of Spanish words; Number of English words
o Number of dysfluencies
o Lexical density (number of unique Spanish words)
o Number of fillers/non-lexical items
• Content analysis of English interviews
o Affective factors
o Reactions to instructional materials
o Student perceived learning
o Language communication and use
o Other specific language problem or comment
Data used here (2)

PROFESSIONAL
ASSESSMENTS OF THE
ROSETTA STONE PROGRAM
Criterion 1:

• Godwin-Jones (2007, 2009)
o Traditional computer-based training programs often informed by
technology specialists, not SLA specialists
o Rosetta Stone provides visualization feedback for specific sounds
(pronunciation); mostly accurate
• Lafford & Sykes (2007)
o Evaluate if programs provide the tools necessary for effective language
learning, based on features that research has shown to be important
(interaction, relevant contextualization of language, etc.)
o “… these products do not incorporate a number of the [necessary]
research-based insights (e.g., the need for culturally authentic, task-
based activities) that informed SLA scholars might have given them.”
Professional Assessments (1)

• Santos (2011)
o Lack of context
o General inability to respond to spontaneous student speech
o What Rosetta Stone calls interaction is “a rather poor and limited
version of what one would encounter in a real-life conversation”
• DeWaard (2013)
o Based on personal experience, professional reactions
o “Not a viable replacement of current instruction at the
postsecondary level”
Professional Assessments (2)

EMPIRICAL STUDIES INTO
THE OUTCOMES OF THE
ROSETTA STONE PROGRAM
Criterion 2:

• Vesselinov (2009)
o Commissioned by Rosetta Stone; RS beginning users
demonstrated increased knowledge of the language after 55
hours of use.
• Nielson (2011)
o Self-study programs in workplace; some success among
committed users, but extreme attrition.
• Stevenson & Liu (2010)
o Use of social interaction in web-based language learning tools;
lack of ability to engage learners in true interaction; users do not
take advantage of network opportunities.
Empirical studies

STUDENT ATTITUDES
TOWARDS THE ROSETTA
STONE PROGRAM
Criterion 3:

• Why did you volunteer for the Rosetta Stone class-
replacement option?
o I heard a lot of good things about RS and wanted to
try it.
o Can better manage my time and schedule and move
more at my own pace without dealing with class.
o Sounded more beneficial.
o I was going to use my own to supplement education
anyway.
Background questionnaire

PERCEIVED QUALITY OF THE
ROSETTA STONE MATERIALS
Criterion 4:

• Comments from English interviews coded for
mentions of the Rosetta Stone materials
o 42/181 comments in RS group
o 80/124 comments in RS+C group
o Difficult to classify as categorically positive or
negative
• Main themes:
o Technology [Flexibility, Ease of use, Glitches]
o Approach to language learning
Student Impressions of
Materials

• Like you’re able to kind of
do it like on your own time,
you know, I’m not like
restricted.
• It’s nice not to go to class.
• I have always preferred to
learn language, like, on my
own.
• It’s more flexible with my
schedule.
Sample student comments
(Flexibility)

• Because it’s a lot of visual stuff,
and I feel like I’m a visual
learner.
• It’s pretty easy to use.
• World is good, I do the games.
• Oh, and you can do it on your
iPad … so I’d do that a lot, like
take it to people’s houses, and
they’ll all be like watching TV,
and I would be like, doing my
Rosetta Stone.
• I like how it’s like uh, more like
a game, so I’m more willing to
actually do it…
(Ease of Use)
• It was just kind of a lot harder [to
use] than I expected.
• I just didn’t show what words I
needed to use before it.
• I [don’t like] the lack of human
interaction.
• Sometimes it’ll show the person
speaking, and sometimes it’ll say like
he or she, and sometimes it’ll be I.
And I couldn’t tell the difference.
• …but it is tedious, a lot of it.

• I’m still having problems
with the computer.
• I can’t get the
microphone to work…
• I was doing my Studio
session and… I had no
audio, like, I could hear
them, but they couldn’t
hear me the whole time.
(Tech glitches)

• I feel like it’s more like how you
naturally learn the language
instead of like, “These are … your
vocabulary words this week”.
• I think the Studio session was the
most interesting thing.
• Speaking it for sure [is hardest],
because when you see it on paper
it’s a little bit… but when it gives
you four options, it’s easier to be
like, it’s one of these four.
• The program is really good with
like teaching like vocabulary.
• With like vocabulary, it’s like really
good, and you get by.
Student Impressions
(Language learning in general)
• It’s just like the grammar, and how to
like, put it together.
• You can’t ask questions.
• You don’t get any writing, and then all
of a sudden there’s one writing thing.
• You have that whole grammar and
conjugation issue on Rosetta, because
they don’t really explain it.
• You really need to have communication
with a real person.
• I would enjoy getting more grammar
lessons … just to get a foundation of
knowledge, then building up on that.
• Rosetta Stone doesn’t like, I don’t know,
hammer it down for you like they
would in a normal grammar class.

AFFECTIVE FACTORS AND
EMOTIONAL REACTIONS TO
THE ROSETTA STONE
PROGRAM
Criterion 5:

• Comments from English interviews coded for any
mention of affect/emotion
o Related to using the materials, learning, etc. and how
the learners felt in that respect
• 33/181 comments in RS group
o 8 positive towards RS, 8 negative, 17 unrelated
• 24/123 comments in RS+C group
o 2 positive towards RS, 9 negative, 13 unrelated
Affective factors

• I don’t have a problem
with staying focused on
my computer at all.
• I like not having a
textbook to worry
about.
• I like it a lot.
Student reactions/emotions
• I’m kind of struggling.
• I didn’t like the stories.
• I’m always just frustrated
because I’m like, I don’t
really understand it.
• I feel like it should be
more structured.

LINGUISTIC OUTCOMES OF
THE ROSETTA STONE
PROGRAM
Criterion 6:

• Comments from English interviews coded for any
mention of perceived learning or outcomes
• 34/181 comments in RS group
o 9 positive assessments, 22 negative, 3 unrelated to
materials
• 14/123 comments in RS+C group
o 8 positive assessments, 3 negative, 3 unrelated
Perceived learning

• I understood like the vocab.
• I feel like I’m slowly learning
and progressing to learn the
language.
• I do like the studio sessions…
When I do do them, like I feel
like it helps me more.
• I can understand almost
always what is being said.
• Learning a language just like
on your computer as opposed
to like in person can actually
be effective.
Perceived learning
• Rosetta Stone doesn’t give you too
much, like, actual instruction so you
don’t learn.
• I don’t know how to use words in
Spanish.
• I’m just kind of like nervous about
going back to class.
• I don’t feel as confident as I did at
that last meeting, honestly.
• We didn’t ever have to like memorize
the conjugations, so I don’t know
them, so I can’t say that it was super
effective.
• It was just kind of a lot harder than I
expected.

Significant decrease among RS on item #11:
“Interacting via chat or telephone is comparable
to interacting face-to-face.”
Significant increase among RS and RS+C on item #19:
“I would prefer to learn a language on my own
time and at my own pace than in a group or
classroom setting.”
Perceived learning
(pre-post changes on relevant
items from attitude survey)

THE ROSETTA STONE
PROGRAM
b) Empirical measurements of communicative abilities
Criterion 6:

CLEP test
Average scores (converted to %)
38.65 39.17
47.50
0.00
10.00
20.00
30.00
40.00
50.00
60.00
70.00
80.00
90.00
100.00
Classroom Rosetta Stone RS+class
p = 0.165
RS = Classroom = RS+Class

27.08 26.25
20.00
0.00
10.00
20.00
30.00
40.00
50.00
60.00
70.00
80.00
90.00
100.00
Classroom Rosetta Stone RS + Class
Versant test
Average scores (converted to %)
p = 0.620
RS = Classroom = RS+Class

• No significant difference
between groups or
test times.
• Classroom males show
slight trend toward
more native-like F1
values in final interview.
Pronunciation
Acoustic analysis of vowel /e/
RS = Classroom = RS+Class ?

THE ROSETTA STONE
PROGRAM
c) Discourse analysis of communicative abilities
Criterion 6:

INTERVIEWER: Cuéntame, ¿qué te gusta hacer en tu tiempo libre, o los fines de semana?
SL: Repitas, please.
INTERVIEWER: ¿Qué te gusta hacer?
SL: Qué te gusta hacer…
INTERVIEWER: ¿Te gusta ir a películas? ¿Te gusta escuchar música?
SL: Uh, ¿fin de semana?
INTERVIEWER: Sí.
SL: Uh, sí. En fin de semana, yo… yo estudio, uh, mucho.
INTERVIEWER: ¿Sí?
SL: Uh, para mis exámenes. Sí. Yo tengo muchos examines en química orgánica, biología, y laboratorio. Uh, sí. Mucho, uh… no,
muy ocupado. So, no películas, no, uh, deportes.
INTERVIEWER: ¿Cuál fue la última película que viste?
SL: Cuál te…
INTERVIEWER: La última vez, the last time, que viste una película.
SL: Phew… Hmm. Let’s see… dos menses.
INTERVIEWER: Meses, mhm.
SL: Meses. Ago. ¿Cómo se dice “ago”?
INTERVIEWER: Hace. Hace dos meses.
SL: Hace, sí.
INTERVIEWER: Wow.
SL: Yo no… yo no veo muchas películas en Gainesville.
INTERVIEWER: ¿Qué película fue esa, hace dos meses? ¿Cómo se llamaba?
SL: Uh, el pelí—la película… ¿cómo se dice “was”?
INTERVIEWER: Era, o fue.
SL: Era. La película era…
INTERVIEWER: ¿No te acuerdas?
SL: Yo no… sí.
Final interview - Classroom

INTERVIEWER: Mhm, ¿y qué haces en Gainesville?
SH: Um… you’re going to have to forgive me, my mind’s like blown… Um, yo estoy estudiar.
INTERVIEWER: ¿Tú estudias? ¿Y qué más?
SH: Yo trabajo en un restaurante de Dragonfly.
INTERVIEWER: Y, ¿con mucha frecuencia, vas de compras?
SH: Yo no entendí, repetirlo, por favor.
INTERVIEWER: ¿Con mucha frecuencia, vas de compras? “Ir de compras” significa go shopping.
SH: All right, say that one more time, please.
INTERVIEWER: ¿Con mucha frecuencia, vas de compras?
SH: Uh, no, uh, no voy a… what did you, how did you say “to go shopping”?
INTERVIEWER: Ir de compras.
SH: No voy de compras.
INTERVIEWER: Y, ¿qué vas a hacer este verano?
SH: Este verano, yo voy a visitar Brazil.
INTERVIEWER: Vas a visitar Brazil, y ¿vas a estudiar en Brazil?
SH: No, um, yo voy a trabajar en Brazil.
INTERVIEWER: Y, em, ¿qué más vas a hacer en Brazil? ¿Vas a leer, vas a jugar deportes?
SH: What am I going to do in Brazil? I thought I just answered that.
INTERVIEWER: ¿Solo trabajar?
SH: I don’t know, I’m going on a missions trip, I don’t know how to express that in Spanish, but…
INTERVIEWER: Pues, buena suerte, muchas gracias.
Final interview – Rosetta Stone

Ratio of L1/L2 words
0.26
0.83
0.68
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
Ratio of English-to-Spanish words used, by group
Control Average
RS + class Average
Rosetta Stone Average
0 = no English words produced
1 = 1 English word produced for every Spanish word

Assistance requests
2.06
1.03
0.11
1.75
0.61
3.24
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
# Clarification requests in Spanish # Clarification requests in English
Average # of clarification / assistance requests by group
Control Average RS + class Average Rosetta Stone Average

1. Professional assessments
2. Empirical studies
3. Student attitudes (a priori)
4. Perceived quality of materials
5. Affective factors
6. Outcomes:
b) Empirical measurements of communicative
abilities
c) Discourse analysis reflecting communicative
abilities
Myth or Reality?

GROUP
Completion
Rate
Average
Score
Total Course
Usage (hours)
Total Class
Time (hours)
Classroom 96.99% 90.77% 70.00 39.00
RS+C 93.67% 98.63% 32.81 37.25
RS 97.67% 95.88% 30.69 NA
What about time on task?

• Further professional assessments of these types of
programs, as new tools and new features are
developed
• Continuing analysis of current data
• Empirical studies of outcomes and effectiveness
o Attitudes and reactions
o Linguistic outcomes
o Larger, more diverse populations
o Consider culture, sociocultural aspects
1. More research is needed.

In certain circumstances:
o Introduction and exposure to new languages
o Refresh skills previously acquired
o Excellent vocabulary presentation and practice
o Flexible for varied populations
o Highly motivated/diligent learners more apt to
make progress
… and are undoubtedly better than nothing!
2. Online (stand-alone)
programs can be effective.

Based on multiple data sources, programs like Rosetta
Stone do not appear to be necessarily any more …
than other methods, such as common classroom-based
approaches, immersion programs, etc.
3. BUT…

4. As language experts, then,
we should…
 … not believe everything the ads promise!
o There is a significant divide between what the ads
promise and what the programs seem capable of
 … keep an open mind.
o New tools and capabilities emerge daily
o Knowledge of another language is always valuable
 … work to develop criteria for assessing these
programs.
o [at tomorrow’s workshop!]

Special thanks to
• UF College of Liberal Arts & Sciences
• UF CLAS Humanities Scholarship
Enhancement Fund
• Laura Bradley, Lisa Frumkes
(Rosetta Stone)
• Caroline Reist, Brandon Shufelt, Keegan
Storrs, Diana Wade (RAs)
• Carlos Enrique Ibarra (statistics)
Thank you
glord@ufl.edu

DeWaard, L. (2013). “Is Rosetta Stone a viable option for L2 learning?” Forthcoming in ADFL Bulletin.
Godwin-Jones, R. (2007). “Emerging technologies; Tools and trends in self-paced language
instruction. Language Learning and Technology,” 11(2), 10-17. Retrieved 26 September 2012 from
http://llt.msu.edu/vol11num2/emerging/
Godwin-Jones, R. (2009). “Emerging technologies: Speech tools and technologies. Language Learning
and Technology,” 13(3), 4-11. Retrieved 26 September 2012 from
http://llt.msu.edu/vol13num3/emerging.pdf
Lafford, B. & Sykes, J. (2007). “Entre dicho y hecho …: An assessment of the application of research
from second language acquisition and related fields to the creation of Spanish CALL materials for
lexical acquisition.” CALICO Journal, 24(3), 427-529.
Nielson, K. B. (2011). “Self-study with language learning software in the workplace.” Language
Learning and Technology, 15(3), 110-129. Retrieved 26 September 2012 from
http://llt.msu.edu/issues/october2011/nielson.pdf
Santos, V. (2011). “Review of Rosetta Stone Portuguese (Brazil) levels 1, 2, & 3.”CALICO Journal, 29(1),
177-194.
Stevenson, M. P. & Liu, M. (2010). “Learning a language with web 2.0: Exploring the use of social
networking features of foreign language learning websites.” CALICO Journal, 27(2), 233-259
Vesselinov, Roumen. Measuring the Effectiveness of Rosetta Stone.
http://resources.rosettastone.com/CDN/us/pdfs/Measuring_the_Effectiveness_RS-5.pdf.
Works Cited

Promise of Online Language Learning Programs: Myth or Reality?

Instrument has been tested for reliability and validity
Versant proficiency test

Rosetta Stone interface
(vocabulary)

(grammar)

(pronunciation)

(World – “play”)

(World – “talk”)

(World – “explore”)

(Studio)

Promise of Online Language Learning Programs: Myth or Reality?

More Related Content

Similar to Promise of Online Language Learning Programs: Myth or Reality? (20)

More from Gillian Lord (20)

Recently uploaded (20)

Promise of Online Language Learning Programs: Myth or Reality?