Abstract image of a woman sitting on books and studying on a laptop

Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)

Lora Aroyo, profile picture
Lora Aroyo

The document discusses the limitations of traditional models of truth in AI and emphasizes the importance of embracing diverse human perspectives through crowdsourcing, known as 'crowdtruth.' It highlights how collective decision-making can outperform individual experts when analyzing complex issues and suggests that increasing disagreement among annotators can lead to richer interpretations and better understanding. Ultimately, the approach advocates for a paradigm shift in how truth is perceived in data annotation and AI development.

Technology
1 of 82
Downloaded 28 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
Web & Media Group http://lora-aroyo.org @laroyo Truth is a Lie Rules & Semantics from Crowd Perspectives
Web & Media Group http://lora-aroyo.org @laroyo “in embracing the diversity of human beings, we will find a surer way to true happiness” Malcolm Gladwell, 2006 in his TED talk “Choice, happiness and spaghetti sauce”
Web & Media Group http://lora-aroyo.org @laroyo “If we treat human brains as processors in a distributed system, each can perform a small part of a massive computation” Luis von Ahn, 2006 “Games with a Purpose”, Computer Magazine
Web & Media Group http://lora-aroyo.org @laroyo “The challenge for Watson was to answer questions posed in every nuance of natural language. The first step in accomplishing this was to have a lot of questions and their correct answers. Any cognitive system must take this first step.” Chris Welty, 2014 “Cognitive Computing and the Future”, 1st Cognitive Computing Forum
Web & Media Group http://lora-aroyo.org @laroyo Take Home Message True  and  False  is  not  enough,     There  is  diversity  in  human  interpreta3on     CrowdTruth  introduces  a  spa3al  representa3on  of   meaning  that  through  crowdsourcing     harnesses  disagreement     Using  CrowdTruth   untrained  workers  can  be  just  as  reliable  as     highly  trained  experts  
Web & Media Group http://lora-aroyo.org @laroyo Is this song …. Passionate Rousing Confident Boisterous Rowdy Literate Poignant Wistful Bittersweet Autumnal Brooding Rollicking Cheerful Fun Sweet Amiable Good-natured Humorous Silly Campy Whimsical Witty Wry Aggressive Fiery Tense Anxious Intense Volatile
Web & Media Group http://lora-aroyo.org @laroyo ?Passionate Rousing Confident Boisterous Rowdy Literate Poignant Wistful Bittersweet Autumnal Brooding Rollicking Cheerful Fun Sweet Amiable Good-natured Humorous Silly Campy Whimsical Witty Wry Aggressive Fiery Tense Anxious Intense Volatile Is this song ….
Web & Media Group http://lora-aroyo.org @laroyo ?- category(“That’s not my name”, x) ?Passionate Rousing Confident Boisterous Rowdy Literate Poignant Wistful Bittersweet Autumnal Brooding Rollicking Cheerful Fun Sweet Amiable Good-natured Humorous Silly Campy Whimsical Witty Wry Aggressive Fiery Tense Anxious Intense Volatile (Lee and Hu 2012)
Web & Media Group http://lora-aroyo.org @laroyo Patients with TYPHUS who were given ANTIBIOTICS exhibited several side-effects. With ANTIBIOTICS in short supply, DDT was used during World War II to control the insect vectors of TYPHUS. ANTIBIOTICS are the first line treatment for indications of TYPHUS. Does this sentence express TREATS(Antibiotics, Typhus)?
Web & Media Group http://lora-aroyo.org @laroyo TREATS(Antibiotics, Typhus) Is this True?
Web & Media Group http://lora-aroyo.org @laroyo What is the treatment for typhus? Antibiotic therapy is recommended for both endemic and epidemic typhus infections because early treatment with antibiotics (for example, azithromycin, doxycycline, tetracycline, or chloramphenicol) can cure most people infected with the bacteria. Consultation with an infectious-disease expert is advised especially if epidemic typhus or typhus in pregnant females is diagnosed. Delays in treatment may allow renal, lung, or nervous system problems to develop. Some patients, especially the elderly, may die. TREATS(Antibiotics, Typhus) Is this True?
Web & Media Group http://lora-aroyo.org @laroyo TREATS(Corticosteroids, Lupus) Is this True? What is the treatment for lupus? In more severe cases of Lupus, medications that modulate the immune system (mainly corticosteroids, immunosuppressants) are used to control the disease and prevent recurrence of symptoms (known as flares). Depending on the dosage, people who require steroids may develop Cushing's syndrome. This may subside if and when the large initial dosage is reduced, but long-term use of even low doses can cause elevated blood pressure and cataracts. Numerous new immunosuppressive drugs are being actively tested. Rather than suppressing the immune system nonspecifically, as corticosteroids do, they target the responses of individual [types of] immune cells.
Web & Media Group http://lora-aroyo.org @laroyo Human Interpretation “I will never understand people” “They are the worst”
Web & Media Group http://lora-aroyo.org @laroyo human subjectivity, ambiguity & uncertainty of expression are part of human semantics
Web & Media Group http://lora-aroyo.org @laroyo current practices in computer science (AI) are based on fallacies
Web & Media Group http://lora-aroyo.org @laroyo single truth
Web & Media Group http://lora-aroyo.org @laroyo experts rule
Web & Media Group http://lora-aroyo.org @laroyo disagreement is bad
Web & Media Group http://lora-aroyo.org @laroyo simplification of context this all results in
Web & Media Group http://lora-aroyo.org @laroyo
Web & Media Group http://lora-aroyo.org @laroyo this means our systems understand only simplified black & white world problem ..
Web & Media Group http://lora-aroyo.org @laroyo reality is complex diverse but …
Web & Media Group http://lora-aroyo.org @laroyo current gold standard practices are not adequate any more so...
Web & Media Group http://lora-aroyo.org @laroyo there is a gap we need to bridge and …
Web & Media Group http://lora-aroyo.org @laroyo interestingly …
Web & Media Group http://lora-aroyo.org @laroyo in humanities … diversity is well accepted
Web & Media Group http://lora-aroyo.org @laroyo in sciences … diversity is taboo
Web & Media Group http://lora-aroyo.org @laroyo despite numerous examples … string theory debate
Web & Media Group http://lora-aroyo.org @laroyo global warming debate and …
Web & Media Group http://lora-aroyo.org @laroyo evolutionary bio. debate gradualismpunctuated equilibrium and …
Web & Media Group http://lora-aroyo.org @laroyo What can we do? so …
Web & Media Group http://lora-aroyo.org @laroyo •  collective decisions of large groups of people •  a group of error-prone decision- makers can be surprisingly good at picking the best choice •  when thumbs up or thumbs down - the chance of picking the right answer needs to be > 50% •  the odds that a most of them will pick the right answer is greater than any of them will pick it on their own •  performance gets better as size grows 1785 Marquis de Condorcet “wisdom of crowds”
Web & Media Group http://lora-aroyo.org @laroyo • asked 787 people to guess the weight of an ox • none got the right answer • their collective guess was almost perfect 1906 Sir Francis Galton “wisdom of crowds”
Web & Media Group http://lora-aroyo.org @laroyo WWII Math Rosies 1942: Ballistics calculations and flight trajectories
Web & Media Group http://lora-aroyo.org @laroyo NASA’s Computer Room transcribe raw flight data from celluloid film & oscillograph paper
Web & Media Group http://lora-aroyo.org @laroyo “Who Wants to Be a Millionaire?” •  help from individual expert or audience poll •  majority of the audience right 91% of the time •  individuals right only 65% of the time “wisdom of crowds” 2004 James Surowiecki
Web & Media Group http://lora-aroyo.org @laroyo Crowdsourcing Human Computing
Web & Media Group http://lora-aroyo.org @laroyo the wise crowd diversity of opinion independent perspectives multitude of contexts gives the big picture James Surowiecki
Web & Media Group http://lora-aroyo.org @laroyo current practices in computer science (AI) are based on fallacies
Web & Media Group http://lora-aroyo.org @laroyo today’s real-world complex problem solving needs understanding of multiple perspectives current practices in computer science (AI) are based on fallacies
Web & Media Group http://lora-aroyo.org @laroyo single truth
Web & Media Group http://lora-aroyo.org @laroyo multiple truths single truth
Web & Media Group http://lora-aroyo.org @laroyo experts rule
Web & Media Group http://lora-aroyo.org @laroyo wisdom of crowds experts rule
Web & Media Group http://lora-aroyo.org @laroyo disagreement is bad
Web & Media Group http://lora-aroyo.org @laroyo disagreement is bad disagreement as signal
Web & Media Group http://lora-aroyo.org @laroyo “best collective decisions are result of disagreement, not consensus or compromise” James Surowiecki because …
Web & Media Group http://lora-aroyo.org @laroyo can we harness it?
Web & Media Group http://lora-aroyo.org @laroyo Is this song …. ? Passionate Rousing Confident Boisterous Rowdy Literate Poignant Wistful Bittersweet Autumnal Brooding Rollicking Cheerful Fun Sweet Amiable Good-natured Humorous Silly Campy Whimsical Witty Wry Aggressive Fiery Tense Anxious Intense Volatile
Web & Media Group http://lora-aroyo.org @laroyo If “One Truth” & “No Disagreement” Worker Mood-C1 Mood-C2 Mood-C3 Mood-C4 Mood-C5 W1 1 W2 1 W3 1 W4 1 W5 1 W6 1 W7 W8 W9 1 W10 1 Totals 1 3 1 2 1
Web & Media Group http://lora-aroyo.org @laroyo Worker Mood-C1 Mood-C2 Mood-C3 Mood-C4 Mood-C5 Other W1 1 1 1 W2 1 1 1 W3 1 1 1 W4 1 1 W5 1 1 W6 1 1 1 W7 1 1 1 W8 1 1 1 W9 1 1 W10 1 1 1 1 1 Totals 3 5 6 5 2 8 If “Many Truths” & “Disagreement”
Web & Media Group http://lora-aroyo.org @laroyo can indicate alternative interpretations Worker Mood-C1 Mood-C2 Mood-C3 Mood-C4 Mood-C5 Other W10 1 1 1 1 1 Totals 3 5 6 5 2 8 Disagreement as Signal can indicate ambiguity in the categorisation can indicate low quality workers
Web & Media Group http://lora-aroyo.org @laroyo Does this sentence express TREATS(Antibiotics, Typhus)? Patients with TYPHUS who were given ANTIBIOTICS exhibited several side-effects. With ANTIBIOTICS in short supply, DDT was used during World War II to control the insect vectors of TYPHUS. ANTIBIOTICS are the first line treatment for indications of TYPHUS. 95% 75% 50% Disagreement is signal here too
Web & Media Group http://lora-aroyo.org @laroyo Sentence Treat Prevent Cause Side Effect Diagnose S1 2 S2 2 S3 2 S4 2 S5 1 1 S6 2 S7 2 S8 1 1 S9 2 S10 2 If “One Truth”, “No Disagreement” & “Experts”
Web & Media Group http://lora-aroyo.org @laroyo •  we rejected 3 fallacies: one truth, experts rule and disagreement is bad •  by encouraging disagreement • we exposed a richer set of possibilities • that help in identifying, processing & understanding context so …
Web & Media Group http://lora-aroyo.org @laroyo •  One truth: data collection efforts assume one correct interpretation for every example •  All examples are created equal: ground truth treats all examples the same – either match the correct result or not •  Detailed guidelines help: if examples cause disagreement - add instructions to limit interpretations •  Disagreement is bad: increase quality of annotation data by reducing disagreement among the annotators •  One is enough: most of the annotated examples are evaluated by one person •  Experts are better: annotators with domain knowledge provide better annotations •  Once done, forever valid: annotations are not updated; new data not aligned with old 7 Myths myths directly influence the practice of collecting human annotated data; Need to be revised with a new theory of truth (CrowdTruth) Lora Aroyo, Chris Welty: Truth is a Lie: 7 Myths about Human Annotation, AI Magazine 2014.
Web & Media Group http://lora-aroyo.org @laroyo
Web & Media Group http://lora-aroyo.org @laroyo crowdtruth.org
Web & Media Group http://lora-aroyo.org @laroyo crowdtruth.org a new approach to understanding semantics harnessing the power & diversity of the crowd
Web & Media Group http://lora-aroyo.org @laroyo crowdtruth.org informs a vector space model of truth instead of a boolean, fuzzy or statistical model
Web & Media Group http://lora-aroyo.org @laroyo observerreferent sign has   triangle of reference (Ogden & Richards, 1923)
Web & Media Group http://lora-aroyo.org @laroyo crowd annotatorannotation example annotation   choices   Lora Aroyo, Chris Welty: The Three Sides of CrowdTruth. J. Human Computation. 1(1). 2014. triangle of reference
Web & Media Group http://lora-aroyo.org @laroyo crowd annotatorannotation example annotation   choices   passionate,   rollicking,   literate,   humorous,  silly,   aggressive,  fiery,   does  not  fit  into   rousing,   cheerful,  fun,   poignant,  wis9ul,   campy,  quirky,   tense,  anxious,   any  of  the  5   confident,   sweet,  amiable,   bi>ersweet,   whimsical,  wi>y,   intense,  vola?le,   clusters   boisterous,   good-­‐natured   autumnal,   wry   visceral       rowdy       brooding               Category 1 Category 3 Category 5 Lora Aroyo, Chris Welty: The Three Sides of CrowdTruth. J. Human Computation. 1(1). 2014. triangle of reference
Web & Media Group http://lora-aroyo.org @laroyo crowd annotatorannotation example annotation   choices   treat   prevent   side  effect   diagnose   cause   other   Treat Cause Prevent Lora Aroyo, Chris Welty: The Three Sides of CrowdTruth. J. Human Computation. 1(1). 2014. ANTIBIOTICS are the first line treatment for indications of TYPHUS. triangle of reference
Web & Media Group http://lora-aroyo.org @laroyo 1 1 1 Worker Vector
Web & Media Group http://lora-aroyo.org @laroyo 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 0 0 4 3 0 0 5 1 0 Sentence Vector OUR NEW THEORY OF TRUTH
Web & Media Group http://lora-aroyo.org @laroyo Unclear relationship between the two arguments reflected in the disagreement Sentence Clarity Feeling the way the CHEST expands (PALPATION), can identify areas of the lung that are full of fluid. ?PALPATIONIs CHEST related to diagnose location associated with is_a otherpart_of 0 0 02 3 0 0 0 1 0 0 44 1
Web & Media Group http://lora-aroyo.org @laroyo Clearly expressed relation between the two arguments reflected in the agreement Sentence Clarity ?CONJUNCTIVITISHYPERAEMIA related toIs 0 0 0 1 0 0 0 013 0 0 0 0 0 symptomcause Redness (HYPERAEMIA), irritation (chemosis) and watering (epiphora) of the eyes are symptoms common to all forms of CONJUNCTIVITIS.
Web & Media Group http://lora-aroyo.org @laroyo 1 13 4 3 1 7 2 1 6 2 1 1 1 1 1 1 3 3 1 1 8 3 1 5 7 1 1 2 1 Relation Similarity
Web & Media Group http://lora-aroyo.org @laroyo Measures how clearly a sentence expresses a relation 0 1 1 0 0 4 3 0 0 5 1 0 Unit vector for relation R6 Sentence Vector Cosine = .55 confidence Sentence-Relation Score
Web & Media Group http://lora-aroyo.org @laroyo Measured per worker averaged over all sentences (whether worker consistently disagrees with everyone else) Worker-Sentence Disagreement 0 1 1 0 0 4 3 0 0 5 1 0 Worker’s sentence vector Sentence Vector AVG (Cosine) Worker Quality
Web & Media Group http://lora-aroyo.org @laroyo Measured for pairs of workers / sentence averaged over all sentences (identify communities of though) Worker-Worker Disagreement 0 1 1 0 0 1 0 0 0 0 1 0 Worker 1 sentence vector Worker 2 Vector AVG (Cosine) Worker Similarity
Web & Media Group http://lora-aroyo.org @laroyo
Web & Media Group http://lora-aroyo.org @laroyo Example Results
Web & Media Group http://lora-aroyo.org @laroyo Example Results
Web & Media Group http://lora-aroyo.org @laroyo Example Results
Web & Media Group http://lora-aroyo.org @laroyo crowdtruth.org experimental results show that CrowdTruth is a better way of capturing context a more accurate way to predict & explain truth
Web & Media Group http://lora-aroyo.org @laroyo Training a Relation Extraction Classifier F1 Cost per sentence CrowdTruth 0.642 $0.66 Expert Annotator 0.638 $2.00 Single Annotator 0.492 $0.08 “wisdom of the crowd” provides training data that is at least as good if not better than experts only with proper analytic framework for harnessing disagreement from the crowd
Web & Media Group http://lora-aroyo.org @laroyo Learning Curves (crowd with pos./neg. threshold at 0.5 above 400 sent.: crowd consistently over baseline & single above 600 sent.: crowd out-performs experts
Web & Media Group http://lora-aroyo.org @laroyo Learning Curves
Web & Media Group http://lora-aroyo.org @laroyo Take Home Message True  and  False  is  not  enough,     There  is  diversity  in  human  interpreta3on     CrowdTruth  introduces  a  spa3al  representa3on  of   meaning  that  through  crowdsourcing     harnesses  disagreement     Using  CrowdTruth   untrained  workers  can  be  just  as  reliable  as     highly  trained  experts  
Web & Media Group http://lora-aroyo.org @laroyo https://www.youtube.com/watch?v=CyAI_lVUdzM To be AND not to be: quantum intelligence? Lora Aroyo & Chris Welty

More Related Content

PDF
Data Science with Humans in the Loop
Lora Aroyo
 
PDF
My ESWC 2017 keynote: Disrupting the Semantic Comfort Zone
Lora Aroyo
 
PPTX
Digital Humanities Benelux 2017: Keynote Lora Aroyo
Lora Aroyo
 
PDF
Data Science with Human in the Loop @Faculty of Science #Leiden University
Lora Aroyo
 
PDF
Harnessing diversity in crowds and machines for better ner performance
oanainel
 
PDF
CrowdTruth Tutorial: Using the Crowd to Understand Ambiguity
Anca Dumitrache
 
PDF
Utilizing Social Health Websites for Cognitive Computing and Clinical Decisio...
CrowdTruth
 
PDF
SXSW2017 @NewDutchMedia Talk: Exploration is the New Search
Lora Aroyo
 
Data Science with Humans in the Loop
Lora Aroyo
 
My ESWC 2017 keynote: Disrupting the Semantic Comfort Zone
Lora Aroyo
 
Digital Humanities Benelux 2017: Keynote Lora Aroyo
Lora Aroyo
 
Data Science with Human in the Loop @Faculty of Science #Leiden University
Lora Aroyo
 
Harnessing diversity in crowds and machines for better ner performance
oanainel
 
CrowdTruth Tutorial: Using the Crowd to Understand Ambiguity
Anca Dumitrache
 
Utilizing Social Health Websites for Cognitive Computing and Clinical Decisio...
CrowdTruth
 
SXSW2017 @NewDutchMedia Talk: Exploration is the New Search
Lora Aroyo
 

What's hot (10)

PDF
Crowdsourcing ambiguity aware ground truth - collective intelligence 2017
Lora Aroyo
 
PDF
Crowds & Niches Teaching Machines to Diagnose: NLeSC Kick off eHumanities pr...
Lora Aroyo
 
PDF
Harnessing Human Semantics at Scale (updated)
Lora Aroyo
 
PDF
Truth is a Lie: 7 Myths about Human Annotation @CogComputing Forum 2014
Lora Aroyo
 
PDF
Social Web 2014: Final Presentations (Part I)
Lora Aroyo
 
PPTX
ESWC - PhD Symposium 2016
oanainel
 
PPTX
"What is Data Science?"
Renee Teate
 
PPT
Living Online
Paul McMahon
 
PDF
State of RecSys: Recap of RecSys 2012
Alan Said
 
PPT
Finding the Phoenix: Feathers, Flight & the Future of Libraries
hblowers
 
Crowdsourcing ambiguity aware ground truth - collective intelligence 2017
Lora Aroyo
 
Crowds & Niches Teaching Machines to Diagnose: NLeSC Kick off eHumanities pr...
Lora Aroyo
 
Harnessing Human Semantics at Scale (updated)
Lora Aroyo
 
Truth is a Lie: 7 Myths about Human Annotation @CogComputing Forum 2014
Lora Aroyo
 
Social Web 2014: Final Presentations (Part I)
Lora Aroyo
 
ESWC - PhD Symposium 2016
oanainel
 
"What is Data Science?"
Renee Teate
 
Living Online
Paul McMahon
 
State of RecSys: Recap of RecSys 2012
Alan Said
 
Finding the Phoenix: Feathers, Flight & the Future of Libraries
hblowers
 
Ad

Viewers also liked (20)

PDF
(Presentation Chris) Crowdsourcing & Semantic Web: Dagstuhl 2014
Lora Aroyo
 
PDF
Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Lora)
Lora Aroyo
 
PPT
Visualization of Disagreement-based Quality Metrics of Crowdsourcing Data
CrowdTruth
 
PDF
Gamification of crowdsourcing tasks: What motivates a medical expert?
CrowdTruth
 
PDF
Crowdsourcing Disagreement on Open-Domain Questions
Benjamin Timmermans
 
PDF
Exploiting disagreement through open ended tasks for capturing interpretation...
Benjamin Timmermans
 
PDF
Towards Better Media Understanding and Searchability
oanainel
 
PDF
Dive+@ICTOpen2017
oanainel
 
PPTX
Dive+ NL eScience symposium 2015
CrowdTruth
 
PDF
CrowdTruth Games @NLeSc eHumanities day 2015
Lora Aroyo
 
PDF
Boosting Named Entity Extraction through Crowdsourcing
oanainel
 
PDF
Kick-off meeting Linkflows project
Vrije Universiteit Amsterdam, IOS Press, Netherlands Institute for Sound & Vision
 
PDF
Europeana GA 2016: Harnessing Crowds, Niches & Professionals in the Digital Age
Lora Aroyo
 
PPTX
DIVE Semantic Web Challenge Presentation
Victor de Boer
 
PDF
Genuine semantic publishing
Tobias Kuhn
 
PDF
Stitch by Stitch: Annotating Fashion at the Rijksmuseum
Lora Aroyo
 
PDF
CrowdTruth for Digital Hermeneutics
Lora Aroyo
 
PDF
Closing Event - Watson Innovation Course
Lora Aroyo
 
PDF
DIVE+ @ NLeSymposium 2015: Towards New Cultural Commons with DIVE+
Lora Aroyo
 
PDF
Keynote @Final NWO CATCH Program Event
Lora Aroyo
 
(Presentation Chris) Crowdsourcing & Semantic Web: Dagstuhl 2014
Lora Aroyo
 
Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Lora)
Lora Aroyo
 
Visualization of Disagreement-based Quality Metrics of Crowdsourcing Data
CrowdTruth
 
Gamification of crowdsourcing tasks: What motivates a medical expert?
CrowdTruth
 
Crowdsourcing Disagreement on Open-Domain Questions
Benjamin Timmermans
 
Exploiting disagreement through open ended tasks for capturing interpretation...
Benjamin Timmermans
 
Towards Better Media Understanding and Searchability
oanainel
 
Dive+@ICTOpen2017
oanainel
 
Dive+ NL eScience symposium 2015
CrowdTruth
 
CrowdTruth Games @NLeSc eHumanities day 2015
Lora Aroyo
 
Boosting Named Entity Extraction through Crowdsourcing
oanainel
 
Kick-off meeting Linkflows project
Vrije Universiteit Amsterdam, IOS Press, Netherlands Institute for Sound & Vision
 
Europeana GA 2016: Harnessing Crowds, Niches & Professionals in the Digital Age
Lora Aroyo
 
DIVE Semantic Web Challenge Presentation
Victor de Boer
 
Genuine semantic publishing
Tobias Kuhn
 
Stitch by Stitch: Annotating Fashion at the Rijksmuseum
Lora Aroyo
 
CrowdTruth for Digital Hermeneutics
Lora Aroyo
 
Closing Event - Watson Innovation Course
Lora Aroyo
 
DIVE+ @ NLeSymposium 2015: Towards New Cultural Commons with DIVE+
Lora Aroyo
 
Keynote @Final NWO CATCH Program Event
Lora Aroyo
 
Ad

Similar to Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote) (20)

PDF
Towards a more comprehensive measure of eudaimonia_Carol Graham.pdf
StatsCommunications
 
PDF
Slidesnotes
Mark Holmgren Consulting
 
PDF
Designing Effective Science Communication
Jennifer Briselli
 
PPTX
Monologistic heuristics
Bounds OutOf
 
PDF
CrowdTruth @VU Faculty Colloquium (June 2015)
Lora Aroyo
 
PDF
Digital Health Literacy Project Curriculum Paper
Megan Wilson, M.A.
 
PPT
Taking a lead in promoting choice, control and valued opportunities for socia...
Iriss
 
PPTX
Dean R Berry Problems and Solutions: Hunger and Food Deprivation
Riverside County Office of Education
 
PPTX
philosophy LESSON 3 ( DETERMINING TRUTH).pptx
MitziDePablo
 
PDF
A STATE OF MIND - Technology, Change, and Wellbeing
University of Hertfordshire
 
PPTX
The Failure of Skepticism: Rethinking Information Literacy and Political Pol...
Chris Sweet
 
DOCX
Research Model1. Select the topic2. Defining the problem3. .docx
audeleypearl
 
DOCX
Research Model1. Select the topic2. Defining the problem3. .docx
verad6
 
PDF
Hacking The Human Interface
Samantha Quiñones
 
DOCX
Complex problem Assignment Help
Nicole Valerio
 
PDF
Wisdom of the Crowd vs. Collective Intelligence.
The New School
 
PPTX
HXR 2016: Sustainable Design -Jen Briselli, James Christie, Mad*Pow
HxRefactored
 
PPTX
Chapter 8 (sub for sharon)
AIB College of Business
 
PPT
Public Health in Scotland - Harry Burns
NHSScotlandEvent
 
PDF
diversity+inclusion=innovation (TNSHRM)
Joe Gerstandt
 
Towards a more comprehensive measure of eudaimonia_Carol Graham.pdf
StatsCommunications
 
Slidesnotes
Mark Holmgren Consulting
 
Designing Effective Science Communication
Jennifer Briselli
 
Monologistic heuristics
Bounds OutOf
 
CrowdTruth @VU Faculty Colloquium (June 2015)
Lora Aroyo
 
Digital Health Literacy Project Curriculum Paper
Megan Wilson, M.A.
 
Taking a lead in promoting choice, control and valued opportunities for socia...
Iriss
 
Dean R Berry Problems and Solutions: Hunger and Food Deprivation
Riverside County Office of Education
 
philosophy LESSON 3 ( DETERMINING TRUTH).pptx
MitziDePablo
 
A STATE OF MIND - Technology, Change, and Wellbeing
University of Hertfordshire
 
The Failure of Skepticism: Rethinking Information Literacy and Political Pol...
Chris Sweet
 
Research Model1. Select the topic2. Defining the problem3. .docx
audeleypearl
 
Research Model1. Select the topic2. Defining the problem3. .docx
verad6
 
Hacking The Human Interface
Samantha Quiñones
 
Complex problem Assignment Help
Nicole Valerio
 
Wisdom of the Crowd vs. Collective Intelligence.
The New School
 
HXR 2016: Sustainable Design -Jen Briselli, James Christie, Mad*Pow
HxRefactored
 
Chapter 8 (sub for sharon)
AIB College of Business
 
Public Health in Scotland - Harry Burns
NHSScotlandEvent
 
diversity+inclusion=innovation (TNSHRM)
Joe Gerstandt
 

More from Lora Aroyo (17)

PDF
NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdf
Lora Aroyo
 
PDF
CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine Learning
Lora Aroyo
 
PDF
Data excellence: Better data for better AI
Lora Aroyo
 
PDF
CHIP Demonstrator presentation @ CATCH Symposium
Lora Aroyo
 
PDF
Semantic Web Challenge: CHIP Demonstrator
Lora Aroyo
 
PDF
The Rijksmuseum Collection as Linked Data
Lora Aroyo
 
PDF
Keynote at International Conference of Art Libraries 2018 @Rijksmuseum
Lora Aroyo
 
PDF
FAIRview: Responsible Video Summarization @NYCML'18
Lora Aroyo
 
PDF
Understanding bias in video news & news filtering algorithms
Lora Aroyo
 
PDF
StorySourcing: Telling Stories with Humans & Machines
Lora Aroyo
 
PDF
DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...
Lora Aroyo
 
PDF
"Video Killed the Radio Star": From MTV to Snapchat
Lora Aroyo
 
PPTX
UMAP 2016 Opening Ceremony
Lora Aroyo
 
PDF
Crowdsourcing & Nichesourcing: Enriching Cultural Heritage with Experts & Cr...
Lora Aroyo
 
PDF
Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...
Lora Aroyo
 
PDF
Semantic Digital Humanities Workshop 2015 @Oxford
Lora Aroyo
 
PDF
#CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2...
Lora Aroyo
 
NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdf
Lora Aroyo
 
CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine Learning
Lora Aroyo
 
Data excellence: Better data for better AI
Lora Aroyo
 
CHIP Demonstrator presentation @ CATCH Symposium
Lora Aroyo
 
Semantic Web Challenge: CHIP Demonstrator
Lora Aroyo
 
The Rijksmuseum Collection as Linked Data
Lora Aroyo
 
Keynote at International Conference of Art Libraries 2018 @Rijksmuseum
Lora Aroyo
 
FAIRview: Responsible Video Summarization @NYCML'18
Lora Aroyo
 
Understanding bias in video news & news filtering algorithms
Lora Aroyo
 
StorySourcing: Telling Stories with Humans & Machines
Lora Aroyo
 
DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...
Lora Aroyo
 
"Video Killed the Radio Star": From MTV to Snapchat
Lora Aroyo
 
UMAP 2016 Opening Ceremony
Lora Aroyo
 
Crowdsourcing & Nichesourcing: Enriching Cultural Heritage with Experts & Cr...
Lora Aroyo
 
Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...
Lora Aroyo
 
Semantic Digital Humanities Workshop 2015 @Oxford
Lora Aroyo
 
#CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2...
Lora Aroyo
 

Recently uploaded (20)

PDF
Five Habits of High-Impact Board Members
OnBoard
 
PDF
Produktkatalog für HOBO Datenlogger, Wetterstationen, Sensoren, Software und ...
Metrics GmbH
 
PPTX
TEXTILE technology diploma scope and career opportunities
kriti38
 
PDF
Improvisation in detection of pomegranate leaf disease using transfer learni...
IAESIJAI
 
PDF
Comparative analysis of machine learning models for fake news detection in so...
IAESIJAI
 
PPTX
Build Your First AI Agent with UiPath.pptx
suhanisingh58689
 
PDF
5-Ways-AI-is-Revolutionizing-Telecom-Quality-Engineering.pdf
Kalilur Rahman
 
PDF
Accessing-Finance-in-Jordan-MENA 2024 2025.pdf
Mustafa Kuğu
 
PDF
STKI Israel Market Study 2025 version august
Dr. Jimmy Schwarzkopf
 
PPTX
Module 1 Introduction to Web Programming .pptx
kamleshkc191
 
PDF
The-2025-Engineering-Revolution-AI-Quality-and-DevOps-Convergence.pdf
Kalilur Rahman
 
PDF
Dell Pro Micro: Speed customer interactions, patient processing, and learning...
Principled Technologies
 
PPTX
GROUP4NURSINGINFORMATICSREPORT-2 PRESENTATION
ClarisejaneSartillo1
 
PDF
The influence of sentiment analysis in enhancing early warning system model f...
IAESIJAI
 
PPTX
Internet of Everything -Basic concepts details
sendilkumar66
 
PPTX
Configure Apache Mutual Authentication
Antonino Piraino
 
PDF
4 layer Arch & Reference Arch of IoT.pdf
sendilkumar66
 
DOCX
Basics of Cloud Computing - Cloud Ecosystem
suthak11
 
DOCX
search engine optimization ppt fir known well about this
StandardCodeLearning
 
PPTX
Training Program for knowledge in solar cell and solar industry
RabindranathKaibarty
 
Five Habits of High-Impact Board Members
OnBoard
 
Produktkatalog für HOBO Datenlogger, Wetterstationen, Sensoren, Software und ...
Metrics GmbH
 
TEXTILE technology diploma scope and career opportunities
kriti38
 
Improvisation in detection of pomegranate leaf disease using transfer learni...
IAESIJAI
 
Comparative analysis of machine learning models for fake news detection in so...
IAESIJAI
 
Build Your First AI Agent with UiPath.pptx
suhanisingh58689
 
5-Ways-AI-is-Revolutionizing-Telecom-Quality-Engineering.pdf
Kalilur Rahman
 
Accessing-Finance-in-Jordan-MENA 2024 2025.pdf
Mustafa Kuğu
 
STKI Israel Market Study 2025 version august
Dr. Jimmy Schwarzkopf
 
Module 1 Introduction to Web Programming .pptx
kamleshkc191
 
The-2025-Engineering-Revolution-AI-Quality-and-DevOps-Convergence.pdf
Kalilur Rahman
 
Dell Pro Micro: Speed customer interactions, patient processing, and learning...
Principled Technologies
 
GROUP4NURSINGINFORMATICSREPORT-2 PRESENTATION
ClarisejaneSartillo1
 
The influence of sentiment analysis in enhancing early warning system model f...
IAESIJAI
 
Internet of Everything -Basic concepts details
sendilkumar66
 
Configure Apache Mutual Authentication
Antonino Piraino
 
4 layer Arch & Reference Arch of IoT.pdf
sendilkumar66
 
Basics of Cloud Computing - Cloud Ecosystem
suthak11
 
search engine optimization ppt fir known well about this
StandardCodeLearning
 
Training Program for knowledge in solar cell and solar industry
RabindranathKaibarty
 

Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)

  • 1. Web & Media Group http://lora-aroyo.org @laroyo Truth is a Lie Rules & Semantics from Crowd Perspectives
  • 2. Web & Media Group http://lora-aroyo.org @laroyo “in embracing the diversity of human beings, we will find a surer way to true happiness” Malcolm Gladwell, 2006 in his TED talk “Choice, happiness and spaghetti sauce”
  • 3. Web & Media Group http://lora-aroyo.org @laroyo “If we treat human brains as processors in a distributed system, each can perform a small part of a massive computation” Luis von Ahn, 2006 “Games with a Purpose”, Computer Magazine
  • 4. Web & Media Group http://lora-aroyo.org @laroyo “The challenge for Watson was to answer questions posed in every nuance of natural language. The first step in accomplishing this was to have a lot of questions and their correct answers. Any cognitive system must take this first step.” Chris Welty, 2014 “Cognitive Computing and the Future”, 1st Cognitive Computing Forum
  • 5. Web & Media Group http://lora-aroyo.org @laroyo Take Home Message True  and  False  is  not  enough,     There  is  diversity  in  human  interpreta3on     CrowdTruth  introduces  a  spa3al  representa3on  of   meaning  that  through  crowdsourcing     harnesses  disagreement     Using  CrowdTruth   untrained  workers  can  be  just  as  reliable  as     highly  trained  experts  
  • 6. Web & Media Group http://lora-aroyo.org @laroyo Is this song …. Passionate Rousing Confident Boisterous Rowdy Literate Poignant Wistful Bittersweet Autumnal Brooding Rollicking Cheerful Fun Sweet Amiable Good-natured Humorous Silly Campy Whimsical Witty Wry Aggressive Fiery Tense Anxious Intense Volatile
  • 7. Web & Media Group http://lora-aroyo.org @laroyo ?Passionate Rousing Confident Boisterous Rowdy Literate Poignant Wistful Bittersweet Autumnal Brooding Rollicking Cheerful Fun Sweet Amiable Good-natured Humorous Silly Campy Whimsical Witty Wry Aggressive Fiery Tense Anxious Intense Volatile Is this song ….
  • 8. Web & Media Group http://lora-aroyo.org @laroyo ?- category(“That’s not my name”, x) ?Passionate Rousing Confident Boisterous Rowdy Literate Poignant Wistful Bittersweet Autumnal Brooding Rollicking Cheerful Fun Sweet Amiable Good-natured Humorous Silly Campy Whimsical Witty Wry Aggressive Fiery Tense Anxious Intense Volatile (Lee and Hu 2012)
  • 9. Web & Media Group http://lora-aroyo.org @laroyo Patients with TYPHUS who were given ANTIBIOTICS exhibited several side-effects. With ANTIBIOTICS in short supply, DDT was used during World War II to control the insect vectors of TYPHUS. ANTIBIOTICS are the first line treatment for indications of TYPHUS. Does this sentence express TREATS(Antibiotics, Typhus)?
  • 10. Web & Media Group http://lora-aroyo.org @laroyo TREATS(Antibiotics, Typhus) Is this True?
  • 11. Web & Media Group http://lora-aroyo.org @laroyo What is the treatment for typhus? Antibiotic therapy is recommended for both endemic and epidemic typhus infections because early treatment with antibiotics (for example, azithromycin, doxycycline, tetracycline, or chloramphenicol) can cure most people infected with the bacteria. Consultation with an infectious-disease expert is advised especially if epidemic typhus or typhus in pregnant females is diagnosed. Delays in treatment may allow renal, lung, or nervous system problems to develop. Some patients, especially the elderly, may die. TREATS(Antibiotics, Typhus) Is this True?
  • 12. Web & Media Group http://lora-aroyo.org @laroyo TREATS(Corticosteroids, Lupus) Is this True? What is the treatment for lupus? In more severe cases of Lupus, medications that modulate the immune system (mainly corticosteroids, immunosuppressants) are used to control the disease and prevent recurrence of symptoms (known as flares). Depending on the dosage, people who require steroids may develop Cushing's syndrome. This may subside if and when the large initial dosage is reduced, but long-term use of even low doses can cause elevated blood pressure and cataracts. Numerous new immunosuppressive drugs are being actively tested. Rather than suppressing the immune system nonspecifically, as corticosteroids do, they target the responses of individual [types of] immune cells.
  • 13. Web & Media Group http://lora-aroyo.org @laroyo Human Interpretation “I will never understand people” “They are the worst”
  • 14. Web & Media Group http://lora-aroyo.org @laroyo human subjectivity, ambiguity & uncertainty of expression are part of human semantics
  • 15. Web & Media Group http://lora-aroyo.org @laroyo current practices in computer science (AI) are based on fallacies
  • 16. Web & Media Group http://lora-aroyo.org @laroyo single truth
  • 17. Web & Media Group http://lora-aroyo.org @laroyo experts rule
  • 18. Web & Media Group http://lora-aroyo.org @laroyo disagreement is bad
  • 19. Web & Media Group http://lora-aroyo.org @laroyo simplification of context this all results in
  • 20. Web & Media Group http://lora-aroyo.org @laroyo
  • 21. Web & Media Group http://lora-aroyo.org @laroyo this means our systems understand only simplified black & white world problem ..
  • 22. Web & Media Group http://lora-aroyo.org @laroyo reality is complex diverse but …
  • 23. Web & Media Group http://lora-aroyo.org @laroyo current gold standard practices are not adequate any more so...
  • 24. Web & Media Group http://lora-aroyo.org @laroyo there is a gap we need to bridge and …
  • 25. Web & Media Group http://lora-aroyo.org @laroyo interestingly …
  • 26. Web & Media Group http://lora-aroyo.org @laroyo in humanities … diversity is well accepted
  • 27. Web & Media Group http://lora-aroyo.org @laroyo in sciences … diversity is taboo
  • 28. Web & Media Group http://lora-aroyo.org @laroyo despite numerous examples … string theory debate
  • 29. Web & Media Group http://lora-aroyo.org @laroyo global warming debate and …
  • 30. Web & Media Group http://lora-aroyo.org @laroyo evolutionary bio. debate gradualismpunctuated equilibrium and …
  • 31. Web & Media Group http://lora-aroyo.org @laroyo What can we do? so …
  • 32. Web & Media Group http://lora-aroyo.org @laroyo •  collective decisions of large groups of people •  a group of error-prone decision- makers can be surprisingly good at picking the best choice •  when thumbs up or thumbs down - the chance of picking the right answer needs to be > 50% •  the odds that a most of them will pick the right answer is greater than any of them will pick it on their own •  performance gets better as size grows 1785 Marquis de Condorcet “wisdom of crowds”
  • 33. Web & Media Group http://lora-aroyo.org @laroyo • asked 787 people to guess the weight of an ox • none got the right answer • their collective guess was almost perfect 1906 Sir Francis Galton “wisdom of crowds”
  • 34. Web & Media Group http://lora-aroyo.org @laroyo WWII Math Rosies 1942: Ballistics calculations and flight trajectories
  • 35. Web & Media Group http://lora-aroyo.org @laroyo NASA’s Computer Room transcribe raw flight data from celluloid film & oscillograph paper
  • 36. Web & Media Group http://lora-aroyo.org @laroyo “Who Wants to Be a Millionaire?” •  help from individual expert or audience poll •  majority of the audience right 91% of the time •  individuals right only 65% of the time “wisdom of crowds” 2004 James Surowiecki
  • 37. Web & Media Group http://lora-aroyo.org @laroyo Crowdsourcing Human Computing
  • 38. Web & Media Group http://lora-aroyo.org @laroyo the wise crowd diversity of opinion independent perspectives multitude of contexts gives the big picture James Surowiecki
  • 39. Web & Media Group http://lora-aroyo.org @laroyo current practices in computer science (AI) are based on fallacies
  • 40. Web & Media Group http://lora-aroyo.org @laroyo today’s real-world complex problem solving needs understanding of multiple perspectives current practices in computer science (AI) are based on fallacies
  • 41. Web & Media Group http://lora-aroyo.org @laroyo single truth
  • 42. Web & Media Group http://lora-aroyo.org @laroyo multiple truths single truth
  • 43. Web & Media Group http://lora-aroyo.org @laroyo experts rule
  • 44. Web & Media Group http://lora-aroyo.org @laroyo wisdom of crowds experts rule
  • 45. Web & Media Group http://lora-aroyo.org @laroyo disagreement is bad
  • 46. Web & Media Group http://lora-aroyo.org @laroyo disagreement is bad disagreement as signal
  • 47. Web & Media Group http://lora-aroyo.org @laroyo “best collective decisions are result of disagreement, not consensus or compromise” James Surowiecki because …
  • 48. Web & Media Group http://lora-aroyo.org @laroyo can we harness it?
  • 49. Web & Media Group http://lora-aroyo.org @laroyo Is this song …. ? Passionate Rousing Confident Boisterous Rowdy Literate Poignant Wistful Bittersweet Autumnal Brooding Rollicking Cheerful Fun Sweet Amiable Good-natured Humorous Silly Campy Whimsical Witty Wry Aggressive Fiery Tense Anxious Intense Volatile
  • 50. Web & Media Group http://lora-aroyo.org @laroyo If “One Truth” & “No Disagreement” Worker Mood-C1 Mood-C2 Mood-C3 Mood-C4 Mood-C5 W1 1 W2 1 W3 1 W4 1 W5 1 W6 1 W7 W8 W9 1 W10 1 Totals 1 3 1 2 1
  • 51. Web & Media Group http://lora-aroyo.org @laroyo Worker Mood-C1 Mood-C2 Mood-C3 Mood-C4 Mood-C5 Other W1 1 1 1 W2 1 1 1 W3 1 1 1 W4 1 1 W5 1 1 W6 1 1 1 W7 1 1 1 W8 1 1 1 W9 1 1 W10 1 1 1 1 1 Totals 3 5 6 5 2 8 If “Many Truths” & “Disagreement”
  • 52. Web & Media Group http://lora-aroyo.org @laroyo can indicate alternative interpretations Worker Mood-C1 Mood-C2 Mood-C3 Mood-C4 Mood-C5 Other W10 1 1 1 1 1 Totals 3 5 6 5 2 8 Disagreement as Signal can indicate ambiguity in the categorisation can indicate low quality workers
  • 53. Web & Media Group http://lora-aroyo.org @laroyo Does this sentence express TREATS(Antibiotics, Typhus)? Patients with TYPHUS who were given ANTIBIOTICS exhibited several side-effects. With ANTIBIOTICS in short supply, DDT was used during World War II to control the insect vectors of TYPHUS. ANTIBIOTICS are the first line treatment for indications of TYPHUS. 95% 75% 50% Disagreement is signal here too
  • 54. Web & Media Group http://lora-aroyo.org @laroyo Sentence Treat Prevent Cause Side Effect Diagnose S1 2 S2 2 S3 2 S4 2 S5 1 1 S6 2 S7 2 S8 1 1 S9 2 S10 2 If “One Truth”, “No Disagreement” & “Experts”
  • 55. Web & Media Group http://lora-aroyo.org @laroyo •  we rejected 3 fallacies: one truth, experts rule and disagreement is bad •  by encouraging disagreement • we exposed a richer set of possibilities • that help in identifying, processing & understanding context so …
  • 56. Web & Media Group http://lora-aroyo.org @laroyo •  One truth: data collection efforts assume one correct interpretation for every example •  All examples are created equal: ground truth treats all examples the same – either match the correct result or not •  Detailed guidelines help: if examples cause disagreement - add instructions to limit interpretations •  Disagreement is bad: increase quality of annotation data by reducing disagreement among the annotators •  One is enough: most of the annotated examples are evaluated by one person •  Experts are better: annotators with domain knowledge provide better annotations •  Once done, forever valid: annotations are not updated; new data not aligned with old 7 Myths myths directly influence the practice of collecting human annotated data; Need to be revised with a new theory of truth (CrowdTruth) Lora Aroyo, Chris Welty: Truth is a Lie: 7 Myths about Human Annotation, AI Magazine 2014.
  • 57. Web & Media Group http://lora-aroyo.org @laroyo
  • 58. Web & Media Group http://lora-aroyo.org @laroyo crowdtruth.org
  • 59. Web & Media Group http://lora-aroyo.org @laroyo crowdtruth.org a new approach to understanding semantics harnessing the power & diversity of the crowd
  • 60. Web & Media Group http://lora-aroyo.org @laroyo crowdtruth.org informs a vector space model of truth instead of a boolean, fuzzy or statistical model
  • 61. Web & Media Group http://lora-aroyo.org @laroyo observerreferent sign has   triangle of reference (Ogden & Richards, 1923)
  • 62. Web & Media Group http://lora-aroyo.org @laroyo crowd annotatorannotation example annotation   choices   Lora Aroyo, Chris Welty: The Three Sides of CrowdTruth. J. Human Computation. 1(1). 2014. triangle of reference
  • 63. Web & Media Group http://lora-aroyo.org @laroyo crowd annotatorannotation example annotation   choices   passionate,   rollicking,   literate,   humorous,  silly,   aggressive,  fiery,   does  not  fit  into   rousing,   cheerful,  fun,   poignant,  wis9ul,   campy,  quirky,   tense,  anxious,   any  of  the  5   confident,   sweet,  amiable,   bi>ersweet,   whimsical,  wi>y,   intense,  vola?le,   clusters   boisterous,   good-­‐natured   autumnal,   wry   visceral       rowdy       brooding               Category 1 Category 3 Category 5 Lora Aroyo, Chris Welty: The Three Sides of CrowdTruth. J. Human Computation. 1(1). 2014. triangle of reference
  • 64. Web & Media Group http://lora-aroyo.org @laroyo crowd annotatorannotation example annotation   choices   treat   prevent   side  effect   diagnose   cause   other   Treat Cause Prevent Lora Aroyo, Chris Welty: The Three Sides of CrowdTruth. J. Human Computation. 1(1). 2014. ANTIBIOTICS are the first line treatment for indications of TYPHUS. triangle of reference
  • 65. Web & Media Group http://lora-aroyo.org @laroyo 1 1 1 Worker Vector
  • 66. Web & Media Group http://lora-aroyo.org @laroyo 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 0 0 4 3 0 0 5 1 0 Sentence Vector OUR NEW THEORY OF TRUTH
  • 67. Web & Media Group http://lora-aroyo.org @laroyo Unclear relationship between the two arguments reflected in the disagreement Sentence Clarity Feeling the way the CHEST expands (PALPATION), can identify areas of the lung that are full of fluid. ?PALPATIONIs CHEST related to diagnose location associated with is_a otherpart_of 0 0 02 3 0 0 0 1 0 0 44 1
  • 68. Web & Media Group http://lora-aroyo.org @laroyo Clearly expressed relation between the two arguments reflected in the agreement Sentence Clarity ?CONJUNCTIVITISHYPERAEMIA related toIs 0 0 0 1 0 0 0 013 0 0 0 0 0 symptomcause Redness (HYPERAEMIA), irritation (chemosis) and watering (epiphora) of the eyes are symptoms common to all forms of CONJUNCTIVITIS.
  • 69. Web & Media Group http://lora-aroyo.org @laroyo 1 13 4 3 1 7 2 1 6 2 1 1 1 1 1 1 3 3 1 1 8 3 1 5 7 1 1 2 1 Relation Similarity
  • 70. Web & Media Group http://lora-aroyo.org @laroyo Measures how clearly a sentence expresses a relation 0 1 1 0 0 4 3 0 0 5 1 0 Unit vector for relation R6 Sentence Vector Cosine = .55 confidence Sentence-Relation Score
  • 71. Web & Media Group http://lora-aroyo.org @laroyo Measured per worker averaged over all sentences (whether worker consistently disagrees with everyone else) Worker-Sentence Disagreement 0 1 1 0 0 4 3 0 0 5 1 0 Worker’s sentence vector Sentence Vector AVG (Cosine) Worker Quality
  • 72. Web & Media Group http://lora-aroyo.org @laroyo Measured for pairs of workers / sentence averaged over all sentences (identify communities of though) Worker-Worker Disagreement 0 1 1 0 0 1 0 0 0 0 1 0 Worker 1 sentence vector Worker 2 Vector AVG (Cosine) Worker Similarity
  • 73. Web & Media Group http://lora-aroyo.org @laroyo
  • 74. Web & Media Group http://lora-aroyo.org @laroyo Example Results
  • 75. Web & Media Group http://lora-aroyo.org @laroyo Example Results
  • 76. Web & Media Group http://lora-aroyo.org @laroyo Example Results
  • 77. Web & Media Group http://lora-aroyo.org @laroyo crowdtruth.org experimental results show that CrowdTruth is a better way of capturing context a more accurate way to predict & explain truth
  • 78. Web & Media Group http://lora-aroyo.org @laroyo Training a Relation Extraction Classifier F1 Cost per sentence CrowdTruth 0.642 $0.66 Expert Annotator 0.638 $2.00 Single Annotator 0.492 $0.08 “wisdom of the crowd” provides training data that is at least as good if not better than experts only with proper analytic framework for harnessing disagreement from the crowd
  • 79. Web & Media Group http://lora-aroyo.org @laroyo Learning Curves (crowd with pos./neg. threshold at 0.5 above 400 sent.: crowd consistently over baseline & single above 600 sent.: crowd out-performs experts
  • 80. Web & Media Group http://lora-aroyo.org @laroyo Learning Curves
  • 81. Web & Media Group http://lora-aroyo.org @laroyo Take Home Message True  and  False  is  not  enough,     There  is  diversity  in  human  interpreta3on     CrowdTruth  introduces  a  spa3al  representa3on  of   meaning  that  through  crowdsourcing     harnesses  disagreement     Using  CrowdTruth   untrained  workers  can  be  just  as  reliable  as     highly  trained  experts  
  • 82. Web & Media Group http://lora-aroyo.org @laroyo https://www.youtube.com/watch?v=CyAI_lVUdzM To be AND not to be: quantum intelligence? Lora Aroyo & Chris Welty