Metrocon-Rise-Of-Crowd-Computing
2 likes913 views
The document summarizes crowd computing and crowdsourcing. It discusses how tasks traditionally performed by employees can now be outsourced to large online groups through platforms like Amazon Mechanical Turk. It provides examples of how crowds have been used for tasks like data labeling, content analysis, and computer vision. It also discusses some of the opportunities, challenges, and open questions around using crowds for human computation, including ensuring data quality, addressing fraud and ethics concerns, understanding the demographics of workers, and regulating the emerging field.
Metrocon-Rise-Of-Crowd-Computing
- 1. The Rise of Crowd Computing Matt Lease School of Information @mattlease University of Texas at Austin [email protected]
- 2. Crowdsourcing • Jeff Howe. Wired, June 2006. • Take a job traditionally performed by a known agent (often an employee) • Outsource it to an undefined, generally large group of people via an open call • New application of principles from open source movement 2
- 3. Amazon Mechanical Turk (MTurk) • Marketplace for crowd labor (microtasks) • Created in 2005 (still in “beta”) • On-demand, scalable, 24/7 global workforce 3
- 4. Labeling Data (“Gold Rush”) @mattlease 4
- 5. Snow et al. (EMNLP 2008) • MTurk annotation for 5 Tasks – Affect recognition – Word similarity – Recognizing textual entailment – Event temporal ordering – Word sense disambiguation • 22K labels for US $26 • High agreement between consensus labels and gold-standard labels 5
- 6. Alonso et al. (SIGIR Forum 2008) • MTurk for Information Retrieval (IR) – Judge relevance of search engine results • Many follow-on studies (design, quality, cost) 6
- 7. Sorokin & Forsythe (CVPR 2008) • MTurk for Computer Vision • 4K labels for US $60 7
- 8. Studying People & Interactive Systems @mattlease 8
- 9. Kittur, Chi, & Suh (CHI 2008) • MTurk for User Studies • “…make creating believable invalid responses as effortful as completing the task in good faith.” 9
- 10. Social & Behavioral Sciences • A Guide to Behavioral Experiments on Mechanical Turk – W. Mason and S. Suri (2010). SSRN online. • Crowdsourcing for Human Subjects Research – L. Schmidt (CrowdConf 2010) • Crowdsourcing Content Analysis for Behavioral Research: Insights from Mechanical Turk – Conley & Tosti-Kharas (2010). Academy of Management • Amazon's Mechanical Turk : A New Source of Inexpensive, Yet High-Quality, Data? – M. Buhrmester et al. (2011). Perspectives… 6(1):3-5. – see also: Amazon Mechanical Turk Guide for Social Scientists 10
- 11. Remote Usability Testing • Liu et al. (ASIS&T’12) • On-site vs. crowdsourced usability testing • Advantages – More Participants – More Diverse Participants – High Speed – Low Cost • Disadvantages – Lower Quality Feedback – Less Interaction – Greater Need for Quality Control – Less Focused User Groups 11
- 12. Beyond MTurk @mattlease 12
- 13. ESP Game (Games With a Purpose) von Ahn & Dabbish (2004) 13
- 14. reCaptcha von Ahn et al. (2008). In Science. 14
- 15. Crowd Sensing & Monitoring • Sullivan et al. (2009). Bio. Conservation (142):10 • Keynote by Steve Kelling (ASIS&T 2011) 15
- 16. Crowd Sensing & Montitoring (2) August 12, 2012 16
- 17. 17
- 18. Human Computation @mattlease 18
- 19. The Mechanical Turk The original, constructed and unveiled in 1770 by Wolfgang von Kempelen (1734–1804) J. Pontin. Artificial Intelligence, With Help From the Humans. New York Times (March 25, 2007) 19
- 20. • What was old is new • Crowdsourcing: A New Branch of Computer Science – D.A. Grier, March 29, 2011 • Tabulating the heavens: computing the Nautical Almanac in 18th-century England – M. Croarken (2003) Princeton University Press, 2005 20
- 21. The Human Processing Unit (HPU) • Davis et al. (2010) HPU 21
- 22. Human Computation • Luis von Ahn (2005) • Use of people to perform computations beyond capabilities of state-of-the-art automation • Computational description of attributes, capabilities, & limitations of human computers • Having people do stuff (instead of computers) 22
- 23. Blending Automation & Human Computation @mattlease 23
- 24. “Amazon Remembers” 24
- 25. Ethics Checking: The Next Frontier? • Mark Johnson’s address at ACL 2003 – Transcript in Conduit 12(2) 2003 • Think how useful a little “ethics checker and corrector” program integrated into a word processor could be! 25
- 26. Soylent: A Word Processor with a Crowd Inside • Bernstein et al., UIST 2010 26
- 27. Translation by monolingual speakers • C. Hu, CHI 2009 27
- 28. fold.it S. Cooper et al. (2010) Alice G. Walton. Online Gamers Help Solve Mystery of Critical AIDS Virus Enzyme. The Atlantic, October 8, 2011. 28
- 29. VizWiz aaaaaaaa Bingham et al. (UIST 2010) Matt Lease - [email protected] 29
- 30. PlateMate (Noronha et al., UIST’10) 30
- 31. Image Analysis and more: Eatery 31
- 32. 32
- 33. @mattlease 33
- 34. What about data quality? • Many papers on statistical methods – Online vs. offline, feature-based vs. content-agnostic – Worker calibration, noise vs. bias, weighted voting – Work in my lab by Jung, Kumar, Ryu, & Tang • Human factors matter – Instructions, design, interface, interaction – Names, relationship, reputation (Klinger & Lease’11) – Fair pay, hourly vs. per-task, recognition, advancement – For contrast with MTurk, consider Kochhar (2010) 34
- 35. What about benchmarks? • How well do alternative methods perform? – Common datasets & tasks enable comparison – Contests drive innovation & measure collective progress • Common tasks today – Translation – Transcription – Search Evaluation – Verification & Correction – Content Generation • NIST TREC Crowdsourcing Track – Two tasks, currently in 2nd Year of Track 35
- 36. How to Design Optimal Workflows? 36
- 37. What about sensitive data? • Not all data can be publicly disclosed – User data (e.g. AOL query log, Netflix ratings) – Intellectual property – Legal confidentiality • Need to restrict who is in your crowd – Separate channel (workforce) from technology – Hot question for adoption at enterprise level 37
- 38. What about fraud? • Some reports of robot “workers” on MTurk – Artificial Artificial Artificial Intelligence – Violates terms of service • Why not just use a captcha? 38
- 39. Requester Fraud on MTurk “Do not do any HITs that involve: filling in CAPTCHAs; secret shopping; test our web page; test zip code; free trial; click my link; surveys or quizzes (unless the requester is listed with a smiley in the Hall of Fame/Shame); anything that involves sending a text message; or basically anything that asks for any personal information at all—even your zip code. If you feel in your gut it’s not on the level, IT’S NOT. Why? Because they are scams...” 39
- 40. Fraud via Crowds
- 41. Wang et al., WWW’12 • “…not only do malicious crowd-sourcing systems exist, but they are rapidly growing…” 41
- 42. Robert Sim, MSR Summit’12 42
- 43. Broader Issues @mattlease 43
- 44. What about regulation? • Wolfson & Lease (ASIS&T’11) • As usual, technology is ahead of the law – employment law – patent inventorship – data security and the Federal Trade Commission – copyright ownership – securities regulation of crowdfunding • Take-away: don’t panic, but be mindful – Understand risks of “just in-time compliance” 44
- 45. What about context? • Workflow research investigates how to decompose complex tasks into simpler ones • Micro-task design removes extraneous context and detail in order to streamline tasks – e.g. “Can you name who is in this photo?” 45
- 46. Context & Informed Consent • What is the larger task I’m contributing to? • Who will benefit from it and how? 46
- 47. What about ethics? • Silberman, Irani, and Ross (2010) – “How should we… conceptualize the role of these people who we ask to power our computing?” – Power dynamics between parties – “Abstraction hides detail” • Fort, Adda, and Cohen (2011) – “…opportunities for our community to deliberately value ethics above cost savings.” 47
- 48. Davis et al. (2010) The HPU. HPU 48
- 49. Who are the workers? • A. Baio, November 2008. The Faces of Mechanical Turk. • P. Ipeirotis. March 2010. The New Demographics of Mechanical Turk • J. Ross, et al. Who are the Crowdworkers? CHI 2010. 49
- 50. HPU: “Abstraction hides detail” 50
- 51. How much to pay? Performance, psychology, economics, and ethics • What motivates strong performance? • Primary or supplemental income? • Effect on local economies? • Ethics of paying something (if low) vs. paying nothing (e.g., games) 51
- 52. Digital Dirty Jobs • The Googler who Looked at the Worst of the Internet • Policing the Web’s Lurid Precincts • Facebook content moderation • The dirty job of keeping Facebook clean • Even linguistic annotators report stress & nightmares from reading news articles! 52
- 53. What about freedom? • Vision: empowering worker freedom: – work whenever you want for whomever you want • Risk: people being compelled to perform work – Digital sweat shops? Digital slaves? – Prisoners used for gold farming – We really don’t know (and need to learn more…) – Traction? Human Trafficking at MSR Summit’12 53
- 54. Conclusion • Crowdsourcing is quickly transforming practice in industry and academia via greater efficiency • Crowd computing is creating a new breed of applications, augmenting state-of-the-art automation (AI) with human computation to offer new capabilities and user experiences • By placing people at the center of this new computing model, we must confront important considerations beyond the technological 54
- 55. Thank You! Students: Past & Present – Catherine Grady (iSchool) – Hyunjoon Jung (iSchool) – Jorn Klinger (Linguistics) – Adriana Kovashka (CS) – Abhimanu Kumar (CS) ir.ischool.utexas.edu/crowd – Hohyon Ryu (iSchool) – Wei Tang (CS) – Stephen Wolfson (iSchool) Support – John P. Commons Fellowship – Temple Fellowship Matt Lease - [email protected] - @mattlease 55
- 56. REFERENCES & RESOURCES August 12, 2012 56
- 57. 2012 Conferences & Workshops • AAAI: Human Computation (HComp) (July 22-23) • AAAI Spring Symposium: Wisdom of the Crowd (March 26-28) • ACL: 3rd Workshop of the People's Web meets NLP (July 12-13) • AMCIS: Crowdsourcing Innovation, Knowledge, and Creativity in Virtual Communities (August 9-12) • CHI: CrowdCamp (May 5-6) • CIKM: Multimodal Crowd Sensing (CrowdSens) (Oct. or Nov.) • Collective Intelligence (April 18-20) • CrowdConf 2012 (October 23) • CrowdNet - 2nd Workshop on Cloud Labor and Human Computation (Jan 26-27) • EC: Social Computing and User Generated Content Workshop (June 7) • ICDIM: Emerging Problem- specific Crowdsourcing Technologies (August 23) • ICEC: Harnessing Collective Intelligence with Games (September) • ICML: Machine Learning in Human Computation & Crowdsourcing (June 30) • ICWE: 1st International Workshop on Crowdsourced Web Engineering (CroWE) (July 27) • KDD: Workshop on Crowdsourcing and Data Mining (August 12) • Multimedia: Crowdsourcing for Multimedia (Nov 2) • SocialCom: Social Media for Human Computation (September 6) • TREC-Crowd: 2nd TREC Crowdsourcing Track (Nov. 14-16) • WWW: CrowdSearch: Crowdsourcing Web search (April 17) 57
- 58. Surveys • Ipeirotis, Panagiotis G., R. Chandrasekar, and P. Bennett. (2009). “A report on the human computation workshop (HComp).” ACM SIGKDD Explorations Newsletter 11(2). • Alex Quinn and Ben Bederson. Human Computation: A Survey and Taxonomy of a Growing Field. In Proceedings of CHI 2011. • Law and von Ahn (2011). Human Computation August 12, 2012 58
- 59. 2013 Events Planned Research events • 1st year of HComp as AAAI conference • 2nd annual Collective Intelligence? Industrial Events • 4th CrowdConf (San Francisco, Fall) • 1st Crowdsourcing Week (Singapore, April) August 12, 2012 59
- 60. Journal Special Issues 2012 – Springer’s Information Retrieval (articles now online): Crowdsourcing for Information Retrieval – IEEE Internet Computing (articles now online): Crowdsourcing (Sept./Oct. 2012) – Hindawi’s Advances in Multimedia Journal: Multimedia Semantics Analysis via Crowdsourcing Geocontext August 12, 2012 60
- 61. 2011 Tutorials and Keynotes • By Omar Alonso and/or Matthew Lease – CLEF: Crowdsourcing for Information Retrieval Experimentation and Evaluation (Sep. 20, Omar only) – CrowdConf (Nov. 1, this is it!) – IJCNLP: Crowd Computing: Opportunities and Challenges (Nov. 10, Matt only) – WSDM: Crowdsourcing 101: Putting the WSDM of Crowds to Work for You (Feb. 9) – SIGIR: Crowdsourcing for Information Retrieval: Principles, Methods, and Applications (July 24) • AAAI: Human Computation: Core Research Questions and State of the Art – Edith Law and Luis von Ahn, August 7 • ASIS&T: How to Identify Ducks In Flight: A Crowdsourcing Approach to Biodiversity Research and Conservation – Steve Kelling, October 10, ebird • EC: Conducting Behavioral Research Using Amazon's Mechanical Turk – Winter Mason and Siddharth Suri, June 5 • HCIC: Quality Crowdsourcing for Human Computer Interaction Research – Ed Chi, June 14-18, about HCIC) – Also see his: Crowdsourcing for HCI Research with Amazon Mechanical Turk • Multimedia: Frontiers in Multimedia Search – Alan Hanjalic and Martha Larson, Nov 28 • VLDB: Crowdsourcing Applications and Platforms – Anhai Doan, Michael Franklin, Donald Kossmann, and Tim Kraska) • WWW: Managing Crowdsourced Human Computation – Panos Ipeirotis and Praveen Paritosh 61
- 62. 2011 Workshops & Conferences • AAAI-HCOMP: 3rd Human Computation Workshop (Aug. 8) • ACIS: Crowdsourcing, Value Co-Creation, & Digital Economy Innovation (Nov. 30 – Dec. 2) • Crowdsourcing Technologies for Language and Cognition Studies (July 27) • CHI-CHC: Crowdsourcing and Human Computation (May 8) • CIKM: BooksOnline (Oct. 24, “crowdsourcing … online books”) • CrowdConf 2011 -- 2nd Conf. on the Future of Distributed Work (Nov. 1-2) • Crowdsourcing: Improving … Scientific Data Through Social Networking (June 13) • EC: Workshop on Social Computing and User Generated Content (June 5) • ICWE: 2nd International Workshop on Enterprise Crowdsourcing (June 20) • Interspeech: Crowdsourcing for speech processing (August) • NIPS: Second Workshop on Computational Social Science and the Wisdom of Crowds (Dec. TBD) • SIGIR-CIR: Workshop on Crowdsourcing for Information Retrieval (July 28) • TREC-Crowd: Year 1 of TREC Crowdsourcing Track (Nov. 16-18) • UbiComp: 2nd Workshop on Ubiquitous Crowdsourcing (Sep. 18) • WSDM-CSDM: Crowdsourcing for Search and Data Mining (Feb. 9) 62
- 63. More Books July 2010, kindle-only: “This book introduces you to the top crowdsourcing sites and outlines step by step with photos the exact process to get started as a requester on Amazon Mechanical Turk.“ 63
- 64. Bibliography J. Barr and L. Cabrera. “AI gets a Brain”, ACM Queue, May 2006. Bernstein, M. et al. Soylent: A Word Processor with a Crowd Inside. UIST 2010. Best Student Paper award. Bederson, B.B., Hu, C., & Resnik, P. Translation by Interactive Collaboration between Monolingual Users, Proceedings of Graphics Interface (GI 2010), 39-46. N. Bradburn, S. Sudman, and B. Wansink. Asking Questions: The Definitive Guide to Questionnaire Design, Jossey-Bass, 2004. C. Callison-Burch. “Fast, Cheap, and Creative: Evaluating Translation Quality Using Amazon’s Mechanical Turk”, EMNLP 2009. P. Dai, Mausam, and D. Weld. “Decision-Theoretic of Crowd-Sourced Workflows”, AAAI, 2010. J. Davis et al. “The HPU”, IEEE Computer Vision and Pattern Recognition Workshop on Advancing Computer Vision with Human in the Loop (ACVHL), June 2010. M. Gashler, C. Giraud-Carrier, T. Martinez. Decision Tree Ensemble: Small Heterogeneous Is Better Than Large Homogeneous, ICMLA 2008. D. A. Grier. When Computers Were Human. Princeton University Press, 2005. ISBN 0691091579 JS. Hacker and L. von Ahn. “Matchin: Eliciting User Preferences with an Online Game”, CHI 2009. J. Heer, M. Bobstock. “Crowdsourcing Graphical Perception: Using Mechanical Turk to Assess Visualization Design”, CHI 2010. P. Heymann and H. Garcia-Molina. “Human Processing”, Technical Report, Stanford Info Lab, 2010. J. Howe. “Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business”. Crown Business, New York, 2008. P. Hsueh, P. Melville, V. Sindhwami. “Data Quality from Crowdsourcing: A Study of Annotation Selection Criteria”. NAACL HLT Workshop on Active Learning and NLP, 2009. B. Huberman, D. Romero, and F. Wu. “Crowdsourcing, attention and productivity”. Journal of Information Science, 2009. P.G. Ipeirotis. The New Demographics of Mechanical Turk. March 9, 2010. PDF and Spreadsheet. P.G. Ipeirotis, R. Chandrasekar and P. Bennett. Report on the human computation workshop. SIGKDD Explorations v11 no 2 pp. 80-83, 2010. P.G. Ipeirotis. Analyzing the Amazon Mechanical Turk Marketplace. CeDER-10-04 (Sept. 11, 2010) 64
- 65. Bibliography (2) A. Kittur, E. Chi, and B. Suh. “Crowdsourcing user studies with Mechanical Turk”, SIGCHI 2008. Aniket Kittur, Boris Smus, Robert E. Kraut. CrowdForge: Crowdsourcing Complex Work. CHI 2011 Adriana Kovashka and Matthew Lease. “Human and Machine Detection of … Similarity in Art”. CrowdConf 2010. K. Krippendorff. "Content Analysis", Sage Publications, 2003 G. Little, L. Chilton, M. Goldman, and R. Miller. “TurKit: Tools for Iterative Tasks on Mechanical Turk”, HCOMP 2009. T. Malone, R. Laubacher, and C. Dellarocas. Harnessing Crowds: Mapping the Genome of Collective Intelligence. 2009. W. Mason and D. Watts. “Financial Incentives and the ’Performance of Crowds’”, HCOMP Workshop at KDD 2009. J. Nielsen. “Usability Engineering”, Morgan-Kaufman, 1994. A. Quinn and B. Bederson. “A Taxonomy of Distributed Human Computation”, Technical Report HCIL-2009-23, 2009 J. Ross, L. Irani, M. Six Silberman, A. Zaldivar, and B. Tomlinson. “Who are the Crowdworkers?: Shifting Demographics in Amazon Mechanical Turk”. CHI 2010. F. Scheuren. “What is a Survey” (http://www.whatisasurvey.info) 2004. R. Snow, B. O’Connor, D. Jurafsky, and A. Y. Ng. “Cheap and Fast But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks”. EMNLP-2008. V. Sheng, F. Provost, P. Ipeirotis. “Get Another Label? Improving Data Quality … Using Multiple, Noisy Labelers” KDD 2008. S. Weber. “The Success of Open Source”, Harvard University Press, 2004. L. von Ahn. Games with a purpose. Computer, 39 (6), 92–94, 2006. L. von Ahn and L. Dabbish. “Designing Games with a purpose”. CACM, Vol. 51, No. 8, 2008. 65
- 66. Bibliography (3) Shuo Chen et al. What if the Irresponsible Teachers Are Dominating? A Method of Training on Samples and Clustering on Teachers. AAAI 2010. Paul Heymann, Hector Garcia-Molina: Turkalytics: analytics for human computation. WWW 2011. Florian Laws, Christian Scheible and Hinrich Schütze. Active Learning with Amazon Mechanical Turk. EMNLP 2011. C.Y. Lin. Rouge: A package for automatic evaluation of summaries. Proceedings of the workshop on text summarization branches out (WAS), 2004. C. Marshall and F. Shipman “The Ownership and Reuse of Visual Media”, JCDL, 2011. Hohyon Ryu and Matthew Lease. Crowdworker Filtering with Support Vector Machine. ASIS&T 2011. Wei Tang and Matthew Lease. Semi-Supervised Consensus Labeling for Crowdsourcing. ACM SIGIR Workshop on Crowdsourcing for Information Retrieval (CIR), 2011. S. Vijayanarasimhan and K. Grauman. Large-Scale Live Active Learning: Training Object Detectors with Crawled Data and Crowds. CVPR 2011. Stephen Wolfson and Matthew Lease. Look Before You Leap: Legal Pitfalls of Crowdsourcing. ASIS&T 2011. 66