Download to read offline
Uploaded byHarvardComms
Social Media Academy 2016 Keynote Presentation, John Wihbey
This document discusses the challenges faced by Northeastern University undergraduates analyzing Boston police homicide data over the decades, highlighting numerous inconsistencies, missing information, and format errors in the dataset. Students engaged in remedial efforts, including data cleaning and visualization projects, to explore temporal patterns and other insights. The case study showcases both the complexities of working with flawed data and the educational value of hands-on experience in digital storytelling and data analysis.
More Related Content
Similar to Social Media Academy 2016 Keynote Presentation, John Wihbey
Social Media Academy 2016 Keynote Presentation, John Wihbey
- 1. Defining a DigitalStorytelling Discipline: Learning, Skills, and Knowledge John Wihbey Northeastern University @wihbey
- 16. Case study: Northeastern University undergradsworking on Boston Police Department data, “as is” - in a general digital skills course
- 17. Murder data fromthe 1960s City of Boston - Homicide data obtained through public records request
- 18. Text text text Murderdata from the 1990s City of Boston - Homicide data obtained through public records request
- 19. Murder data fromthe 2000s City of Boston - Homicide data obtained through public records request
- 20.
- 21.
- 22. List of problems/errorsin structure and format of homicide data 1. Inconsistencies in case column, e.g. “01/06” vs “ ’09/06 ” 2. No indication of meaning of red text 3. No key for case column IDs 4. Different text formats/styles for entire rows and cells 5. Inconsistent descriptions of intersection addresses, e.g. “Washington @ Cedar St” vs “Willowood & Woodrow Ave” vs “Shawmut Ave. / Dwight” 6. No key for weapon codes 7. Race and gender are collapsed into single column 8. No codes for race/gender (race: “W”, “B”, gender: “M”, “F”, “H”) 9. Some R/G codes are “W/H/M”, “B/N/H” making it impossible to systematically split columns into 2 using the only delimiting character (/) 10. Some R/G codes have NO delimiter (e.g., 2009 sheet), so cannot split at all 11. Data for 2007 and later have two additional columns not in 2006: “defendant” and “DOA” (no indication of what DOA means) 12. Some rows have merged cells 13. Some merged cells have multiple values 14. Missing data/empty cells – what do these mean?
- 23. List of problems/errorsin structure and format of homicide data (cont.) 15. Location data is incomplete – no zip code information and Boston assumed as city (except in cases where “Dor” is appended at end of address) 16. Only first couple of sheets have column header information; column headers have to be assumed for remaining ones to follow those with labeled headers 17. Mysterious unexplained extra characters in date columns (e.g., (w) and xxx) 18. Inconsistent syntax for times: 12:00am, 7:10pm, 02:16am, 2:56hrs, 15oo hrs (Letter “o” instead of number 0), 1:49 AM, 7:24 PM, 21:25, 21:32 PM, 12:39P.M., 2:22p.m., 1:49:pm, 12;24AM, :14 am 19. Inconsistent syntax for dates: “07/24/2006” vs “2006/7/24” vs “6/31/64” vs “08/31/06” 20. Inconsistent syntax for age: “1’7” 21. Sheets for 2012/13/14 have new columns not in previous sheets 22. Motive/Relation columns look identical but are not both present in all sheets, impossible to know which labels are which in those sheets without column headers 23. Simple spelling errors: “Tauma” 24. Inconsistent coding: Unk, UNK, unk 25. Unexplained “DV” column that only appears in 2013 26. No explanation for meaning of row breaks – are these separating data rows into groups of some sort? Are these data that one existed but were removed? 27. Multiple columns with same (non-unique) headers – “R/G”, “Age”, “DOB” for both VICTIM and DEFENDANTS 28. Inconsistent district labels and squadron personnel names 29. For cells with multiple data/names in cell merged column, have to assume respective values in adjacent cells are provided in same order
- 24. Existential experience of: #datafail& #GIGO risk
- 25. Good student requestsfor clarification
- 26. Many noble studentattempts at cleaning, analysis, exploration, viz:
- 27. Data viz usingPlot.ly
- 28. Data viz usingCarto
- 29.
- 31. Fun with linegraphs - an attempt to look at time-of-day patterns
- 32. Experiments in vizfor exploratory purposes
- 33. D'où Venons Nous/ Que Sommes Nous / Où Allons Nous - Paul Gauguin, 1897
- 36.
- 37.
- 39.
- 41.
- 43.
- 44. G. Chaucer, TheCanterbury Tales (courtesy: library.arizona.edu) 1400s
- 52.
- 53.
- 54.
- 55.
- 56.
- 57.
- 69. Pew Research Center PolarizedCrowd: Two large dense groups with little interconnection
- 70. Pew Research Center TightCrowd: Highly interconnected group with few isolated participants
- 71. Pew Research Center Brandclusters: Products, services, celebrities discussed by disparate persons
- 72. Pew Research Center Communityclusters: Popular topics attracting multiple smaller groups
- 73. Pew Research Center Broadcastnetworks: Media-centric, with audience proliferating information
- 74. Pew Research Center Supportnetwork: Customer complaints, with hub-and-spoke dynamics
- 75. Six degrees -Wikimedia
- 76.
- 77. Nebraska local politicians;network graph - Matt Waite http://www.poynter.org/2013/how-to-visually-explore-local- politics-with-network-graphs/218543/
- 78. Chicago community -homicide network (Andrew Papachristos et al.)
- 79.
- 80. Gary King, etal, IQSS, Harvard
- 81. Global supply chain,Sourcemap.com
- 82.
- 89. D'où Venons Nous/ Que Sommes Nous / Où Allons Nous - Paul Gauguin, 1897
- 92.