Lesson3 esa summer_school_brovelli
0 likes290 views
This document discusses several citizen science and crowdsourcing projects conducted by Politecnico di Milano's Geomatics Laboratory. It summarizes three main applications: 1) Sensing slow mobility in Lombardy region by analyzing geotagged social media data to identify popular locations, 2) A webGIS platform to collect and visualize European land use/cover maps and allow users to add photos, 3) Several applications to visualize and analyze social media and telecommunications big data from Milan, including filtering data by date and land use and visualizing patterns over time.
Lesson3 esa summer_school_brovelli
- 1. GEO Laboratory Citizen Generated Content and FOS Participative Platforms: geocrowdsourced data Politecnico di Milano, DICA – GEO Laboratory Maria Antonia Brovelli
- 2. 2 http://www.internetlivestats.com/ ● 7,290 Tweets sent in 1 second ● 734 Instagram photos uploaded in 1 second ● 1,142 Tumblr posts in 1 second ● 2,214 Skype calls in 1 second ● 36,800 GB of Internet traffic in 1 second ● 55,467 Google searches in 1 second ● 127,793 YouTube videos viewed in 1 second ● 2,510,411 Emails sent in 1 second Geocrowdsourcing (Big Geo Data) Retrieved July 15, 2002 from http://www.internetlivestats.com/one-second/
- 3. 3 ✔ http://onemilliontweetmap.com/ ✔ http://www.flickr.com/map Location based social network
- 4. 4 Citizen- Generated Geographic Information Source: Craglia, M., & Granell Canut, C. (2014). Citizen Science and Smart Cities. Contributed Geographic Information (CGI) refers to geographic information "that has been collected without the immediate knowledge and explicit decision of a person using mobile technology that records location”
- 5. 5 Applications ✔ Sensing Slow Mobility and Interesting Locations for Lombardy Region (Italy): a Case Study Using Pointwise Geolocated Open Data. An approach for collecting, unifying and analysing pointwise geolocated open data available from different sources with the aim of identifying the main locations and destinations of slow mobility activities. ✔ Land Coverage Platform. A WebGIS platform designed to publish the available land use and land cover maps of Europe at continental scale, were users can add to the platform photos from popular photo sharing services, in order to have a visual assessment of the available land coverages based on other user-generated contents available on the Internet.
- 6. 6 Applications ✔ Sensing the City. A series of applications and procedures for the visualization and analysis of Social Media and Telecommunications Data (user- generated mobile network traffic). ✔ Sensing the city: calls and tweets. A Web application for visualizing the number of calls exchanged between callers located in Milan and receivers located in other provinces in Italy ✔ Social media data management with Rasdaman: Web application for testing the Web Coverage Processing Service (WCPS) OGC standard provided by Rasdaman ✔ Big data to netCDF: Web application for creating netCDF files from time series telecommunications data ✔ Visualizing social media data with EST-WA: EST-WA is a tool developed by GEOlab @Polimi for representing 4D variables (3D location of the variable values at different times) provided in netCDF format ✔ Relationships Between Telecommunications and Weather Data Meteorological measurements of precipitation and temperature, as well as user-generated mobile network traffic is being analysed on a common space-time basis with a Two-Way Analysis of variance ANOVA on the city of Milan
- 7. 7 Sensing Slow Mobility and Interesting Locations for Lombardy Region (Italy): a Case Study Using Pointwise Geolocated Open Data. Applications - 1
- 8. 8 Sensing Slow Mobility - 1 Aim of the study The analysis purpose is to identify attractive locations and destinations of slow mobility activities (e.g. hiking, biking, etc.) within Lombardy Region (Italy) according to user’s reported activities
- 9. 9 Sensing Slow Mobility - 2 Selected CGI platforms: ✔ Wikiloc (http://www.wikiloc.com): specialized platform for sharing and gathering insights on outdoor activities. Content is mainly GPX tracks. The collection is allowed only through manual download ✔ Flickr (https://www.flickr.com), Twitter (https://twitter.com) and Foursquare (https://it.foursquare.com): general purpose platforms that allow sharing different kind of content (pictures, check-ins, text messages, etc.). APIs are available to obtain content metadata in JSON format
- 10. 10 Sensing Slow Mobility - 3 Data Collection and Storing
- 11. 11 Application Programming Interface connections Information on how to build the requests: ✔ Flickr: https://www.flickr.com/services/api/. ✔ Twitter: https://dev.twitter.com/overview/documentation ✔ Foursquare: https://developer.foursquare.com/ ✔ Check : https://github.com/mazucci/geocollect for all the information on how to connect to the APIs Sensing Slow Mobility - 4
- 12. 12 Sensing Slow Mobility - 5 Data filtering ✔ Tracks speed was calculated with a python script: the difference between position of the beginning and end of the track gave the distance traveled, same approach for the timestamp gives time traveled. With distance and time the speed was calculated.
- 13. 13 Sensing Slow Mobility - 6 Data filtering
- 14. 14 Sensing Slow Mobility - 7 Data Analysis Purpose: Identification of the most visited locations by looking for atypical spatial patterns as well as concentration of user-generated content within the study region ✔ Comparison between different platforms ✔ Comparison between user activities during weekdays and weekends ✔ Comparison between different spatial analysis techniques Selected techniques: ✔ Concentration Maps ✔ Hot-spot Analysis (Exploratory Spatial Data Analysis)
- 15. 15 Sensing Slow Mobility - 8 Techniques overview Concentration Maps ✔ Interpolated surface showing the density of occurrence of sparse point trough a color gradient or patches ✔ Requires to define an interpolating function and influence radius to compute the density surface Hot-Spot Analysis ✔ Underlines where locational similarity is matched by attribute correlation in a spatial dataset by mean of statistical analysis (i.e. Getis-Ord GI* local statistic) ✔ Requires sparse points aggregation into representative points for any parcels of the study area as well as the identification of distance threshold to compute the local statistics in a defined region surrounding any points
- 16. 16 Sensing Slow Mobility - 9 Concentration maps
- 17. 17 Sensing Slow Mobility - 10 Hot-Spot maps
- 18. 18 Results and Discussions ✔ Wikiloc data better describes locations for slow mobility activities with respect to the other platforms ✔ Flickr, Foursquare and Twitter data shows redundant places of interest across the region focused on the main cities (which are reasonably popular locations) ✔ Hot-spots concentrate around some of the main cities as well as in the alpine area. During weekend a strong hot-spot concentration appears all along the subalpine area and lakes. ✔ Cold-spots are located mainly in the plain area. ✔ Concentration maps retrace closely the patters highlighted by the hot-spot maps Further improvements ✔ Results may be improved for the general purpose platforms performing specific data filtering (e.g. through keywords, hashtags, venues category etc.) ✔ Inclusion of Explanatory Spatial Data Analysis tools into QGIS* Sensing Slow Mobility - 11 *Oxoli D., Zurbarán M.A., Shaji S., Muthusamy A.K. (2016) Hotspot analysis: a first prototype Python plugin enabling exploratory spatial data analysis into QGIS. PeerJ Preprints 4:e2204v2 https://doi.org/10.7287/peerj.preprints.2204v2
- 19. 19 Land Coverage Platform. A WebGIS platform designed to publish the available land use and land cover maps of Europe Applications - 2
- 20. 20 Land Coverage Platform - 1 Aim of the study Implementing an open-source WebGIS aiming to collect, visualize, analyze and compare the land use and land cover datasets freely available for the Europe area in a single platform. Research topics: ✔ Comparison between the LULC datasets in order to detect similarities and discrepancies ✔ Assessment of the classification quality of the LULC datasets
- 21. 21 Selected CGI platforms: ✔ Geograph (http://www.geograph.org.uk): a project limited to UK and Ireland which aims to collect geographically representative photographs at each node of a square grid with 1 km side ✔ Flickr (http://flickr.com): probably the most popular image hosting website for sharing personal photos. ✔ Panoramio (http://www.panoramio.com): a popular photo sharing website owned by Google. User submitted photos are published on the platform upon acceptance. The collection currently counts over 90 million photos Land Coverage Platform - 2
- 22. 22 Application Architecture Land Coverage Platform - 3
- 23. 23 Application Programming Interface connections Information on how to build the requests: ✔ Flickr: https://www.flickr.com/services/api/. ✔ Panoramio: http://www.panoramio.com/api/widget/api.html ✔ Geograph: http://www.geograph.org.uk/help/api Example of a Geograph request url: var geograph_URL = "http://api.geograph.org.uk/api/photo/" + obj + "/" + service_Photo.key + "?output=json"; That becomes: http://api.geograph.org.uk/api/photo/4342537/e79cb167fb?output=json Land Coverage Platform - 4
- 24. 24 EU-LULC platform: client overview Land Coverage Platform - 5 Prototype available at http://131.175.143.84/LULC/
- 25. 25 Results and Discussions ✔ EU-LULC WebGIS is entirely built on open source infrastructure and open standards ✔ It enables the visualization and visual comparison of the available LULC maps of Europe Further improvements ✔ Add other LULC available datasets ✔ Allow the upload of user data (raster and vector maps, photos) ✔ Improve the platform with processing functionalities to quantitatively compare the LULC maps: ✔ Compute statistics on land cover classes distribution for user-defined areas ✔ Assess land cover changes over time ✔ Evaluate the accuracy of a LULC map through the confusion matrix approach Land Coverage Platform - 6
- 26. 26 Sensing the City. A series of applications and procedures for the visualization and analysis of Social Media and Telecommunications Data (user-generated mobile network traffic). Applications - 3
- 27. 27 https://dandelion.eu/datamine/open-big-data/ Sensing the City - 1 Geo Big Data: Milano GRID ✔ Two months of data, with a temporal step of 10 minutes ✔ Grid of 100 x 100 cells with size = 235 m
- 28. 28 Sensing the City - 2 Geo Big Data: Milano GRID ✔ Received SMS: a Call Detail Record (CDR) is generated each time a user receives an SMS ✔ Sent SMS: a CDR is generated each time a user sends an SMS ✔ Incoming Calls: a CDR is generated each time a user receives a call ✔ Outgoing Calls: CDR is generated each time a user issues a call ✔ Internet: a CDR is generate each time : ➔ a user starts an internet connection ➔ a user ends an internet connection ➔ during the same connection one of the following limits is reached: ➔ 15 minutes from the last generated CDR ➔ 5 MB from the last generated CDR ✔ Geolocalized Twetts (Anonymized twitter users)
- 29. 29 Sensing the City - 3
- 30. 30 Sensing the City - 4 http://landcover.como.polimi.it/BGDV/
- 31. 31 Sensing the City - 5 Filtering with date and land coverage classes
- 32. 32 Sensing the City - 6 http://landcover.como.polimi.it/socialmedia_rasdaman/
- 33. 33 Sensing the City - 7 http://landcover.como.polimi.it/BigNetCDF/
- 34. 34 Sensing the City - 8 Filtering Interactive multidimensional web visualisation - ESTWA Web World Wind
- 35. 35 http://landcover.como.polimi.it//BigNetCDF/cumulative.php Received SMS from Friday, December 13th to Thursday, December 19th for all Milano grid cells Sensing the City - 9
- 36. 36 ✔ Weather data comes from ARPA Lombardia's mesoscale meteorological network (http://www2.arpalombardia.it/siti/arpalombardia/meteo/osservazioniedati /datitemporeale/rilevazioni-in-tempo-reale/Pagine/Rilevazioni-in-tempo- reale.aspx) ✔ Land use data is being considered as well, taken from the Global Land Cover 30m (www.globallandcover.com) ✔ Data processing is being made with GIS Open tools such as : ➔ GRASS GIS (https://grass.osgeo.org/) for preprocessing, basic statistics and filtering ➔ QGIS (http://www.qgis.org/) for data visualization ➔ R (https://www.r-project.org/) for advanced statistics analysis ➔ Python Pandas, Scipy and Numpy libraries (https://www.python.org/) for advanced statistics analysis ✔ Data storage is being explored with MongoDB (www.mongodb.com) and RASDAMAN (http://www.rasdaman.com/ Sensing the City - 10
- 37. 37 Geomatics Laboratory, Politecnico di Milano – Como Campus Data Processing Sensing the City - 4 Results next time!Results next time!
- 38. 38 Contacts Thanks for your attention! Politecnico di Milano Laboratorio di Geomatica – Polo Territoriale di Como Via Valleggio 11, 22100 Como (Italy) [email protected] Thanks to all people of my team contributing on these topics: Carolina Arias, Eylul Kilsedar, Marco Minghini, Monia Molinari, Daniele Oxoli, Marco Pelucchi, Gabriele Prestifilippo, Giorgio Zamboni, Mayra Zurbaran