Download to read offline
Uploaded byBigData_Europe
SC4 Hangout - Luigi Selmi, Transport pilot architecture
The document describes the SC4 Pilot Online Hangout which will test a new big data system for collecting and processing multisource transport data in Thessaloniki, Greece. The pilot will use Kafka and Flink to ingest taxi and Bluetooth data, integrate the data, process it using R and PostGIS, store it in Elasticsearch, and present it in Kibana. The components are containerized and the pilot will test deploying them on a 3 node Docker Swarm for fault tolerance and scalability. Current work involves setting up the swarm and future work will test the system.
![Ryan Betts [InfluxData] | InfluxDB Platform Performance | InfluxDays Virtual ...](https://cdn.slidesharecdn.com/ss_thumbnails/influxdaysna-ryan-201110022532-thumbnail.jpg?width=600ounds&width=560&fit=bounds)
SC4 Hangout - Luigi Selmi, Transport pilot architecture
- 1. SC4 Pilot OnlineHangout Luigi Selmi - FhG27 June 2016
- 2. Pilot Scenario Multisource datacollection for the provision of accurate info-mobility and advanced transport planning services in Thessaloniki ◎FCD Taxi ◎BT P2P Detectors
- 3. Present FCD Pipeline Storageand processing in MSSQL Server
- 4. Aims of theSC4 Pilot A new system with Big Data characteristics ◎Volume (Scalability and Fault-tolerance) ◎Velocity (Low latency) ◎Variety (Formats, Unstructured, Structured) ◎Easy Deployment
- 5.
- 6.
- 7. Semantic Lifting To improvethe quality of the data and make a better use of it, the semantics of the data will be made explicit at the ingestion phase using standard ontologies or creating new ones.
- 8. SC4 Pilot Components ◎DataIngestion: Kafka Producer (Flink Jobs) ◎Data Integration: Kafka ◎Data Processing: Flink + Rserve + PostGis ◎Data Storage: Elasticsearch ◎Data Presentation: Kibana ◎Pilot Pipeline
- 9.
- 10. SC4 Pilot GithubRepos 5 Projects on the BDE Github Repository ◎Pilot-sc4-kafka-producer ◎Pilot-sc4-docker-r ◎Pilot-sc4-mapmatcher (for testing) ◎Pilot-sc4-flink-kafka-consumer ◎Pilot-sc4-pipeline (docker-compose)
- 11. BDE Components The BDEplatform provides ◎Docker images for all the components ◎A tool for integration and initialization based on Docker Swarm ◎Integrated UI
- 12.
- 13. Current and FutureWork 1. Set up a Docker Swarm for the pilot in 3 nodes (real and VMs) using the dockerized version of all the components with Init Daemon and BDE Integrated Dashboard 2. Test the pilot swarm for fault-tolerance and scalability