Download as PDF, PPTX
Uploaded byWSO2
Stream Processing in Action
The document discusses the WSO2 Stream Processor, an open-source, cloud-native analytics tool designed for real-time insights in digital businesses. It outlines the processor's capabilities such as event processing, data transformation, and machine learning, as well as its applications in industries like finance, retail, and operations analytics. Various use cases demonstrate its effectiveness in scenarios like fraud detection, recommendations, and fleet management.
More Related Content
![[WSO2Con EU 2018] Hybrid Cloud API Management - API Microgateways Anywhere](https://cdn.slidesharecdn.com/ss_thumbnails/hybridapimanagement-wso2coneu2018-sameerajayasoma-181114093328-thumbnail.jpg?width=640&height=640&fit=bounds)
![[WSO2Con EU 2018] WSO2 API Manager - Why, What, How, and What's Next](https://cdn.slidesharecdn.com/ss_thumbnails/eucon-2018-wso2apimanagerwhywhathowandwhatsnext-181114092949-thumbnail.jpg?width=640&height=640&fit=bounds)
![[WSO2 Summit Americas 2020] Creating Smart Endpoints Using Integration Micros...](https://cdn.slidesharecdn.com/ss_thumbnails/wso2summitamericas2020creatingsmartendpointsusingintegrationmicroservices-200826201750-thumbnail.jpg?width=640&height=640&fit=bounds)
![[WSO2 API Day Toronto 2019] Cloud-native Integration for the Enterprise](https://cdn.slidesharecdn.com/ss_thumbnails/wso2apidaytoronto2019cloud-nativeintegrationfortheenterprise-191113063107-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Stream Processing in Action
![[WSO2Con EU 2018] The Rise of Streaming SQL](https://cdn.slidesharecdn.com/ss_thumbnails/1-181113084942-thumbnail.jpg?width=640&height=640&fit=bounds)
![[WSO2Con EU 2018] Streaming SQL in the Real World](https://cdn.slidesharecdn.com/ss_thumbnails/wso2coneu2018streamingsqlintherealworld-181113090808-thumbnail.jpg?width=640&height=640&fit=bounds)
![[WSO2Con EU 2017] Deriving Insights for Your Digital Business with Analytics](https://cdn.slidesharecdn.com/ss_thumbnails/ftl1u7i2sbihhixstme6-signature-6abac760590306a9c092f3ba66249d9a805ee6b52c9eb804a80eabb33b95c5ef-poli-171106133401-thumbnail.jpg?width=640&height=640&fit=bounds)
![[Roundtable] Choreo - The AI-Native Internal Developer Platform as a Service](https://cdn.slidesharecdn.com/ss_thumbnails/choreo-deck-250328074645-511dded7-thumbnail.jpg?width=640&height=640&fit=bounds)
Stream Processing in Action
- 1. Stream Processing inAction Webinar Nirmal Fernando Senior Lead Solutions Engineer, WSO2 Nov, 2018
- 2. Contents • Introduction • WSO2Stream Processor Overview • Industry Use Cases • Demo
- 3. Diverse Industries ->Unique Challenges http://www.solgenie.com/industries/
- 4. “The price oflight is less than the cost of darkness.” - Arthur C. Nielsen, Market Researcher & Founder of ACNielsen
- 5. Value of InsightsDegrade Fast http://www.history.com/news/ask-history/who-determined-the-speed-of-light
- 6.
- 7. Stream Processor Core WSO2ANALYTICS OFFERING 7 ▪ Consumes events, and publish alerts and summarizations to and from various enterprise systems. ▪ Event Processor Core with Streaming Complex Event Processing, Incremental Time Series Aggregations, and Streaming Machine Learning. ▪ Stream Processing Functionalities via Extension Store ▪ High Available and Scalable Analytics Fabric ▪ Prebuilt and custom analytics solutions Events JMS, Thrift, SMTP, HTTP, MQTT, Kafka Analytics Fabric Complex Event Processing Incremental Time Series Aggregation Machine Learning Extension Store FinancialandBanking Analytics RetailAnalytics LocationAnalytics OperationalAnalytics SmartEnergyAnalytics Custom Analytics Solutions ... Solutions
- 8.
- 9. WSO2 Stream Processor Anopen source, cloud-native analytics product optimized to create real-time, actionable insights for agile digital businesses. 9
- 10. WSO2 Stream Processor(WSO2 SP)
- 11. 1. Data collection 2.Data cleansing 3. Data transformation 4. Data enrichment 5. Data summarization 6. Rule processing 7. Machine Learning & Artificial Intelligence 8. Data pipelining 9. Data Publishing 10. On demand processing 11. Data Presentation Stream Processing Patterns
- 12. ● Lightweight, lean,and high performance ● Best suited for ○ Streaming Data Integration, ○ Streaming Analytics ● Streaming SQL & graphical drag-and-drop editor ● Multiple deployment options ○ Process data at the edge (java, python) ○ Micro Stream Processing ○ High availability with 2 nodes ○ Highly scalable distributed deployments ● Support for streaming ML & Long running aggregations ● Monitoring tools and citizen integration options WSO2 Stream Processor
- 13. • Source andSinks – HTTP, Kafka, TCP, Email, JMS, File, Rabbitmq, MQTT, Web-Socket, Twitter, Amazon SQS • Message Formats – JSON, XML, Text, Binary, Key-value, CSV • Data Stores – RDBMS, Solr, MongoDB, HBase, Cassandra, Elasticsearch, Hazelcast, Redis Supported connectors
- 14. (Streaming) Machine Learning ▪Running PMML Models for predictions - Built via Apache Spark MLlib, Python, H2O.ai (for deep learning algos) or R - Export as PMML - Load precreated PMML Model into Siddhi to predict in Realtime ▪ Supporting native models for predictions - Spark MLlib Models, Java based Tensorflow Models ▪ Online Learning and predictions - Regression Analytics - Data Classification - K-Means Clustering - Markov Models - Anomaly Detections - … more on the way
- 15. Supported Extensions • Geographical processing • NLP • Graph • Reordering • Timeseries • Import Machine Learning models – PMML, TensorFlow, etc • Streaming Machine Learning – Clustering, Classification, Regression – Makove Models, Anomaly detection, etc. • .. more Image : http://www.weewatch.com/wp-content/uploads/2016/01/28fc45d.jpg 60+ https://store.wso2.com/store/assets/analyticsextension/list
- 16. High Availability with2 Nodes • 2 node minimum HA – Process upto 100k events/sec – While most other stream processing systems need around 5+ nodes • Does not require Kafka • Incremental state persistence and recovery • Multi data center support Stream Processor Stream Processor Event Sources Dashboard Notification Invocation Data Source Siddhi App Siddhi App Siddhi App Siddhi App Siddhi App Siddhi App Event Store
- 17. • Exactly-once processing • Faulttolerance • Highly scalable • No back pressure • Distributed via annotations • Native support for Kubernetes Scaling with Distributed Deployment
- 18.
- 19.
- 20. Use Case 1- Fraud Detection • Detecting fraud via known patterns using generic rules • Detecting unknown types of fraud via machine learning • Detecting rare activity sequences using Markov Modeling • Reduce false alarms using fraud scoring • Caught them in the act - what next? Demo: https://goo.gl/xo6Wf5
- 21. Use Case 2- Risk Management • Finding real-time Value at Risk (VaR) – Historical simulation – Variance-covariance – Monte Carlo simulation
- 22. • Identifying FrontRunning with Patterns Use Case 3 - Stock Market Surveillance Hey Jude, Mike is going to buy large qty of ABC at $21. You better buy now! Great! I bought. ABC is just $18.9 right now! Trade 1 Followed by Trade 2 Jude sells to Mike at $21. Broker: Bob Client Client MikeJude
- 23. Use Case 3- Stock Market Surveillance • Identifying Pump with Regression • Identifying signs of Insider Dealing • Model “Perfect Trader” in order to detect fraudsters
- 24.
- 25. Use Case 1- Recommendations • Recommendations based on the buying products • Recommendations based on the buying history of the customer • Seasonal recommendations • Contextual, intelligent recommendations
- 26. Use Case 2- Ad Optimization • Display personalized advertisements on online shopping stores – by identifying person’s living location – by identifying person’s buying history – by identifying person’s interest • Optimize displaying advertisements on shopping stores – what impact the Ad made? Is it worth the cost?
- 27. • Send personalizedinformation when a customer enters the store • If a customer is spending a considerable amount of time near a shop, suggest offers, send an agent to him etc. Use Case 3 - Proximity Marketing
- 28. • Detect thatthe customer leaves the store and send him a gift coupon • Number of customers & employees in each floor (heat maps) • Number of customers & average time spent per product • Shopping path, purchased items, average time spent by each customer • Overall statistics about given and used offers Use Case 3 - Proximity Marketing
- 29.
- 30. Use Case -Fleet Management • Real-time monitoring - where is your fleet now?, Visualize your fleet • Geo-fencing based alerts - get alerted if a driver exceeds a defined speed limit within an interactively given geo area • Predicting travel times - historical data can be used to build a machine learning model in order to predict travel times in advance and alert the subscribers
- 31.
- 32. Use Cases • Collectdata from different stages of business operations • Tracking the progress of your business operations • Detect Service Level Agreement violations at each stage and generate alerts • Visualize your business operations real-time • Generate alerts and notify the subscribers
- 33.
- 34.
- 35. Download and Tryout WSO2Stream Processor https://wso2.com/analytics-and-stream-processing/ Documentation
- 36. Further Reading ● WSO2Stream Processor: Making Real-time Stream Processing Available to the Masses ● Making Real-Time Applications Simpler with WSO2 Stream Processor ● How to use a Stream Processor as a Notification Manager? ● Synchronous Request-Response based Real-time Processing with WSO2 SP ● Distributed Stream Processing with WSO2 SP
- 37.