Download as PDF, PPTX
![/search Format
Request
{
"target" : "select metric",
"refId" : "E"
}
Response
[
"Metric Name 1",
"Metric Name2",
]
An array of strings
12](https://image.slidesharecdn.com/makingsenseofyourdata-jug-170522045234/85/Making-sense-of-your-data-jug-12-320.jpg)
![/annotations Format
Request
{ "annotation" : {
"name" : "Test",
"iconColor" : "rgba(255, 96, 96, 1)",
"datasource" : "Simple Example DS",
"enable" : true,
"query" : "{"name":"Timeseries A"}" },
"range" : {
"from" : "2016-06-13T12:23:47.387Z",
"to" : "2016-06-13T12:24:19.217Z" },
"rangeRaw" : {
"from" : "2016-06-13T12:23:47.387Z",
"to" : "2016-06-13T12:24:19.217Z"
} }
Response
[ { "annotation": {
"name": "Test",
"iconColor": "rgba(255, 96, 96, 1)",
"datasource": "Simple Example DS",
"enable": true,
"query": "{"name":"Timeseries A"}" },
"time": 1465820629774,
"title": "Marker",
"tags": [
"Tag 1",
"Tag 2" ] } ]
13](https://image.slidesharecdn.com/makingsenseofyourdata-jug-170522045234/85/Making-sense-of-your-data-jug-13-320.jpg)
![/query Format
Request
{ "panelId" : 1,
"maxDataPoints" : 1904,
"format" : "json",
"range" : {
"from" : "2016-06-13T12:23:47.387Z",
"to" : "2016-06-13T12:24:19.217Z" },
"rangeRaw" : {
"from" : "2016-06-13T12:23:47.387Z",
"to" : "2016-06-13T12:24:19.217Z" },
"interval" : "20ms",
"targets" : [ {
"target" : "Time series A",
"refId" : "A" },] }
Response
[ { "target":"Timeseries A",
"datapoints":[
[1936,1465820629774],
[2105,1465820632673],
[4187,1465820635570],
[30001,1465820645243] },
{ "target":"Timeseries B",
"datapoints":[ ] }
]
14](https://image.slidesharecdn.com/makingsenseofyourdata-jug-170522045234/85/Making-sense-of-your-data-jug-14-320.jpg)
Uploaded byGerald Muecke
Making sense of your data jug
The document details the development of a custom MongoDB datasource for Grafana using Vert.x, which addresses the challenges of time series data visualization. It discusses the architecture of Grafana, the benefits of using Vert.x for high performance and scalability, and outlines the steps to implement an efficient datasource that optimizes data queries. Key takeaways emphasize Grafana's utility in data visualization and the reactive nature of Vert.x for creating responsive applications.
More Related Content
Similar to Making sense of your data jug
![Getting Started with Apache Spark: Big Data Made Simple [Free Meetup]](https://cdn.slidesharecdn.com/ss_thumbnails/apachesparkgettingstarted-260203175547-8361bcc3-thumbnail.jpg?width=640&height=640&fit=bounds)
Making sense of your data jug
- 1. Making Sense of yourData BUILDING A CUSTOM MONGODB DATASOURCE FOR GRAFANA WITH VERTX 1
- 2. About me ITConsultant & Java Specialist at DevCon5 (CH) Focal Areas Tool-assisted quality assurance Performance (-testing, -analysis, -tooling) Operational Topics (APM, Monitoring) Twitter: @gmuecke 2
- 3. The Starting Point Customer stored and keep response time measurement of test runs in a MongoDB Lots of Data Timestamp & Value No Proper Visualization 3
- 4. What are timeseriesdata? a set of datapoints with a timestamp and a value time value
- 5. What is MongoDB? MongoDB NoSQL database with focus on scale JSON as data representation No HTTP endpoint (TCP based Wire Protocol) Aggregation framework for complex queries Provides an Async Driver 5
- 6. What is Grafana? A Service for Visualizing Time Series Data Open Source Backend written in Go Frontend based on Angular Dashboards & Alerts
- 7. Grafana Architecture 7 GrafanaServer • Implemented in GO • Persistence for Settings and Dashboards • Offers Proxy for Service Calls Browser Angular UI Data Source Data Source Plugin... Proxy DB DB
- 8. Datasources for Grafana8 Grafana Server • Implemented in GO • Persistence for Settings and Dashboards • Offers Proxy for Service Calls Browser Datasource Angular UI Data Source Plugin • Angular • JavaScript HTTP
- 9. Connect Angular Directlyto Mongo? 9
- 10. From 2 Tierto 3 Tier 10 Grafana (Angular) Mongo DB Grafana (Angular) Mongo DB Datasource Service HTTP Mongo Wire Protocol
- 11. Start Simple SimpleJsonDatasource (Plugin) 3ServiceEndpoints /search Labels – names of available timeseries /annotations Annotations – textual markers /query Query – actual time series data 11 https://github.com/grafana/simple-json-datasource
- 12. /search Format Request { "target" :"select metric", "refId" : "E" } Response [ "Metric Name 1", "Metric Name2", ] An array of strings 12
- 13. /annotations Format Request { "annotation": { "name" : "Test", "iconColor" : "rgba(255, 96, 96, 1)", "datasource" : "Simple Example DS", "enable" : true, "query" : "{"name":"Timeseries A"}" }, "range" : { "from" : "2016-06-13T12:23:47.387Z", "to" : "2016-06-13T12:24:19.217Z" }, "rangeRaw" : { "from" : "2016-06-13T12:23:47.387Z", "to" : "2016-06-13T12:24:19.217Z" } } Response [ { "annotation": { "name": "Test", "iconColor": "rgba(255, 96, 96, 1)", "datasource": "Simple Example DS", "enable": true, "query": "{"name":"Timeseries A"}" }, "time": 1465820629774, "title": "Marker", "tags": [ "Tag 1", "Tag 2" ] } ] 13
- 14. /query Format Request { "panelId": 1, "maxDataPoints" : 1904, "format" : "json", "range" : { "from" : "2016-06-13T12:23:47.387Z", "to" : "2016-06-13T12:24:19.217Z" }, "rangeRaw" : { "from" : "2016-06-13T12:23:47.387Z", "to" : "2016-06-13T12:24:19.217Z" }, "interval" : "20ms", "targets" : [ { "target" : "Time series A", "refId" : "A" },] } Response [ { "target":"Timeseries A", "datapoints":[ [1936,1465820629774], [2105,1465820632673], [4187,1465820635570], [30001,1465820645243] }, { "target":"Timeseries B", "datapoints":[ ] } ] 14
- 15. Structure of theSource Data { "_id" : ObjectId("56375bc54f3c4caedfe68aca"), "t" : { "eDesc" : "some description", "eId" : "56375ae24f3c4caedfe68a07", "name" : "some name", "profile" : "I01", "rnId" : "56375b694f3c4caedfe68aa0", "rnStatus" : "PASSED", "uId" : "anonymous" }, "n" : { "begin" : NumberLong("1446468494689"), "value" : NumberLong(283) } } 15
- 16. Custom Datasource Shouldbe Lightweight Fast / Performant Simple 16
- 17. Microservice? Options forimplementation Java EE Microservice (i.e. Wildfly Swarm) Springboot Microservice Vert.x Microservice Node.js ... 17
- 18. The Alternative Options Node.js Single Threaded Child Worker Processes Javascript Only Not best-choice for heavy computation Spring / Java EE Multithreaded Clusterable Java Only Solid Workhorses, cumbersome at times 19
- 19. Why Vert.x? HighPerformance, Low Footprint Asynchronous, Non-Blocking Actor-like Concurrency Event & Worker Verticles Message Driven Polyglott Java, Groovy, Javascript, Scala … Scalable Distributed Eventbus Multi-threaded Event Loops 20
- 20.
- 21. Asynchronous non-blocking vs Synchronousblocking 23 © Fritz Geller-Grimm © Dontworry
- 22. Event Loop 24 Photo:Andreas Praefcke
- 23. Event Loop andVerticles 25 Photo: RokerHRO 3rd Floor, Verticle A 2nd Floor, Verticle B 1st Floor, Verticle C
- 24.
- 25.
- 26.
- 27.
- 28. CPU Multi-Reactor 31 Core CoreCore Core Eventbus Other Vert.x Instance Browser Verticle Verticle
- 29. Event & WorkerVerticles Event Driven Verticles Worker Verticles 32 Verticle Verticle Verticle Thread Pool Thread Pool Verticle Verticle Verticle Verticle Verticle
- 30. Implementing the datasource Http Verticle Routing requests & sending responses Verticles querying the DB Searching timeseries labels Annotation Timeseries data points Optional Verticles for Post Processing 33
- 31. What is thechallenge? Optimization Queries can be optimized Large datasets have to be searched, read and transported Source data can not be modified VS data redundancy Sizing How to size the analysis system without knowing the query-times? How to size thread pools or database pools if most of the queries will take 100ms – 30s ? 34
- 32.
- 33. Step 1 –The naive approach Find all datapoints within range 36
- 34.
- 35. Step 2 –Split Request Split request into chunks (#chunks = #cores) Use multiple Verticle Instance in parallel (#instances = #cores) ? 38 CPU
- 36. CPU Datasource Architecture 39 HTTP Service Split/Merge Request Eventbus Query Database Query Database Query Database Query Database HTTP Request HTTP Response DB
- 37. Step 3 –Aggregate Datapoints Use Mongo Aggregation Pipeline Reduce Datapoints returned to service 40
- 38. CPU Datasource Architecture 41 HTTP Service Split/Merge Request Eventbus Query Database Query Database Query Database Query Database HTTP Request HTTP Response DB
- 39. Step 4 –Percentiles (CPU) Fetch all data Calculate percentiles in service 42 CPU
- 40. Step 4 –Percentiles (DB) Build aggregation pipeline to calculate percentiles Algorithm, see http://www.dummies.com/education/math/statistics/how-to- calculate-percentiles-in-statistics/ 43 DB
- 41. CPU Datasource Architecture 44 HTTP Service Split/Merge Request Eventbus Query Database Query Database Query Database Query Database HTTP Request HTTP Response DB
- 42. Step 5 -Postprocessing Apply additional computation on the result from the database 45
- 43. CPUCPU Datasource Architecture (final)46 HTTP Service Split Request Eventbus Query Database Query Database Query Database Query Database Merge Result HTTP Request HTTP Response DB Post Process Post Process Post Process Post Process Eventbus
- 44. Anyone recognize this chip? CPU of the PS3 (Cell BE) Central Bus (EIB) 1 General Purpose CPU Runs the Game (Event) Loop 8 SPUs Special Processing Sound Physics AI ... 230 GFLOPS (vs 103 GFLOPS PS4) 47
- 45. Adding more stats& calculation Push Calculation to DB if possible Add more workers / node for complex (post-) processing Aggregate results before post-processing DB performance is king 48
- 46. Custom vs. timeseriesDB Custom: No migration of existing data No redundant data storage More flexibility Better extensibility Custom views Custom aggregation More options scalability retention Timeseries DB: Better out-of-the-box experience / performance integration functionality Built-in retention policies Built for scalability 49
- 47. Takeaways Grafana ispowerful tool for visualizing data Information becomes consumable through visualization Information is essential for decision making Vert.x is Reactive, Non-Blocking, Asynchronous, Scalable Running on JVM Polyglott Fun 52 Source code on: https://github.com/gmuecke/grafana-vertx-datasource
- 48. Still hungry? FOR GEEKSTUFF 53
- 49. Let’s read alarge data file Datafile is large (> 1GB) Every line of the file is a datapoint The first 10 characters are a timestamp The dataset is sorted The datapoints are not equally distributed Grafana requires reads ~1900 datapoints per chart request 54
- 50. The Challenges (pickone) How to randomly access 1900 datapoints without reading the entire file into memory? How to read a huge file efficiently into memory? 55 Index + Lazy refinement Index + Lazy load
- 51. Let’s build anindex Indexes can be build using a tree- datastructure Node: Timestamp Leaf: offset position in file or the datapoint Red-Black Trees provide fast access Read/Insert O(log n) Space n 56 © Cburnett, Wikipedia
- 52.
- 53. java.util.TreeMap isa red-black tree based implementation* TreeMap<Long,Long> index = new TreeMap<>(); *actually all HashMaps are 58
- 54. How to buildan index (fast)? Read datapoint from offset positions Build a partial index 59 Dataset
- 55. On next query Locate Block Refine Block Update Index 60
- 56. CPUCPU Datasource Architecture (again)61 HTTP Service Split Request Eventbus Read File Read File Read File Read File Merge Result HTTP Request HTTP Response Dataset Post Process Post Process Post Process Post Process Eventbus
- 57.
- 58.