Power of SPL Workshop
0 likes512 views
The document is a presentation about the Power of Splunk Search Processing Language (SPL). It provides an overview of SPL, including that it has over 140 search commands and was originally based on Unix pipelines and SQL. It then discusses examples of using SPL for tasks like finding specific events, charting statistics, enriching data sources, mapping geographic data, identifying anomalies, and data exploration. The presentation also covers creating custom visualizations in Splunk and using the Machine Learning Toolkit with SPL.
Power of SPL Workshop
- 1. © 2017 SPLUNK INC.© 2017 SPLUNK INC. Power of Splunk Search Processing Language (SPL™) May 23rd, 2017 – Bellevue, WA Jeff Spencer | Sr. Sales Engineer
- 2. © 2017 SPLUNK INC. During the course of this presentation, we may make forward-looking statements regarding future events or the expected performance of the company. We caution you that such statements reflect our current expectations and estimates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-looking statements, please review our filings with the SEC. The forward-looking statements made in this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or accurate information. We do not assume any obligation to update any forward-looking statements we may make. In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only and shall not be incorporated into any contract or other commitment. Splunk undertakes no obligation either to develop the features or functionality described or to include any such feature or functionality in a future release. Splunk, Splunk>, Listen to Your Data, The Engine for Machine Data, Splunk Cloud, Splunk Light and SPL are trademarks and registered trademarks of Splunk Inc. in the United States and other countries. All other brand names, product names, or trademarks belong to their respective owners. © 2017 Splunk Inc. All rights reserved. Forward-Looking Statements
- 3. © 2017 SPLUNK INC. Set Up Before You Can Play Download the following at splunk.com and splunkbase.com ▶ Splunk Enterprise: • https://www.splunk.com/download ▶ Power of SPL App: • https://splunkbase.splunk.com/app/3353/
- 4. © 2017 SPLUNK INC. Set Up Before You Can Play ▶ Install Power of SPL App ▶ Now we’re ready to go!
- 5. © 2017 SPLUNK INC. ▶ License expired (already had older version installed) • Close browser, empty cache, open browser. If that doesn’t work: • Stop Splunk. • Uninstall all Splunk versions • Windows Control Panel->Uninstall programs->Splunk • OS X. Finder->Applications->Right click Splunk, Move to trash • Reinstall • Start Splunk ▶ Can’t start Splunk • Windows, Search Control panel ->Services->Splunk start • Linux; cd <SPLUNK dir>/splunk/bin;./splunk start Common problems at this point
- 6. © 2017 SPLUNK INC. 1. Installation and Setup (~15min) 2. Power of SPL Walkthrough (~1h 30min) • Overview & Anatomy of a Search • SPL Commands & Examples for Searching, charting, converging, mapping, transactions, anomalies, exploring data, custom 3. Custom Visualizations (~30min) 4. SPL and the Machine Learning Toolkit (~45min) Agenda
- 7. © 2017 SPLUNK INC. SPL Overview
- 8. © 2017 SPLUNK INC. ▶ Over 140 search commands ▶ Syntax was originally based upon the Unix pipeline and SQL and is optimized for time-series data ▶ The scope of SPL includes data searching, filtering, modification, manipulation, enrichment, insertion and deletion ▶ Includes machine learning such as anomaly detection SPL Overview Disk Intermediate results table Intermediate results table Final results table
- 9. © 2017 SPLUNK INC. ▶ Flexibility and effectiveness on small and big data ▶ Late-binding schema ▶ More/better methods of correlation ▶ Not just analyze, but visualize Why Create a New Query Language? Data BIG Data
- 10. © 2017 SPLUNK INC. search and filter | munge | report | cleanup | rename sum(KB) AS "Total KB" dc(clientip) AS "Unique Customers" | eval KB=bytes/1024 sourcetype=access* | stats sum(KB) dc(clientip) SPL Basic Structure
- 11. © 2017 SPLUNK INC. SPL Examples
- 12. © 2017 SPLUNK INC. ▶ Find the needle in the haystack ▶ Charting statistics and predicting values ▶ Enriching and converging data sources ▶ Map geographic data in real time (We’ll add some custom viz here!) ▶ Identifying anomalies ▶ Transactions ▶ Data exploration & finding relationships between fields ▶ Custom commands SPL Examples and Recipes
- 13. © 2017 SPLUNK INC. ▶ Find the needle in the haystack ▶ Charting statistics and predicting values ▶ Enriching and converging data sources ▶ Map geographic data in real time ▶ Identifying anomalies ▶ Transactions ▶ Data exploration & finding relationships between fields ▶ Custom commands SPL Examples and Recipes
- 14. © 2017 SPLUNK INC. Eval – Just Getting Started! Splunk Search Quick Reference Guide
- 15. © 2017 SPLUNK INC. ▶ Find the needle in the haystack ▶ Charting statistics and predicting values ▶ Enriching and converging data sources ▶ Map geographic data in real time ▶ Identifying anomalies ▶ Transactions ▶ Data exploration & finding relationships between fields ▶ Custom commands SPL Examples and Recipes
- 16. © 2017 SPLUNK INC. Stats/Timechart – But Wait, There’s More! Splunk Search Quick Reference Guide
- 17. © 2017 SPLUNK INC. ▶ Find the needle in the haystack ▶ Charting statistics and predicting values ▶ Enriching and converging data sources ▶ Map geographic data in real time ▶ Identifying anomalies ▶ Transactions ▶ Data exploration & finding relationships between fields ▶ Custom commands SPL Examples and Recipes
- 18. © 2017 SPLUNK INC. Converging Data Sources Index Untapped Data: Any Source, Type, Volume Ask Any Question Application Delivery Security, Compliance and Fraud IT Operations Business Analytics Industrial Data and the Internet of Things On-Premises Private Cloud Public Cloud Storage Online Shopping Cart Telecoms Desktops Security Web Services Networks Containers Web Clickstreams RFID Smartphones and Devices Servers Messaging GPS Location Packaged Applications Custom Applications Online Services DatabasesCall Detail Records Energy Meters Firewall Intrusion Prevention
- 19. © 2017 SPLUNK INC. ▶ Find the needle in the haystack ▶ Charting statistics and predicting values ▶ Enriching and converging data sources ▶ Map geographic data in real time (Let’s add some viz!) ▶ Identifying anomalies ▶ Transactions ▶ Data exploration & finding relationships between fields ▶ Custom commands SPL Examples and Recipes
- 20. © 2017 SPLUNK INC. Custom Visualizations
- 21. © 2017 SPLUNK INC. ▶ Native charts and maps • Bar / Line / Area charts • Bubble / Scatter plots • Gauges • Maps • Single Value Displays • Tables ▶ Generalized to fit use cases across many different areas ▶ Can be customized to some extent to cover specific use cases Native Visualizations In Splunk 21
- 22. © 2017 SPLUNK INC. ▶ Many use cases require a more specific visualization ▶ Specific custom appearance ▶ Represent data where native visualizations are not suitable • You can Splunk everything! • We won’t be able to predict every possible use case • Still uses SPL to drive visualizations Custom Visualizations FTW!
- 23. © 2017 SPLUNK INC. ▶ Platform extensibility framework and API ▶ Targeted at internal and external developers with web development / JS skills and basic knowledge of the Splunk platform ▶ Developers can make use of any third party libraries (d3.js, three.js, highcharts.js, etc…) that run in the browser* * with minor adjustments, and if third party license permits such use Custom Visualizations
- 24. © 2017 SPLUNK INC. Custom Visualizations For Admins In-product • Packaged as an app! • Installed like any other app • Users can search for visualizations on Splunkbase and directly in the product Installation
- 25. © 2017 SPLUNK INC. ▶ Choose from potentially dozens of installed visualizations! ▶ Appears as a first-class citizen alongside native visualizations • Looks and works just like packaged native visualizations ▶ Customize functionality and appearance of the visualization without touching any code, straight from the UI ▶ SPL Example provided as you hover over each visualization option. Custom Visualizations How-to
- 26. © 2017 SPLUNK INC. New Splunk Visualizations Treemap Sankey Diagram Punchcard Calendar Heat Map Parallel Coordinates Bullet GraphLocation Tracker Horseshoe Meter Machine Learning Charts Timeline Horizon Chart Multiple use cases across IT, security, IoT, and business analytics
- 27. © 2017 SPLUNK INC. Box Plot 3D scatter plot New Partner/Community Visualizations Wordcloud Donut Chart Heat Map
- 28. © 2017 SPLUNK INC. New Partner/Community Visualizations Geo Heatmap Custom Cluster Map Clustered Single Value Map Missile Map
- 29. © 2017 SPLUNK INC. Custom Visualizations – Demo!
- 30. © 2017 SPLUNK INC. ▶ Find the needle in the haystack ▶ Charting statistics and predicting values ▶ Enriching and converging data sources ▶ Map geographic data in real time ▶ Identifying anomalies ▶ Transactions ▶ Data exploration & finding relationships between fields ▶ Custom commands SPL Examples and Recipes
- 31. © 2017 SPLUNK INC. ▶ Find the needle in the haystack ▶ Charting statistics and predicting values ▶ Enriching and converging data sources ▶ Map geographic data in real time ▶ Identifying anomalies ▶ Transactions ▶ Data exploration & finding relationships between fields ▶ Custom commands SPL Examples and Recipes
- 32. © 2017 SPLUNK INC. ▶ Find the needle in the haystack ▶ Charting statistics and predicting values ▶ Enriching and converging data sources ▶ Map geographic data in real time ▶ Identifying anomalies ▶ Transactions ▶ Data exploration & finding relationships between fields ▶ Custom commands SPL Examples and Recipes
- 33. © 2017 SPLUNK INC. Data Exploration | analyzefields | anomalies | arules | associate | cluster | contingency | correlate | fieldsummary
- 34. © 2017 SPLUNK INC. ▶ Predict Numeric Fields ▶ Predict Categorical Fields ▶ Detect Numerical Outliers ▶ Detect Categorical Outliers ▶ Forecast Time Series ▶ Cluster Events Machine Learning Toolkit and Showcase Examples
- 35. © 2017 SPLUNK INC. ▶ Find the needle in the haystack ▶ Charting statistics and predicting values ▶ Enriching and converging data sources ▶ Map geographic data in real time ▶ Identifying anomalies ▶ Transactions ▶ Data exploration & finding relationships between fields ▶ Custom commands SPL Examples and Recipes
- 36. © 2017 SPLUNK INC. ▶ What is a Custom Command? • “| haversine origin="47.62,-122.34" outputField=dist lat lon” ▶ Why do we use Custom Commands? • Run other/external algorithms on your Splunk data • Save time munging data (see Timewrap!) • Because you can! ▶ Create your own or download as Apps • Haversine (Distance between two GPS coords) • Timewrap (Enhanced Time overlay) • Levenshtein (Fuzzy string compare) • Base64 (Encode/Decode) Custom Commands
- 37. © 2017 SPLUNK INC. SPL & The Machine Learning Toolkit
- 38. © 2017 SPLUNK INC. ▶ Predict Numeric Fields ▶ Predict Categorical Fields ▶ Detect Numerical Outliers ▶ Detect Categorical Outliers ▶ Forecast Time Series ▶ Cluster Events Machine Learning Toolkit and Showcase Examples
- 39. © 2017 SPLUNK INC. Machine Learning with the Splunk Platform Visualize Share Clean Transform Operationalize Monitor Alert Build Model Search Explore Collect Data Test, Improve Models Ecosystem MLTK Choose Algorithm Ecosystem Splunk Splunk Splunk Splunk MLTK Splunk MLTK Splunk MLTK Splunk Ecosystem Splunk Real-time Data Science Pipeline Ecosystem MLTK Splunk Splunk’s App Ecosystem contains 1000’s of free add-ons for getting data in, applying structure and visualizing your data giving you faster time to value. The Machine Learning Toolkit delivers new SPL commands, custom visualizations, assistants, and examples to explore a variety of ml concepts. Splunk Enterprise is the mission-critical platform for indexing, searching, analyzing, alerting and visualizing machine data. Packaged: UBA, ITSI
- 40. © 2017 SPLUNK INC. ML SPL Visualize Share Correlate Clean Munge Operationalize Monitor Alert Build Model Search Explore Universal Indexing Test, Improve Models Ecosystem MLTK Choose Algorithm Ecosystem Splunk Splunk Splunk Splunk MLTK Splunk MLTK Splunk MLTK Splunk Ecosystem Splunk fit sample apply listmodels deletemodel summary eval rex stats eventstats streamstats table … timechart chart stats geostats geom sendalert sendemail table … MLTK Library predict (cmd) anomalydetection (cmd) analyzefields anomalies arules associate cluster contingency correlate fieldsummary …
- 41. © 2017 SPLUNK INC. MLTK Commands The Machine Learning Toolkit contains several custom search commands that implement classic machine learning and statistical learning tasks: • fit: Fit and apply a machine learning model to search results. • apply: Apply a machine learning model that was learned using the fit command. • summary: Return a summary of a machine learning model that was learned using the fit command. • listmodels: Return a list of machine learning models that were learned using the fit command. • deletemodel: Delete a machine learning model that was learned using the fit command. • sample: Randomly sample or partition events.
- 42. © 2017 SPLUNK INC. ML-SPL Demo
- 43. © 2017 SPLUNK INC. Set Up Before You Can Play Download the following at splunkbase.com ▶ Machine Learning Toolkit: • https://splunkbase.splunk.com/app/2890/ ▶ Python for Scientific Computing: • https://splunkbase.splunk.com/app/2881/ *Note – For the Python for Scientific Computing App you need to download the platform specific version – Mac, Linux, Windows
- 44. © 2017 SPLUNK INC. ▶Additional information can be found in: • Power of SPL App! • Docs - Search Manual • Docs - MLTK Search Commands • MLTK Quick Reference Guide • Blogs • Answers • Exploring Splunk For More Information
- 45. © 2017 SPLUNK INC. • SPL Examples App Other Useful Apps to download! • Splunk 6.x Dashboard Examples • Splunk 6.x Overview App
- 46. © 2017 SPLUNK INC. • 6,000+ IT and Business Professionals • 175+ Sessions • 80+ Customer Speakers PLUS Splunk University • Three days: Sept 23-25, 2017 • Get Splunk Certified for FREE! • Get CPE credits for CISSP, CAP, SSCP SEPT 25-28, 2017 Walter E. Washington Convention Center Washington, D.C. CONF.SPLUNK.COM .conf2017: The 8th Annual Splunk Conference
- 47. © 2017 SPLUNK INC. Q & A
- 48. © 2017 SPLUNK INC.© 2017 SPLUNK INC. Thank You