SplunkLive! Seattle - Splunk for Developers
Download as PPTX, PDF•3 likes•1,046 views
This document discusses Splunk's developer platform and resources for application development. It provides an overview of empowering developers to gain application intelligence, build Splunk apps, and integrate and extend Splunk. The document discusses building Splunk apps and provides resources for developers including tutorials, code samples, downloads, developer guidance, Splunk Base, GitHub, Twitter, and blogs. It also promotes Splunk's developer license and platform approach with search, analytics, and an open ecosystem to build solutions.
SplunkLive! Seattle - Splunk for Developers
- 1. Grigori Melnik, Principal Product Manager – Splunk Developer Platform Copyright © 2015 Splunk Inc. Splunk for Developers Grigori Melnik Principal Product Manager Developer Platform @gmelnik Seattle
- 2. Grigori Melnik, Principal Product Manager – Splunk Developer Platform2 EMPOWERING DEVELOPERS Gain Application Intelligence Build Splunk Apps Integrate & Extend Splunk
- 3. Grigori Melnik, Principal Product Manager – Splunk Developer Platform3 3 www.splunk.com/apptitude
- 4. Grigori Melnik, Principal Product Manager – Splunk Developer Platform Copyright © 2015 Splunk Inc. Splunk for Application Development
- 5. Grigori Melnik, Principal Product Manager – Splunk Developer Platform Build Unit Testing Code Check-in Integration Testing Deploy Staging Application Development Challenges 5
- 6. Grigori Melnik, Principal Product Manager – Splunk Developer Platform Build Unit Testing Code Check-in Integration Testing Deploy Staging Lack of visibility across the product development lifecycle Pressure to increase velocity and agility with DevOps Limited insights into behavior and performance from application logs Application Development Challenges 6
- 7. Grigori Melnik, Principal Product Manager – Splunk Developer Platform Quickly trace and identify errors anywhere in the codebase with real-time search and monitoring Instrument your app logs to gain application intelligence Break down dev tool silos with real-time insights from machine data GAIN END-TO-END VISIBILITY ACROSS THE DEV TOOL CHAIN FIND AND FIX ISSUES FASTER PUSH BETTER CODE USING ANALYTICS Splunk for Application Lifecycle Intelligence 7
- 8. Grigori Melnik, Principal Product Manager – Splunk Developer Platform Real-time dashboards show error rate in production and impact of pushing new builds Developers can search and visualize web logs, Java logs, eventlogs etc; trace tx without complex instrumentation Alerts notify developers as soon as a problem arises Find and Fix Issues Faster 8
- 9. Grigori Melnik, Principal Product Manager – Splunk Developer Platform Gain end-to-end visibility to make informed decisions Analytics insights without the need for additional analytics tools Ask questions while exploring and collecting data Push Better Code Using Analytics 9
- 10. Grigori Melnik, Principal Product Manager – Splunk Developer Platform10 1 CI / Build Servers Project and Issue Tracking Code Repository QA / Testing Tools End-To-End Visibility Across The Dev Tool Chain Deployment Servers / Automation
- 11. Grigori Melnik, Principal Product Manager – Splunk Developer Platform11
- 12. Grigori Melnik, Principal Product Manager – Splunk Developer Platform12 CI / Build Server Code Review Task Tracking What Data Can You Splunk? Logs – Which code has already been reviewed for this release/sprint? Who has completed the most code reviews? What code has NOT been reviewed? Logs/API – Who is changing files? What kinds of files are being changed? What branches are most active? What types of activities are occurring for a branch? Version Control Logs/API – How many builds completed today/this week/this month? Which check-in kicked off this build? Which tests ran against this failed build? Logs – Which tasks are assigned to which developers? What progress is being made to complete assigned tasks? What tasks remain for this release/sprint? 1
- 13. Grigori Melnik, Principal Product Manager – Splunk Developer Platform13 Key Benefits of Application Lifecycle Intelligence Reduced Time to Market Shrink the time it takes to get code through dev/test to market through faster issue identification and resolution Increased Agility With real-time visibility into processes like code check-ins, builds and tests to support DevOps practices like continuous integration “Our devs are now able to find and fix issues five to ten times faster.” “We can monitor all the automation and handoffs it takes to deploy 5-10 times a day” Application Insights Instrument customer application logs to capture critical business events and user behavior “My code isn’t ready until it’s Splunk-ready” 1
- 14. Grigori Melnik, Principal Product Manager – Splunk Developer Platform Copyright © 2015 Splunk Inc. Demo: ADLC
- 15. Grigori Melnik, Principal Product Manager – Splunk Developer Platform Copyright © 2015 Splunk Inc. Touring the Splunk Development Platform
- 16. Grigori Melnik, Principal Product Manager – Splunk Developer Platform16 Evolving the Splunk Platform Collection Indexing Search Processing Language Core Functions Inputs, Apps, Other Content SDKs & plug-ins Operational Intelligence Platform Content Core Engine User and Developer Interfaces Web Framework REST API
- 17. Grigori Melnik, Principal Product Manager – Splunk Developer Platform17 Powerful Platform for Enterprise Developers 1 REST API Build Splunk Apps Extend and Integrate Splunk Simple XML JavaScript/CSS Extensions C# JavaScript Python Ruby Java PHP Data Models Search Extensibility Modular Inputs SDKs KV Store
- 18. Grigori Melnik, Principal Product Manager – Splunk Developer Platform Log directly to Splunk via TCP, UDP, HTTP Integrate search results with other applications using custom visualizations Create and run searches from other applications The REST API and SDKs 18 VisualizeSearch Manage Add/Delete Users Manage Inputs Index
- 19. Grigori Melnik, Principal Product Manager – Splunk Developer Platform19 The Splunk REST API Exposes an API method for every feature in the product – Whatever you can do in the UI – you can do through the API – Index, Search, Visualize, Manage API is RESTful – Endpoints are served by splunkd – Requests are GET, POST, and DELETE HTTP methods – Responses are Atom XML & JSON – Versioning as of Splunk 5.0 – Search results can be output in CSV/JSON/XML 1
- 20. Grigori Melnik, Principal Product Manager – Splunk Developer Platform20 SDKs Overview 20 • Stay true to the semantics of the particular language • E.g. Keep Python “pythonic” • E.g. C#: Fully async , PCL, support for Rx • Provide implementation that feels natural to the developer • E.g. Project, build, IDE (where applicable) support • Cover REST API endpoints based on use cases of language • Namespaces • owner: splunk username (defaults to current user) • app: app context (defaults to default app) • sharing: user | app | global | system
- 21. Grigori Melnik, Principal Product Manager – Splunk Developer Platform A Developer’s Smörgåsbord Data ingestion – Input Scripted inputs Modular inputs Custom (trained) source types Custom sources – Data ingestion pipeline Field extractions Field transformations – Indexing Custom indexes Searching – Search authoring Custom search commands Macros (basic, parametrized) Saved searches – Data classification Event types Transactions – Data enrichment Lookups KV store collections Workflow actions – Data normalization Tags Aliases – Data mining cluster & dedup anomalousvalue kmeans predict commands … Processing & reporting – Search-time mapping Data models – CIM extensions – Custom UI/visualizations Pages, views & dashboards JS Extensions CSS Extensions Custom setup screens – Scheduled processing Scheduled reports – Alerting Scripted alerts – Branding & navigation Custom app navigation & branding – Manageability Custom splunkweb controllers Custom splunkd endpoints
- 22. Grigori Melnik, Principal Product Manager – Splunk Developer Platform Copyright © 2015 Splunk Inc. Building Splunk Apps
- 23. Splunk Developer Guidance Splunk Reference Apps Complete, working real-world Splunk solutions built together with partners (Conducive; Auth0) ̶ 2 (pseudo-) production releases ̶ entire code & test repos on GitHub ̶ under Apache 2.0 Associated Guidance I. Start-to-Finish Journey Documentary II. Essentials dev.splunk.com/goto/devguide
- 24. 1. Started with a Questions BacklogArchitecture – What does a typical Splunk application reference architecture look like? – What common paradigms are applicable to Splunk app development? – What are the typical deployment topologies? Why should I choose a specific one? What are the confounding factors on the choice of my topology? – How do I partition my Splunk solutions? – What are the tradeoffs of various types of inputs? – How do I architect my Splunk solution and deployment for a very large scale? – How do I architect my Splunk solution for the cloud? What are specific considerations for deploying to AWS or Azure? – What’s the landscape of Splunk extension points? – How do I integrate data from Splunk into existing applications and systems? – How do I plan and design a robust alerting and monitoring subsystem on top of Splunk? – What should I consider for my sizing requirements? – What are recommended configurations of Splunk deployment to meet my sizing requirements? – Should I architect my solution to index my data in local data center (zone) or centrally? – What are things we can automatically degrade so we can make sure our core experience is working? – When something happens, how do I effectively propagate the info and react to it? – How are other solutions on Splunk built? What were the challenges? How have they been addressed? Packaging and Deployment – How do I piece together various parts of a Splunk app (custom search commands, mod inputs etc.)? – How do I package a Splunk solution with a single install that automatically rolls out all the necessary dependencies? – How do I manage my Splunk solution versioning, backward and future compat? – What's the best way to split up custom apps for deployment? Development – How should I set up my development environment to be productive with Splunk? – What are different ways of how I develop my Splunk app ? Pros and cons of using specific SDK vs REST APIs? Pros and cons of using SimpleXML vs Advanced XML vs Web Framework … – How do I analyze a data source for a TA? – What are the different ways of enriching the data in Splunk? What are their tradeoffs? – When should I use event types and transactions for data classification? – How do I extend Splunk to define a custom input capability? – When should I use modular inputs vs scripted inputs vs..? – What are streaming vs non-streaming outputs considerations? – How do I deal with long-running scripts? Handling shutdown/restart of Splunk? Concurrency? State persistence etc. – Why should I not use transactions? – When should I use pivot vs tstats? – Why should I use data models? – When my data source touches on many data models, should I assume complete separation or heavy inheritance? – How do I extend an existing data model? – What does CIM offer and why should I build CIM-compliant apps? – In the context of CIM, what are the tradeoffs of using my props.conf and transforms.conf and rewriting them on indexing, completely discarding the vendor supplied field names? How do I reconcile the advantages of a clean interface & normalisation, but at the cost of losing alignment with published vendor documentation, and a learning curve for existing users? – How do I manage my solution declarative configuration? How do I detect/troubleshoot bad config? – How do I log and analyze data that is not event driven (certain web feeds, html parsing, image meta data)? – Compare and contrast ad-hoc searching vs background searching – How do I handle transient faults? – How do I effectively manage credentials? – What’s the effect of search head location on my app and the overall user experience? – How do I develop an integrated mechanism to let me connect Splunk to my MOM (messaging middleware) and index my messages? – How do I handle the requirement that app configs must be different across different server types in a distributed environment (e.g. apps on search heads shouldn't have inputs enabled)? Quality/Compliance – What quality gates should I consider? What kind of para-functional characteristics are important to consider? – What heuristics do I use to bless/block a release? – How do I test a data model? – How do I prepare event generation when building/testing an app? – What kind of perf testing should I do and how? – How do I test UI? – How do I security certify my solution? – How do I design to satisfy my retention and compliance policies? – How do I architect to design my availability requirements? – How do I handle geographic disaster recovery / fault tolerance? – How do I properly instrument my solution so that I know what’s happening? Sustained Engineering – How do I maintain/service/support Splunk apps? – How do my customers handle updating their customized configs once new versions of my app come out? Business – Why should I build on Splunk? – What kind of skill do I need my devs to have to build a Splunk solution? – What is the community building? How are current devs creating unique experiences using Splunk – I typically want to see some marketplace success – Cost and pricing are very important to me as a entrepreneur developer. If I am coming in to build a tool that will be commercialized I need to know that the cost structure of Splunk won’t cause my service to be economically unprofitable. What does a typical Splunk application architecture look like? How should I set up my dev environment to be productive with Splunk? How do I integrate Splunk into existing systems? How do I prepare my event generation when developing & testing an app? How do I package an app? deal with app versioning and updates?
- 25. 2. Mined business requirements with partner 3. Formulated learning objectives 4. Reconciled 2 & 3 with our designs …
- 26. Data Search language Aggregating siloed metrics into meaningful KPIs Data manipulation Data normalization Sub-searches Config-driven Persistence with KV store Macros Viz: Dynamic scaling Customizing in-the box viz controls General search patterns Search optimizations Ux Prototyping Adapting 3rd party viz library Composite charts with interactions Dealing with high-volume data sets Troubleshooting perf issues Post-process or not-post-process – deployment implications Automated UI testing (w.Selenium) Setting the stage Overall Splunk app structure UI technology selection: Simple XML vs SplunkJS Modularity Dev & test env Dev workflow Modularity Data onboarding CIM compliance Tools Post-processing Integrating with 3rd party component Unit testing (w.Mocha) Persisting state (per user) Data modeling Using lookups Building a baseline lookup table Windows of time/Custom time ranges Overlaying time data Using sub-searches to correlate data Troubleshooting searches Custom nav Ux activities permeating all dev Data mining: Exploration Preparation: filtering/deduping/ bucketing Using advanced statistics functions Threshold-based anomaly detection Evaluating goodness /accuracy Plus non-functional topics: App versioning Packaging Installation Security review Deployment Publishing to splunkbase App certification
- 27. Grigori Melnik, Principal Product Manager – Splunk Developer Platform Copyright © 2015 Splunk Inc. Demo: Building solutions with Splunk Reference App
- 28. Copyright © 2015 Splunk Inc.28 Splunk Reference App comes preinstalled in the Cloud Sandbox - www.splunk.com/goto/cloud
- 29. Grigori Melnik, Principal Product Manager – Splunk Developer Platform Copyright © 2015 Splunk Inc. Resources
- 30. Grigori Melnik, Principal Product Manager – Splunk Developer Platform30 Splunk Developer License 3
- 31. Grigori Melnik, Principal Product Manager – Splunk Developer Platform31 Where to go for more Info • Tutorials, Code Samples, Getting Started, Downloads – http://dev.splunk.com • Splunk Developer Guidance – http://dev.splunk.com/goto/devguide • Splunk Base (Apps) – https://splunkbase.splunk.com • GitHub – https://github.com/splunk • Twitter – https://twitter.com/splunkdev • Blogs – http://blogs.splunk.com/dev 31
- 32. Copyright © 2015 Splunk Inc.32 Takeaways Application development intelligence Platform, not just an engine Open & extensible On-prem and cloud Developer Guidance : learn and reuse for the win! Reach out to my team ([email protected]) and tell us about your experience @gmelnik / [email protected]
- 33. 33 The 6th Annual Splunk Worldwide Users’ Conference • September 21-24, 2015 • The MGM Grand Hotel, Las Vegas • 4000 IT & Business Professionals • 2 Keynote Sessions • 3 days of technical content – 165+ sessions • 3 days of Splunk University – Sept 19-21, 2015 – Get Splunk Certified for FREE! – Get CPE credits for CISSP, CAP, SSCP, etc. – Save thousands on Splunk education! • 80 Customer Speakers • 80 Splunk Speakers • 35+ Apps in Splunk Apps Showcase • 65 Technology Partners • Ask The Experts and Security Experts, Birds of a Feather, Chalk Talks and a new & improved Partner Pavilion! • Register at conf.splunk.com
- 34. 34 We Want to Hear your Feedback! After the Breakout Sessions conclude Text Splunk to 878787 And be entered for a chance to win a $100 AMEX gift card!
Editor's Notes
- #12: Roll Ubisoft Video