Splunk 6.4 Administration.pdf

Listen to your data.
Splunk 6.4 Administration
Listen to your data. 1 Copyright © 2016 Splunk, Inc. All rights reserved | 7 July 2016
Generated for chandrika seela (chandrika.seela@accenture.com) (C) Splunk Inc, not for distribution

Document Usage Guidelines
Y Should be used only for enrolled students
Y Not meant to be a self-paced document, an instructor is needed
Y Do not distribute
7 July 2016

Course Prerequisites
Y Required:
– Using Splunk
– SplunkArchitecture Overview (eLearning)
Y Strongly Recommended:
– Searching and Reporting with Splunk
– Creating Splunk Knowledge Objects
Note
In order to receive credit for this
course, you must complete all lab
exercises.

Course Goals
Y Build and manage a production Splunk environment
Y Create and maintain Splunk indexes
Y Manage users, roles, and authentication options
Y Deploy forwarders with Forwarder Management
Y Configure common Splunk data inputs
Y Customize the input parsing process
Y Configure a distributed search environment
Y Monitor a Splunk instance with Management Console
Y Configure system alerts

Course Outline
SettingupaSplunk Enterprise Environment
–  Mod1:SettingupSplunk
–  Mod2:LicenseManagement
–  Mod3:SplunkApps
–  Mod4:SplunkConfigurationFiles
–  Mod5:SplunkIndexManagement
–  Mod6:Users,Roles,andAuthentication
BuildingaBasicProduction Environment
–  Mod7:UniversalForwarders
–  Mod8:ForwarderManagement
SplunkInputs
–  Mod9:GettingDataIn
–  Mod10:MonitorInputs
–  Mod11:NetworkandScriptedInputs
–  Mod12:WindowsandAgentlessInputs
–  Mod13:Fine-tuningInputs
ParsingandSearching
–  Mod14:ParsingPhaseandDataPreview
–  Mod15:ManipulatingRawData
–  Mod16:SupportingKnowledgeObjects
–  Mod17:DistributedSearch
SplunkResourceManagement
–  Mod18:BasicPerformanceTuning
–  Mod19:ProblemIsolationOverview
–  Mod20:IntroductiontoLarge-scaleDeployment
–  CourseWrap-up

Module 1:
Setting up Splunk

Module Objectives
Y Describe the Splunk installation options
Y Identify Splunk instance types
Y Identify Splunk hardware requirements
Y List steps to install Splunk
Y Perform post-installation configuration tasks
Y Start, stop, and restart Splunk
Y Enable Distributed Management Console (DMC)

Y Splunk can be deployed in a variety of configurations
Y Scales from a single server to a distributed infrastructure
–  Acceptsanytext data as input
–  Parsestheinputs into events
–  Storesevents in indexes
–  Searchesand reports
–  Authenticates users
Splunk Overview
DB
Servers
Networks Servers Web
Services
Mobile
Devices
Custom
Apps
Security
Any Text Data
Searching
Users
Input
Parsing
Indexing

Splunk Deployment – Standalone
Y Single Server
– All functions in a single instance of
Splunk
– For testing, proof of concept,
personal use, and learning
– This is what you get when you
download Splunk and install with
default settings
Y Recommendation
– Have at least one test/development
setup at your site
Parsing
Indexing
Input
Searching
Note
This is the initial configuration in
class.

Splunk Deployment – Basic
Y Splunk server
– Similar to server in standalone
configuration
– Manage deployment of forwarder
configurations
Y Forwarders
– Forwarders collect data and send it
to Splunk servers
– Install forwarders at data source
(usually production servers)
Parsing
Indexing
Input
Searching
Forwarder
Management
Note
Your lab environment will evolve to
include a separate forwarder.

Splunk Deployment – Distributed
Y Splunk can be distributed and scaled
in a variety of ways
–  Moreindexerstohandlemoreinput
–  MoreindexersANDsearchheadsto
handlemoresearching
Y Manage forwarder configurations
from a dedicated Deployment Server
Parsing
Indexing
Input
Searching
Search Head
Indexers
(Search peers)
Forwarders
Deployment
Server
Forwarder
Management
Note
You will add a single search peer
to your environment in a later lab
exercise.

Y Included in the Splunk Enterprise software package
Y Included in the Universal Forwarder software package
What Software Do You Install?
Indexer
(Search peer)
Search Head Heavy
Forwarder
Universal Forwarder
Deployment
Server
License
Master
Cluster
Master
Search Head
Cluster
Deployment Client
Splunk Enterprise

Reference Servers
Y Hardware requirements and sizing are discussed in detail in
– Architecting and Deploying Splunk class
– docs.splunk.com/Documentation/Splunk/latest/Capacity/Referencehardware
Indexer Search Head
OS Linux or Windows 64-bit distribution
Network 1Gb Ethernet NIC
Optional second NIC for a management network
Memory 12 GB RAM
CPU Intel 64-bit chip architecture
12 CPU cores
Running at 2+ GHz
Intel 64-bit chip architecture
4 CPUs, quad-core per CPU
Running at 2+ GHz
Disk Disk subsystem capable of 800 IOPS 2 x 10K RPM 300GB SAS drives - RAID 1

Further Reading: Hardware and Virtualization
Y System requirements
docs.splunk.com/Documentation/Splunk/latest/Installation/Systemrequirements#Recommended_hardware
Y Hardware capacity planning (Dimensions of a Splunk deployment)
docs.splunk.com/Documentation/Splunk/latest/Capacity/DimensionsofaSplunkEnterprisedeployment
Y Summary of performance recommendations
docs.splunk.com/Documentation/Splunk/latest/Capacity/Summaryofperformancerecommendations
Y Virtualization tech brief
www.splunk.com/web_assets/pdfs/secure/Splunk_and_VMware_VMs_Tech_Brief.pdf

Splunk Installation Overview
Y Pre-installation Checklist
– Start-up account
– Time synchronization
– Splunk ports
– Linux setting recommendations
Y Installation
– Splunk directory structure
Y Post-installation configuration
– Run Splunk at boot
– Configure system settings
– Optionally, enable Distributed Management Console

Start-up Account
Y Best practice: Do not run Splunk as super-user
– For example, root on *NIX, administrator on Windows
Y  Create a user account that is used to run Splunk
– For input, Splunk must be able to access data sources
ê  On *NIX, /var/log is not typically open to non-root accounts
– On *NIX, non-root accounts cannot access ports < 1024
– On Windows
ê  Use a domain account if Splunk has to connect to other servers
ê  Otherwise, use a local machine account
– Make sure the Splunk account can access scripts used for inputs and alerts

Time Synchronization
Y Best practice: Use a time synchronization service such as NTP
Y Splunk searches depend on accurate time
– Correct event timestamping is essential
Y It is imperative that your Splunk indexer and production servers have
standardized time configuration
– Clock skew between hosts can affect search results

Splunk Ports
Usage Splunk Enterprise Universal Forwarder
splunkd 8089 8089
Splunk Web 8000 -
Web app-server proxy 8065 -
KV Store 8191 -
S2S receiving port(s) No default -
Any network/http input(s) No default No default
Index replication port(s) Optional (no default) -
Search replication port(s) Optional (no default) -

Linux Setting Recommendations
Y Increase ulimit settings
– The following OS parameters need to be increased to allow for a large number of
buckets/forwarders/users
docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/ulimitErrors
Y Turn Transparent Huge Pages (THP) off on Splunk Enterprise servers
docs.splunk.com/Documentation/Splunk/latest/ReleaseNotes/SplunkandTHP
ulimit -a
ulimit -c 1073741824 (1 GB) (unlimited)
ulimit -n 48 x default (48 x 1024 = 49,152) (65536)
ulimit -u 12 x default (12 x 1024 = 12,288) (258048)
core file size
open files
max user processes

Installation
Y Download Splunk Enterprise from www.splunk.com/download
Y Installation: (as account running Splunk)
– *NIX – un-compress the .tar.gz file in the path you want Splunk to run from
– Windows – execute the .msi installer and follow the wizard steps
Y Complete installation instructions at:
docs.splunk.com/Documentation/Splunk/latest/Installation/Chooseyourplatform
Y After installation:
– Splunk starts automatically on Windows
– Splunk must be manually started on *NIX until boot-start is enabled

Run Splunk at Boot
Y *NIX
– Splunk on *NIX does not auto-start at boot time (default)
– To enable boot-start, run as root:
– This modifies the *NIX boot-up configuration
ê  Modifies /etc/init.d depending on your *NIX flavor
– Pass the -user parameter to start Splunk as the correct user
Y Windows
ê  Runs as splunkd service and starts child processes
ê  The service starts and stops like any Windows services
# ./splunk enable boot-start -user splunker

Splunk Directory Structure
Note
$SPLUNK_HOME depicted in the
documentation is not an exported
environment variable. It is used as
a placeholder for "the top directory
where Splunk is installed.”
SPLUNK_HOME is used in this
training.
SPLUNK_HOME
bin etc var
system apps users
search launcher <custom app>
lib
splunk
executables licenses, configs
indexes

Splunk Processes
splunkd
Y Runs on port 8089 (default) using SSL
Y Spawns and controls Splunk child
processes (helpers):
– Splunk Web proxy, KV store, and
Introspection services
– Each search, scripted input, or
scripted alert
Y Accesses, processes, and indexes
incoming data
Y Handles all search requests and
returns results
Splunk Web
Y Splunk browser-based user interface
– Provides both a search and
management front end for splunkd
process
Y Runs on port 8000 by default
– http://<server_name>:<port>
Y Initial default login:
– Username: admin
– Password: changeme

Splunk Web - Server Settings
Y Administrators can select Settings > Server settings > General settings
1
2
3
-  Overall server configuration
-  Used to set server options

Describing General Settings
docs.splunk.com/Documentation/Splunk/latest/Security/
AboutsecuringyourSplunkconfigurationwithSSL
Identifies this server to
other Splunk servers
SPLUNK_HOME
Splunkd port
Default host name
assigned to events
from this server
Change if indexes are
on a different volume
Set minimum free space
- Prevents the file system from being filled by Splunk
- Splunk data loss can occur if this value is reached
Changes require a restart of Splunk
6
5
4
3
2
1
7
6
5
4
3
2
1
7
8191

Restarting the Server from Splunk Web
1
2
3
Note
Any changes to General settings
generates a message. Clicking the
indicator opens a message
prompting you to restart.
You can also restart by selecting
Settings > Server controls or
from the CLI. (splunk restart)

The Splunk Command Line Interface (CLI)
Y splunk is the program in the bin directory to run the CLI
– Same syntax on all supported platforms
Command Operation
splunk help Display a usage summary
splunk help <object> Display the details of a specific object
splunk [start | stop | restart] Manages the Splunk processes
splunk start –-accept-license Automatically accept the license without prompt
splunk status Display the Splunk process status
splunk show splunkd-port Show the port that the splunkd listens on
splunk show web-port Show the port that Splunk Web listens on
splunk show servername Show the servername of this instance
splunk show default-hostname Show the default host name used for all data inputs

Distributed Management Console (DMC)
Y Splunk collects a lot of data about itself
Y DMC is a Splunk admin-only app that lets you monitor and investigate
Splunk performance, resource usage, and more
Note
You will use DMC to monitor your
activities as you learn more about
Splunk components.

Enabling DMC in Standalone Mode
Y DMC runs un-configured in standalone mode by default
Y To enable, click Settings > General Setup >Apply Changes
3
2
1
The default server roles

More Resources
Y Splunk Documentation: http://docs.splunk.com/Documentation
Y SplunkApp Repository: https://splunkbase.splunk.com/
Y SplunkAnswers: http://answers.splunk.com/
Y Splunk Blogs: http://blogs.splunk.com/
Y Splunk Wiki: http://wiki.splunk.com/

Lab Exercise 1 – Configure Splunk
Time: 15 - 20 minutes
Tasks:
– Log into Splunk Web
– Change your Splunk server name
– Restart Splunk
– Enable DMC and check the system overview
– Access your environment with SSH (Linux) or Remote Desktop Connection (Windows)
– Use Splunk CLI to confirm the status and changes

Lab Exercise 1 – Configure Splunk (cont.)
ssh {EIP}
RDC {EIP}
Provision:
Go to Splunk Web with your browser
using the external IP address
(xxx.xxx.xxx.xxx:8000)
Verification:
Linux:
1.  Connect to your indexer via SSH
2.  Execute Splunk CLI
Windows:
1.  Connect to your indexer via RDC
2.  Run cmd to execute Splunk CLI
http://{EIP}:8000
OR
Your
Computer
Your
Computer

Module 2:
License Management

Module Objectives
Y Identify license types
Y Describe license violations
Y Add and remove licenses

Managing Licenses
Select Settings > Licensing
Change license group
Add a license
Check license alerts and violations
View stacks
Edit and add pools
Designate the license server type
– Master or slave
6
1
2
3
4
5
6
5
4
3
2
1

Splunk License Types
Y Enterprise trial license
– Downloads with product
– Features same as Enterprise except for 500mb per day limit
– Only valid for 60 days, after which one of the other 3 license types must be
activated
– Sales trial license is a trial Enterprise license of varying size and duration
Y Enterprise license
– Purchased from Splunk
– Full functionality for indexing, search head, deployment server, etc.
– Sets the daily indexing volume

Splunk License Types (cont.)
Y Free license
– Disables alerts, authentication, clustering, distributed search, summarization, and
forwarding to non-Splunk servers
– Allows 500mb/day of indexing and forwarding to other Splunk instances
– After 60 days on enterprise trial license, you'll be automatically prompted to convert to
this license type
ê  Canbeactivatedbefore60daysbychanginglicensetype
Y Forwarder license
– Sets the server up as a heavy forwarder
– Applies to non-indexing forwarders
– Allows authentication, but no indexing
Y Splunk license comparison: www.splunk.com/view/SP-CAAAE8W

Adding a License
Y CanuseCLI,upload,orcopy&paste
–  Licensegroupchangerequiresarestart
Y Licensesarestoredunder
SPLUNK_HOME/etc/licenses
Y Canaddmultiplelicenses(stacked)
splunk add licenses <path_to_file>

License Warnings and Violations
Y If the indexing exceeds the allocated daily quota in a pool, an alert is raised
– The daily license quota resets at midnight and you have until then to fix
Y If you don't correct the situation, the alert becomes a warning
– The warning message persists for 14 days
Y The fifth warning in a rolling 30-day period causes violation
– Search is disabled for all non-internal indexes
– All other features remain functional, such as indexing and forwarding
– Violation remains in effect for 30 days
– Free license gets only three warnings
Y To unlock the license and enable searching, contact Splunk Support or
Sales

What Counts As Daily License Quota?
Y All data that is indexed, regardless of source, sourcetype, or host
– About how much data you index each day
– Not about how much data you store in Splunk
Y What does not count as daily quota?
– Data that is replicated in a cluster
ê  Data is metered once; the copies do not count
– Summary indexes, using a summarization technique
– Splunk internal logs that are stored in _internal, _audit, etc.
– Data that is eliminated during the parsing process
– Metadata fields, search terms, etc.

Viewing Alerts

Change to Slave
Change an instance to slave by
entering the master license server URI
License Master
with a license stack
All instances collectively share the stack entitlement

Y Pools allow licenses to be subdivided and assigned to a group of indexers
– Can be created for a given stack
– Warnings and violations occur per pool
Y Example: Master has a stack for a total of 500GB
License Pooling
Default Pool
500 GB Shared Entitlement
Enterprise Stack with Single Pool
(Most common)
Pool 2
(200GB Entitlement)
Pool 3
(200GB Entitlement)
Default Pool
(100GB Entitlement)
Enterprise Stack with Multiple Pools
(Multi-tenant environment)

Managing Soft License Warnings
Y DO NOT ignore license warnings
Y Proactively monitor the consumption of your Splunk license
– DMC provides a couple of alerts
– If possible, give yourself wiggle room by rearrange license pools

Lab Exercise 2 –Add and Configure Splunk Licenses
Y Time: 15 – 20 minutes
Y Tasks:
– Add a license with Splunk Web UI (optionally with CLI)
– Enable DMCAlert -Total License Usage Near Daily Quota

Module 3:
Splunk Apps

Module Objectives
Y Describe Splunk apps and add-ons
Y Install an app on a Splunk instance
Y Manage app accessibility and permissions

.
.
.
What is an App?
Y An app is an independent collection of:
– Configuration files
ê  Defininginputs,indexes,sourcetypes,field
extractions,transformations
ê  Providingeventtypes,tags,reports,dashboards
andotherknowledgeobjects
– Scripts, web assets, etc.
Y  Most apps are focused on:
–  Aspecific type of data from a vendor, operating
system, or industry
–  Aspecific business need
Y  Apps may be installed on any Splunk instance
Y  Splunk includes a number default apps
app.conf
tags.conf
mylookup.csv
indexes.conf
myviews.xml
default.meta
transforms.conf
props.conf
inputs.conf
App B
App A

View All Installed Apps
Y From the Search app, select Apps > ManageApps
Y Or, from the Home view, click
Y Apps can be Visible or hidden
– Several apps are installed by default that are hidden or disabled
ê  Internal apps used by Splunk should not be modified
ê  Legacy apps
ê  Sample apps
Y Apps are installed under SPLUNK_HOME/etc/apps

Managing Apps
Controls who can
use/modify an app
Enable or disable
an app
Add more apps

Installing an App from splunkbase.splunk.com
Y From the Apps page, click Browse more apps
Y Or, click Apps > Find MoreApps
– Splunk Web will try to access splunkbase.splunk.com
– Search and browse to find the app you want
– Select Install (most apps are free)
ê  You must provide your Splunk.com user ID and password
ê  The app is installed into a sub-directory below SPLUNK_HOME/etc/apps
ê  Some apps may require a restart
Y Or, go directly to splunkbase.splunk.com and download the app as a file
Note
Anyone can sign up for a user
account in Splunk.com.
A support contract is not required.

Installing an App From a File
Y Download the file for the app from splunkbase.splunk.com
– The file may be a .tar.gz, .tgz, .zip, or .spl file
Y Install the app:
– From Splunk Web, click Install app from file
– Using the CLI
splunk install app path-to-appfile
– Or extract the app in the proper location
cd SPLUNK_HOME/etc/apps
tar –xf path-to-appfile
Y Some apps may require a restart
Y Configure the app after the install according to its documentation

Apps on Forwarders?
Y Universal forwarders don't have a web interface, but they can still benefit
from an app
Y An add-on is a subset of an app
– Usually contains data collection but no GUI (reports or dashboards)
Y To install an add-on or app on a forwarder
– Run the CLI command described earlier, or
– Deploy it using a deployment server
ê  deployment server is discussed in a later module

Deleting an App
Y When you delete an app, all of its related configurations and scripts are
removed from a Splunk server
– User’s private app artifacts remain untouched
Y To remove an app:
– splunk remove app <app_folder>
– Or, navigate to SPLUNK_HOME/etc/apps and delete the app's folder and all its
contents
– Restart the Splunk server
Y It can be reinstalled later
Y Alternatively, either disable it or move it to another (backup) location

App Permissions
Y Users with read permission can see the
app and use it
Y Users with write permission can add/
delete/modify knowledge objects used in
the app
– By default, the user role does not have write
permissions within the search app

Lab Exercise 3 -- Install an App
Y Time: 5 - 10 minutes
Y Tasks:
– Download an app
– Install the app
– Change the app permission
– Verify if the app's dashboard displays reports

Module 4:
Splunk Configuration Files

Module Objectives
Y Describe Splunk configuration directory structure
Y Understand configuration layering process
– Index-time process
– Search-time process
Y Use btool to examine configuration settings

Y Each configuration file governs a particular aspect of Splunk functionality
Y All configuration changes are saved in .conf files under SPLUNK_HOME/etc/...
– .conf files are text files using a simple stanza
and name/value (attribute) format
– The syntax is case-sensitive
Y You can change settings using Splunk Web, CLI, SDK,
app install, and/or direct edit
Y All .conf files have documentation and examples:
– SPLUNK_HOME/etc/system/README
ê  *.conf.spec
ê  *.conf.example
ê  Splunkdocumentation: docs.splunk.com
Splunk Configuration Files
[default]
host=www
[monitor:///var/log/httpd]
sourcetype = access_common
ignoreOlderThan = 7d
index = web
Splunk Web
CLI SDK
inputs.conf

Configuration Directories
SPLUNK_HOME
etc
system apps users
joe mary admin
search
unix
local
local
default local unix
default local
search
default local
Out-of-the-box
Custom

Default vs. Local Configuration
Y Splunk ships with default .conf files
– Stored in the default directories
Y Add all configurations and edits to the
local directory
– Most configurations apply to only one app
Y Avoid storing configurations in
SPLUNK_HOME/etc/system
– Use the Searching and Reporting app
as the default location for storing your
configurations that are not app-specific
default local
Shipped with
Splunk or the app
Y Overwritten on
update
Y Do not modify
Your specific
configuration
changes
Y Preserved on
update

Index time vs. Search time
Y The priority of layered configurations are based on the context
– Global context: a network input to collect syslog data
– App/User context: Mary's private report in the Search app
Y For a list of configuration files and their context, go to:
docs.splunk.com/Documentation/Splunk/latest/Admin/Wheretofindtheconfigurationfiles
Index time Global context User-independent and background tasks
such as inputs, parsing, indexing, etc.
Search time App/User context User-related activity, such as searching

default
SPLUNK_HOME
etc
system apps
default local unix
default local
search
local
Index-Time Precedence
1.  etc/system/local
2.  etc/apps/search/local
3.  etc/apps/unix/local
4.  etc/apps/search/default
5.  etc/apps/unix/default
6.  etc/system/default
6 1
5 3 4 2
Server is performing
non-user background
processing
Note
If two or more apps at the same
level of precedence have conflicts
between them, the conflicts are
resolved in ASCII order by app
directory name.

SPLUNK_HOME
etc
system apps users
joe mary admin
search
unix
local
local
default local unix
default local
search
default local
Search Time Precedence Order
7
1
3 2 5 4
Example:
user mary working in
the unix app context
6
Note
Splunk evaluates and if
objects from the app are exported
globally with .meta file setting.
5
4

Runtime Merging of Configurations
Y When Splunk starts, configuration files are merged together into a single
run-time model for each file type
– Regardless of the number of inputs.conf files in various apps or the system
path, only one master inputs configuration model exists in memory at runtime
Y If there are no duplicate stanzas or common settings between the files, the
result is the union of all files
Y If there are conflicts, the setting with the highest precedence is used
– Remember that local always takes precedence over default

Example of Runtime Merging (No Conflict)
[a]
x=1
y=2

[b]
j=1
k=2
[b]
p=1
q=1

[c]
s=1
t=2
[a]
x=1
y=2

[b]
j=1
k=2
p=1
q=1

[c]
s=1
t=2
SPLUNK_HOME/etc/
system/local/
example.conf
SPLUNK_HOME/etc/
apps/search/local/
example.conf
Runtime
example.conf

Example of Runtime Merging (Conflict)
[a]
x=1
y=2

[b]
x=0
z=2
[b]
x=1
y=1

[c]
x=1
y=2
[a]
x=1
y=2

[b]
x=0
y=1
z=2

[c]
x=1
y=2
SPLUNK_HOME/etc/
system/local/
example.conf
SPLUNK_HOME/etc/
apps/search/local/
example.conf
Runtime
example.conf

Configuration Validation Command
Y splunk btool conf-name list [options]
– Shows on-disk configuration for requested file
– Useful for checking the configuration scope and permission rules
ê  Use--debugtodisplaytheexact .conffilelocation
ê  Add--user=<user>--app=<app>toseetheuser/appcontextlayering
Y Examples:
–  splunk help btool
–  splunk btool check
–  splunk btool inputs list
–  splunk btool inputs list monitor:///var/log
–  splunk btool inputs list monitor:///var/log --debug
docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/Usebtooltotroubleshootconfigurations

Scenario: Where are the /var/log/secure.log input configurations specified?
btool Example
> splunk btool inputs list monitor:///var/log/secure.log --debug

etc/apps/search/local/inputs.conf [monitor:///var/log/secure.log]
etc/system/local/inputs.conf host = myIndexer
etc/system/default/inputs.conf index = default
etc/apps/search/local/inputs.conf sourcetype = linux_secure

[monitor:///var/log/secure.log]
host=myIndexer

[monitor:///var/log/secure.log]
sourcetype=linux_secure
host=webserver
etc/system/local/inputs.conf etc/apps/search/local/inputs.conf

Overriding Defaults
Y There are default settings in SPLUNK_HOME/etc/system/default and
SPLUNK_HOME/etc/apps/search/default
Y The correct method to override these settings, if needed, is to do so in the
local directory at the same scope
– Only add the items you are overriding—not a whole copy of the default conf file
Y Example:
– To disable a default attribute TRANSFORMS for [syslog]:
# etc/system/default/props.conf
[syslog]
TRANSFORMS = syslog-host
REPORT-syslog = syslog-extractions
...
# etc/system/local/props.conf
[syslog]
TRANSFORMS =


Reloading Configuration Files After Edit
Y Changes made using Splunk Web or the CLI do not require restart
– Amessage appears if restart is required (i.e. changing server settings)
Y Changes made manually are not automatically detected
Y To force reload, go to http://servername:webport/debug/refresh
– Reloads many of the configurations, including inputs.conf, but not all
Y To reload all configurations, restart Splunk
– Splunk Web: Settings > Server controls > Restart Splunk
– CLI: splunk restart

Lab Exercise 4 – Configuration Files
Y Tasks:
– Run the same search as different users
– Check the search results and compare
– Use the btool command to investigate the configurations

Module 5:
Splunk Index
Management

Module Objectives
Y Understand index structure and buckets
Y Create new indexes
Y Apply a data retention policy
Y Monitor indexes with DMC
Y Enable index integrity check
Y Reset monitor input checkpoints
Y Delete data from an index
Y Back up and restore frozen data

Y Splunk stores events in indexes under
SPLUNK_HOME/var/lib/splunk
–  SetinSettings > Server Settings
–  Canoverride on a per-index basis
Y The main index is
–  Usedwhenan input does not specifyanindex
–  Agoodexampleofahighvolumeindex
–  Locatedindefaultdbdirectory
Y Splunk users can explicitly specify which
index(es) to search
What are Indexes?
index=web action=purchase
main web
[Input X]
[Input Y]
index = web

Preconfigured Indexes
Splunk ships with several indexes already set up besides main:
summary – default index for summary indexing system
_internal – Splunk indexes its own logs and metrics from its processing here
_audit – Splunk stores its audit trails and other optional auditing information
_introspection – tracks system performance and Splunk resource usage data
_thefishbucket – contains checkpoint information for file monitoring inputs

Why Create Your Own Indexes?
Y Access control – segregate data into separate indexes to limit access by
Splunk role
Y More use cases are discussed inArchitecting and Deploying Splunk class
Web Team
Security Team
Web Index
web data
Security Index
security data

Y Retention
– Retention is set on a per-index basis
– Separate data into different indexes based on retention time
– Splunk data retention can be managed by data age and/or by size
Proxy Inputs
Why Create Your Own Indexes? (cont.)
Security Inputs
Web Inputs
Keep for 6 weeks
Keep for 12 months
Keep for 6 months Purge
Archive
Purge
web
security
proxy

Buckets
Y Internally, an index stores events in
buckets
Y Abucket is a directory containing a
set of rawdata and indexing data
Y Buckets have a maximum data size
and a time span
– Both can be configured
wiki.splunk.com/Deploy:UnderstandingBuckets
$SPLUNK_DB
Buckets restored
from archive
Hot & warm buckets
Cold buckets
Indexes

Index
Data Flow Through an Index
Y Hot:These are the newest buckets– still open for write
Y Warm: Recent data, buckets are closed (read only)
Y Cold: Oldest data still in the index (read only)
Y Frozen: No longer searchable; buckets either get archived or deleted
Archive
Delete
or
Inputs

Hot – Building Buckets
Y After data is read and parsed, it goes through the license meter and the
event is written into a hot bucket
Y All buckets are implemented as directories
– Hot buckets have a name that begins with hot_
– All buckets have unique identifiers within an index
Y When hot buckets reach their max size or time span, they are closed and
converted to warm status
– Hot buckets also roll to warm automatically when the indexer is restarted
– Hot and warm buckets are stored in the db directory for the index

Warm and Cold Bucket Names
Y Hot buckets are renamed when they roll to warm
– Bucket names identify the time range for the data they contain
– When a warm bucket rolls to cold, the entire bucket directory is moved
Y At search time, Splunk scans the time range on a bucket directory name to
determine whether or not to open the bucket and search inside
db_1389230491_1389230488_5
db_1390579247_1390579086_18
Youngest event in the
bucket
Oldest event in the bucket
Unique ID
Note
When clustering is used, index
replication adds further identifiers
to the directory name.

Freezing: Data Expiration
Y The oldest bucket is deleted from the index when:
– The index's maximum overall size is reached
– The bucket's age exceeds the retention time limit
ê  All the events in the bucket have expired
Y Splunk will never exceed the maximum overall size of an index
– Therefore, buckets can be deleted even if they have not reached the time limit
Y You can optionally configure the frozen path
–  Splunk copies the bucket's rawdata to this location before deletion
– Once frozen, buckets are not searchable
Y Frozen data can be brought back (thawed) into Splunk if needed

Index
Storing Buckets in a Separate Volume
Y Best Practice: Use a single high performance file system to store indexes
– The time span of the buckets and their storage type can affect search performance
Y However, you can use multiple volume partitions for index data
– Specify a separate volume for hot/warm and cold buckets during index creation
– Hot and warm buckets should be on the fastest partition and are searched first
– Cold can be located on a slower, cheaper storage (or SAN/NAS)
Y wiki.splunk.com/Deploy:BucketRotationAndRetention
On Fast SSD:
From -90d to Now
On Slower SAN/NAS
Older than -90d

Estimating Index Growth Rate
Y Splunk compresses raw data as it is indexed
– Index data is then added to each bucket
ê  If your data has many searchable terms, the index data is larger
ê  If the data contains fewer searchable terms and less variety, the index is smaller
Y Best practice: get a good growth estimate
– Input your data in a test/dev environment over a sample period
ê  You should index more than one bucket of data
– Examine the size of the index's db directory compared to the input
ê  DMC: Indexing > Indexes and Volumes > Index Detail: Instance
http://docs.splunk.com/Documentation/Splunk/latest/Capacity/Estimateyourstoragerequirements

Calculating Index Storage
Y Limiting size on disk is the most common method of controlling index growth
Y Allocate disk space to meet data retention needs
– Daily Rate * Compression Factor * Retention Period (in days) + Padding
Y Example: 5 GB/day of security data searchable for 6 months (with compression factor of .5)
– 900 GB (5 GB x 180 days) * .5 (CF) + 50 GB (padding) = 500 GB
– On average, data moves to frozen in this index after ~6 months
500

Managing Indexes with Splunk Web
Select Settings > Indexes
Click New to create a new index
Custom indexes can be enabled/disabled or deleted
Click an index name to edit it
Displays the app the index is
configured in and its home path

Adding an Index With Splunk Web
Specifynameforthenewindex
–  MustbeASCIIcharactersandcannotstartwithan“_”or“-”
Leaveblanktousethedefaultlocations
–  Default=$SPLUNK_DB/soc/[db|colddb|thaweddb]
–  Tousecustomlocations,specifyfullindexpaths
Enabledataintegritycheck(optional)
Setmaximumindexsize(default=500GB)
–  Thissettingoverridesallothersizeandretentionsettings
Setmaxsizeofabucket(default=auto(750MB))
–  Useauto_high_volume whendailyvolumeis >10GB
–  Or,provideaspecificsize
Specifythepathtoarchivetherawdatabuckets
Selectwheretheindexes.conffileshouldbesaved
300
soc

/mnt/ssd/soc/db

volume:raid/soc/colddb
6
5
4
3
2
1
6
5
4
3
2
1
auto_high_volume
7
7

Index Data Integrity Check
Y Provides an ability to validate that data has not been tampered with after indexing
docs.splunk.com/Documentation/Splunk/latest/Security/Dataintegritycontrol
Y When enabled, produces calculated hash files for auditing and legal purposes
– Works on index level (including clustering)
– Not for inflight data from forwarders
– To prevent data loss, use the indexer acknowledgment capability (useACK)
Y To verify the integrity of an index/bucket:
– splunk check-integrity -bucketPath [bucket_path] [verbose]
– splunk check-integrity -index [index] [verbose]
Y To re-generate hash files:
– splunk generate-hash-files [-bucketPath|-index] [bucket_path|index]

indexes.conf
Y Many advanced/optional attributes are not available in Splunk Web
– The index stanza is created in local/indexes.conf of the selected app
[volume:raid]
path = /datastore/splunk
[soc]
homePath = /mnt/ssd/soc/db
coldPath = volume:raid/soc/colddb
thawedPath = $SPLUNK_DB/soc/thaweddb
maxDataSize = auto_high_volume
maxTotalDataSizeMB = 307200
enableDataIntegrityControl = 0
Index name in
square brackets
You must specify
home, cold, and
thawed paths,
even when using
the defaults
Size of bucket set
to 10 GB
Setting the total
size of the index at
300 GB

Lab Exercise 5a – Add Indexes
Y Tasks:
– Create two new indexes: securityops and itops
– Add a file monitor input to send events to the securityops index.

Customizing Index Retention Policies
Y To set more advanced/specific options, edit the stanza in indexes.conf
Y New indexes default to 3 hot buckets at a time
– If it is likely that an index will receive events that are not in
time-sequence order, increase the number of available
hot buckets
Y High-volume indexes should have up to 10 hot buckets
– Set with the maxHotBuckets key
Y Best practice for high-volume indexes:
– Examine and copy settings of main index stanza and adjust for your case
Warning
Inappropriate retention settings
can cause premature bucket
rotation and even stop Splunk.

indexes.conf Options
Number of Hot buckets
(maxHotBuckets=3)
Age
(maxHotSpanSecs=7776000)
Bucket size
(maxDataSize = auto)
db directory size
(homepath.maxDataSizeMB=0)
Number of Warm buckets
(maxWarmDBCount=300)
Maximum Index Size (maxTotalDataSizeMB=500000)
Age (frozenTimePeriodInSecs=188697600)
Create Rename Move Delete or Archive
colddb directory size
(coldPath.maxDataSizeMB=0)
[itops]
frozenTimePeriodInSecs = 31536000
maxHotSpanSecs = 86400
homePath.maxDataSizeMB = 100000
coldPath.maxDataSizeMB = 400000
...
Note
Values in parenthesis
are the defaults.

Strict Time-based Retention Policies
Y Example: Purge HR data when it is more than 90 days old, but no sooner
Y Issues to consider:
– Splunk freezes entire buckets, not individual events
– If a bucket spans more than one day, you can't meet the 90 day requirement
Y Configuration option:
frozenTimePeriodInSecs=7776000 (90 days)
maxHotSpanSecs = 86400
ê  Automatically "snaps" the span to the beginning of the time period
ê  For 86400 (24 hours), hot buckets roll at midnight
ê  Consider increasing max. number of hot buckets: maxHotBuckets = 10

Volume-based Retention Policies
Y Example: Prevent data bursts in one index from triggering indexing issues
elsewhere in the same volume
– Splunk in itself cannot determine the maximum size for non-local volumes
– Hot/warm and cold buckets can be in different volumes
– If the volume runs out space, buckets roll to frozen before frozenTimePeriodInSecs
Y Configuration Options: Use volume reference if a retention based on size is desired
[volume:fast]
path = /mnt/ssd/
maxVolumeDataSizeMB = 500000
[volume:slow]
path = /mnt/raid/
maxVolumeDataSizeMB = 4000000
[soc]
homePath = volume:fast/soc/db
homePath.maxDataSizeMB = 50000
coldPath = volume:slow/soc/colddb
coldPath.maxDataSizeMB = 200000


Viewing Indexing Activity and Health
Y Distributed Management Console
– Provides comprehensive indexing activity details
– Snapshot shows averages over the previous 15 minutes
– Historical exposes trending and possible decaying health
Y  Queue fill-ratio
Y  Indexing rate
Y  CPU activity
Y  Volume usage per index
Y  Index size over time
Y  Detailed Indexing status
Y  Retention policies
Y  Bucket configuration details
Displays the usage vs. capacity,
if the volume-based retention is
used (maxVolumeDataSizeMB)

Inspecting Bucket Details
Y Search command: | dbinspect [index=name] [span|timeformat]
– Returns information like status, number of events, timestamps of oldest and
newest events, total bucket size, filepath, etc.
– docs.splunk.com/Documentation/Splunk/latest/SearchReference/Dbinspect

What to Back Up
Y Indexed event data – the Splunk index
– SPLUNK_HOME/var/lib/splunk/
– Or the directories where you placed your indexes (see indexes.conf for details)
Y (Optional) the source log data for additional redundancy
Y SPLUNK_HOME/etc for config and other important files
apps
users
system/local
licenses
init.d
passwd
andmore

Backup Recommendation
Y Use the incremental backup of your choice
– Warm and cold buckets of your indexes
– Configuration files
– User files
Y Hot buckets cannot be backed up without stopping Splunk
– Use the snapshot capability of underlying file system to take a snapshot of hot,
then back up the snapshot
– Schedule multiple daily incremental backups of warm
ê  Works best for high data volumes

Moving an Entire Index
1. Stop Splunk
2. Copy the entire index directory to new location while preserving
permissions and all subdirectories
– *NIX: cp -rp <source> <target>
– Windows: xcopy <source> <target> /s /e /v /o /k (or, robocopy)
3. If this is a global change, unset the SPLUNK_DB environment variable and
update SPLUNK_HOME/etc/splunk-launch.conf
4. Edit indexes.conf to indicate the new location
5. Start Splunk
6. After testing and verifying new index, the old one can be deleted

Removing Indexed Data
Y Sometimes you have unwanted data in an index
– There are no index editors
– First, you should change your configuration to omit the data in the future
Y What are your options for the data already in the index?
– Let the data age-out normally
– Use the delete command to make the unwanted data not show up in searches
– Run splunk clean command to delete all data from the index
– Delete the index

Deleting Events
Y Assign users to the can_delete role
– Specifically set up for deletions
– NOTrecommended to give this capability to other roles
ê  By default, even the admin role does not have the ability
Y Delete CANNOT be undone:
– Log into Splunk Web as a user of the can_delete role
– Create a specific search that identifies the data you want to delete
ê  Double check that the search ONLYincludes the data to delete
ê  Pay special attention to which index you are using and the time range
– After you are certain you’ve targeted only the data you want to delete,
pipe the delete command
Note
This is a "virtual" delete. Splunk
marks the events as deleted and
they never show in searches
again. However, they continue to
take up space on disk.

Cleaning Out an Index
Y To flush indexed data and reset an index, use the CLI clean command
– DATAWILLBE PERMANENTLYDESTROYED
– Typically used on test/dev systems, not for production systems
Y Command syntax:
splunk clean [eventdata|userdata|all] [-index name]
ê  eventdata – delete indexed events and metadata on each event
ê  userdata – delete user accounts
ê  all – everything - including users, saved searches, and alerts
– ALWAYS SPECIFYAN INDEXTOAVOIDTEARS
ê  If no index is specified, the default is to clean all indexes

The Fishbucket
Y The fishbucket index stores the checkpoint data for monitor inputs
Y To reset the individual input checkpoint, use the btprobe command:
Y Requires stopping the forwarder or indexer
Y Other options:
– splunk clean eventdata _thefishbucket
ê  Force re-indexing of all file monitors in the indexer
– rm -r ~/splunkforwarder/var/lib/splunk/fishbucket
ê  Manually delete the fishbucket on forwarders
splunk cmd btprobe –d SPLUNK_HOME/var/lib/splunk/
fishbucket/splunk_private_db --file <source> --reset
Note
Resetting the monitor checkpoint
re-indexes the data, resulting in
more license usage.

Restoring a Frozen Bucket
Y To thaw an archived bucket:
– Copy the bucket directory from the archive to the index's thaweddb directory
– Stop Splunk
– Run splunk rebuild <path to bucket directory>
ê  Also works to recover a corrupted directory
ê  Does not count against license
– Start Splunk
Y Data in thaweddb is searchable along with other data, is not frozen, and does not
count against index total size
– Delete the bucket directory when no longer needed and restart Splunk

Index Replication
Y Splunk indexers can function as a cluster
– Indexers in a cluster can be configured to replicate buckets amongst themselves
Y Index replication allows for rapid failure recovery
Y Fully configurable replication allows you to balance speed of recovery and
overall disk usage
Y Index replication requires additional disk space
Y Discussed in detail in Splunk ClusterAdministration class
Y Basic indexer cluster concepts:
– http://docs.splunk.com/Documentation/Splunk/latest/Indexer/Basicconcepts

Further Reading
Y docs.splunk.com/Documentation/Splunk/latest/Indexer/
HowSplunkstoresindexes
Y docs.splunk.com/Documentation/Splunk/latest/Indexer/
Setlimitsondiskusage
Y docs.splunk.com/Documentation/Splunk/latest/Indexer/Automatearchiving
Y wiki.splunk.com/Deploy:BucketRotationAndRetention

Lab Exercise 5b –Configure Retention Policies
Y Tasks:
– Configure a more strict time-based retention policy for securityops
– Configure a volume-based retention policy for itops
– Use DMC to monitor indexing activities and retention settings

Module 6:
Users, Roles, and
Authentication

Module Objectives
Y Describe user roles in Splunk
Y Create a custom role
Y Integrate Splunk with LDAP or SAML

Managing Users and Roles
Y Users and roles define user privileges
Y To have access to a Splunk instance, a
user must have:
– ASplunk user account
– Assignment to one or more Splunk roles
Y User accounts can be assigned to:
– Native Splunk accounts
– LDAPorActive Directory
– SAML
– Scripted access to PAM, RADIUS, or other
user account systems
Select Settings > Access controls

Identifying Roles
Y Five built-in user roles:
– admin, power, and user
ê  Users can be assigned these roles
– can_delete
ê  Will be discussed separately
– splunk-system-role
ê  Special role that allows system services to
run without a defined user context
Y Administrators can add custom user
roles

Defining Custom User Roles
Give the role a name and
select a default app
Optional
Restrict searches on
certain fields, sources,
hosts, etc
Default is -1
(no restriction)

Defining Custom User Roles (cont.)
Optional user-level and role-level limits

Describing Role Inheritance
Y Anew role can be based on one or more existing roles
Y The new role inherits both capabilities and index access
Y You cannot turn off inherited capabilities or access

Implications of Inheritance
Y If you create a new role that inherits from another role—such as, user:
– The new role has all the capabilities of the inherited role
ê  For example, "run real-time searches"
– The new role inherits the index settings (both default and allowed)
– In the new role, you cannot turn off capabilities or index access that were
inherited from the original role
Y If you want a role that is "like" user but with some capabilities turned off:
– Make a new role that does not inherit from any other role
– Turn on all of the same capabilities as in User, except those you want turned off
– Assign the appropriate indexes to the new role

Defining Role Capabilities
Add or remove capabilities (authorize.conf.spec)
docs.splunk.com/Documentation/Splunk/latest/Admin/authorizeconf

edit_roles_grantable Capability
Y Example: I want to separate and delegate administration tasks between
sys-admins and data admins without granting full admin role
– With edit_roles and edit_user capabilities, users can promote self to full
admin role
– Want to restrict grantable capabilities only to the level sub-admins currently have
– Add the edit_roles_grantable capability to the sub-admin role
ê  Can only create roles with subset of the capabilities that the current user role has
ê  Must use in conjunction with the edit_user capability

Example: edit_roles_grantable Capability
Add new role user_admin
Y Inheritance:
Y  power
Y  user
Y Capabilities:
Y  edit_roles_grantable
Y  edit_user
admin acurry
Assign acurry to the user_admin role acurry can only assign limited roles to users
New users

Defining Role Index Search Options
Y You can specify which indexes are searched if the user does
not specify "index=<index_name>"
– Usually, this should match the Indexes setting

Defining Role Index Access Options
Y Most important index setting
–  Controlswhichindexestheusersinthisrolecanaccess
–  Ifnotselected,userscannotsearchorevenseethisindex
Y Inherited indexes are still available even when they are not listed
Y The default is All non-internal indexes
–  Revisitaccesscontroleachtimeanewindexhasbeenadded

Splunk Authentication Options
Y You need the change_authentication Splunk capability to configure
Y Saves the settings in authentication.conf
– Splunk native
– LDAP
– SAML2.0
– Scripted SSO

Splunk Native Authentication
Y You can create user accounts directly in Splunk
– Example: the default admin user
Y Passwords are stored in SPLUNK_HOME/etc/passwd
Y Use a blank passwd file to completely disable native authentication BUT
– In all authentication scenarios, best practice is to keep a failsafe account here
with a VERYstrong password
Y You can have a mix of Splunk and LDAP or other users
– Splunk native authentication always takes precedence over others
Y Select Settings >Access Controls > Users to manage all users and to
create Splunk user accounts

Managing Users in Splunk
Y Splunk native users can be edited or deleted
Y Only time zone and default app can be changed on LDAPor other users
Add new Splunk user
Click to edit the user settings

Splunk Native Authentication: Add Users
Y Required:
– Username and password
Y Optional:
– Full name and email address (defaults to
none)
– Time zone (defaults to search head time zone)
– Default app (defaults to role default app, or
home if no role default app)
– Role(s)
ê  Defaults to user

Directory Server Integration
Y Best practice: integrate Splunk with a directory server
– Works with multiple LDAPservers, including OpenLDAPandActive Directory
– You can configure from Splunk Web settings
Y User accounts are stored in directory server
– Enforce the same user account and password policy
– Users use the same user name and password in Splunk that they use elsewhere
– Optionally, the groups in the directory server can be mapped to Splunk roles
ê  Or, this can be done manually in Splunk

LDAPAuthentication
LDAP maintains the user credentials - user ID and password, plus other
information - centrally and handles all authentication
Log user in Splunk
Web
Client
LDAP
Server
Splunk
Request Splunk login
Authentication granted
Check authentication
Create user session
Check group mapping

Creating an LDAP Strategy
1.  SelectLDAP
2.  ClickConfigure Splunk touse LDAP
–  Thelistofcurrent LDAPstrategies
displays
–  Astrategyisaconnectiontooneor
moreLDAPnodes on an LDAPserver
–  Candefinemultiple LDAPservers
3.  ClickNewtoaddanewLDAPstrategy
–  Namethestrategy and fillout the form
1
2
3

LDAP Strategy Settings
Y Normally the configuration is based on
the information given to you by the
LDAPadministrators
– LDAPconnection settings
– User settings
ê  DeterminewhichpartoftheLDAP
directorystoresSplunkusers
– Group settings & Dynamic group
settings
ê  Determinewhichnodeinthedirectory
containsyourgroupdefinitions
– Advanced settings
host = 10.0.0.150
port = 389
SSLEnabled = 0
bindDN = adsuser@buttercupgames.local
bindDNpassword = <some_hashed_pw>

userBaseDN = OU=splunk,DC=buttercupgames,DC=local
userNameAttribute = samaccountname
realNameAttribute = displayname

groupBaseDN = OU=splunk,DC=buttercupgames,DC=local
groupNameAttribute = cn
groupMemberAttribute = member
nestedGroups = 0
groupMappingAttribute = dn

network_timeout = 20
sizelimit = 1000
timelimit = 15

Mapping LDAP Groups to Roles
Select Map groups to define
relationships between LDAP
groups and Splunk roles
Click a LDAP group name to
map it to one or more Splunk
roles

Mapping LDAP Groups to Roles (cont.)
Y Not all groups must be mapped
Y Mappings can be changed at any time
– The LDAPserver is rechecked each time a user logs into Splunk
Click one or more role names
to map them to to this group
After completing the mapping for
all LDAP groups, the mapped
roles are shown here

Further Information about LDAP
Y Splunk caches user data from LDAP
Y New user data is cached the first time that user logs in
Y See docs for more details on setting up LDAPmapping:
docs.splunk.com/Documentation/Splunk/latest/Security/SetupuserauthenticationwithLDAP
Y Can also be done in config files: docs.splunk.com/Documentation/Splunk/latest/Admin/authenticationconf
Click this to force an
immediate reload

SAML 2.0 Single Sign On
Identity provider (IDP) maintains the user credentials and handles
authentication
Direct to default app
Web
Client
IDP
Splunk
Request Splunk login
Post Assertion (grant/deny)
Redirect authentication
Create session cookie
Validate assertion
Configured with trusted binding
Challenge for credentials
Check group mapping

Configuring SAML
Y Configure this on a search head:
1.  Select SAML
2.  Click Configure Splunk to use
SAML
ê  The list of current SAMLgroups
displays
3.  Click SAMLConfiguration to
configure the trusted binding and
other connection details
4.  Click New Group to map the
roles
ê  Group-to-Role mapping is same
as LDAP
1
2
3 4

Configuring SAML (cont.)
1.  Enter the following info and save: (provided by the IDPadministrator)
– Single Sign On (SSO) URL
– IDP's certificate file
– Entity ID
– Attribute query URL
– User / Password
2.  Export Splunk (Service Provider) metadata:
http://<splunk_web>/saml/spmetadata
– IDPadministrator imports it into its system and
configures its settings
– IDPadministrator exports its configured metadata
3.  Import IDPmetadata and update the SAML
configuration settings

Splunk
Single Sign On with Reverse Proxy
Y Splunk SSO allows you to use a web proxy to handle Splunk authentication
–  Authentication is moved to a web proxy,whichpassesalongauthenticationtoSplunkWeb
–  Webproxycanuseanymethodtoauthenticate(IDPinexample)
docs.splunk.com/Documentation/Splunk/latest/Security/HowSplunkSSOworks
Web
Client
Splunk request
Proxy passes request with user name
Proxy authorizes client
Proxy
Server
IDP
Splunk Web returns page to proxy
Proxy returns page

Scripted Authentication
Y There are other types of authentication systems that Splunk can integrate
with using scripts
Y For the most up-to-date information on scripted authentication, see the
README file in:
SPLUNK_HOME/share/splunk/authScriptSamples/
– The directory includes sample authentication scripts

Lab Exercise 6 – Add Roles and Users
Y Tasks:
– Create a custom role
– Configure Splunk to use LDAPauthentication
– Map LDAPgroups to Splunk roles
– Verify the configurations
Y Lab notes:
– open.sesam3 is the password for all LDAPaccounts

Module 7:
Universal Forwarders

Module Objectives
Y Install a universal forwarder
Y Configure the forwarder to connect to an indexer
Y Test the forwarder connection
Y Describe optional forwarder settings

Forwarders and Indexers
Y In a production environment
– Splunk indexer(s) runs on dedicated
servers
– The data you want is on remote machines
Y Install Splunk forwarders on the remote
machines to
– Gather the data
– Send it across the network to the Splunk
indexer(s)
Y Indexers listen on a receiving port for
the forwarded data

Splunk Universal Forwarder
Y Universal forwarder gathers data from a host and sends it to an indexer
Y Specifically designed to run on production servers
– Alightweight Splunk instance designed to run on a mission-critical system
– Minimal CPU and memory usage
– Output bandwidth constrained to 256 KBps by default
– No web interface
Y Aseparate installation binary
– Free built-in license, no limits
Y Best Practice: use the Universal Forwarder if possible

Configuration Steps
1. Set up a receiving port on each indexer
– It is only necessary to do this once
2. Download and install Universal Forwarder
– Change password from changeme
3. Set up forwarding on each forwarder
4. Add inputs on forwarders, using one of the following:
– Forwarder management (discussed in the next module)
– CLI
– Manually

Configure the Receiving Port on Each Indexer
Y In Splunk Web:
1.  Select Settings > Forwarding and receiving
2.  Next to Configure receiving, select Add new
3.  Enter a port number and click Save
Y  Or, with CLI:
splunk enable listen <port>
Y  The configuration is saved in
inputs.conf as:
[splunktcp://portNumber]

Installing Universal Forwarder Manually
Y *NIX: unpack the .tgz or .tgz.Z in the desired location
Y Windows: execute the .msi or use the command line
– Installed as a service
Y SPLUNK_HOME is the installation directory:
– /opt/splunkforwarder or c:Program FilesSplunkUniversalforwarder
Y Same splunk command-line interface in SPLUNK_HOME/bin
– Same commands for start/stop, restart, etc.
– The initial admin account password is changeme
ê  Use splunk edit user admin –password newpassword
Y When installing large numbers of forwarders, use an automated method

Using the Interactive Windows Installer
Y Most forwarder settings can be configured using the installer wizard
– Can run as a domain user without the domain user local administrator privileges
Y CLI installation is available for scripted installations
–  docs.splunk.com/Documentation/Splunk/latest/Forwarding/DeployaWindowsdfviathecommandline

Forwarder Configuration Files
Y Forwarders require outputs.conf
– outputs.conf points the forwarder to the receiver(s)
– Can specify additional options for load balancing, SSL, compression, alternate
indexers, and indexer acknowledgement
[tcpout:splunk_indexer]
server = 10.1.2.3:9997
Production Server with Forwarder
[splunktcp://9997]

outputs.conf inputs.conf
Data feeds
from
inputs.conf
Receiver 10.1.2.3 (indexer)
TCP
stream to
port 9997
This stanza instructs the indexer to
listen on port 9997 for feeds from
Splunk forwarders
server includes one or more target receivers,
separated by commas
server can be IP or DNS name plus receiver port

Defining Target Indexer on the Forwarder
Y Run: splunk add forward-server indexer:receiving-port
– For example, splunk add forward-server 10.1.2.3:9997
configures the outputs.conf as:
docs.splunk.com/Documentation/Splunk/latest/Forwarding/Configureforwarderswithoutputs.confd
[tcpout]
defaultGroup = default-group
[tcpout:default-group]
server = 10.1.2.3:9997
[tcpout-server://10.1.2.3:9997]

Testing the Connection
Y After running splunk add forward-server, the forwarder should be
communicating with the indexer
– Splunk forwarder logs are automatically sent to the indexer's _internal index
Y To check for successful connection:
– On the indexer, search index=_internal host=forwarder_hostname
– On the indexer, run splunk display listen
– On the forwarder, run splunk list forward-server
Y To remove the target indexer setting:
– On the forwarder, run splunk remove forward-server indexer:port

Troubleshooting Forwarder Connection
Y Is the forwarder sending data to the indexer?
– Check SPLUNK_HOME/var/log/splunk/splunkd.log on the forwarder
Y Does the indexer receive any data on the listening port?
– Search on indexer:
– To get the <uf>, run on the forwarder:

tail –f var/log/splunk/splunkd.log | egrep 'TcpOutputProc|TcpOutputFd'
index=_internal sourcetype=splunkd component=TcpInputConfig OR
(host=<uf> component=StatusMgr)
splunk show default-hostname

Additional Forwarding Options
Y Compressing the feed
Y Securing the feed
Y Automatic load balancing to multiple indexers
Y Forwarder queue size
Y Indexer acknowledgement to forwarder
Y Selectively forwarding data to indexers

Compressing the Feed
Receiving indexer 10.1.2.3
server = 10.1.2.3:9997
compressed = true

Forwarders
[splunktcp:9997]
compressed = true

Set compression on both sides
(slightly increases CPU usage)
Compressed
stream
Non-compressed feeds for this
port are ignored

Securing the Feed – SSL
Receiver 10.1.2.3
server = 10.1.2.3:9997

sslPassword = ssl_for_m3
sslCertPath = SPLUNK_HOME/etc/auth/cert1/server.pem
sslRootCAPath = SPLUNK_HOME/etc/auth/cert1/cacert.pem
Forwarders
[splunktcp-ssl:9997]

[ssl]
password = ssl_for_m3
serverCert = SPLUNK_HOME/etc/auth/cert1/server.pem
rootCA = SPLUNK_HOME/etc/auth/cert1/cacert.pem

outputs.conf
inputs.conf
Secure
Feed
SSL settings for all inputs in
dedicated SSL stanza
Turning on SSL:
Y  Can increase the CPU usage
Y  Automatically compresses the feed

Notes About SSL
Y Splunk uses OpenSSLto generate its default certificates
– Default certificate password is password
Y You should use external certs OR create new ones using Splunk’s
OpenSSL
Y docs.splunk.com/Documentation/Splunk/latest/Security/
AboutsecuringyourSplunkconfigurationwithSSL
Y docs.splunk.com/Documentation/Splunk/latest/Security/Aboutsecuringdatafromforwarders
Y wiki.splunk.com/Community:Splunk2Splunk_SSL_DefaultCerts
Y wiki.splunk.com/Community:Splunk2Splunk_SSL_SelfSignedCert_NewRootCA
Y wiki.splunk.com/Community:Splunk2Splunk_SSL_3rdPartyCA

Automatic Load Balancing
Y Automatic load balancing switches from server to server in a list based on a
default 30 second time interval
– Switch happens only when the forwarder detects an EOF
Y Load balancing is the key to making distributed search or clustering work
docs.splunk.com/Documentation/Splunk/latest/Forwarding/Setuploadbalancingd
[tcpout:splunk_indexer] 
server = splunk1:9997,splunk2:9997,splunk3:9997
autoLB = true
forceTimebasedAutoLB = true
autoLBFrequency = 40

Caching/Queue Size in outputs.conf
Y maxQueueSize = 500kb (default) is the maximum amount of data the
forwarder queues if the target receiver cannot be reached
– In load-balanced situations, if the forwarder can’t reach one of the indexers, it
automatically switches to another and only queues if all indexers are down or
unreachable
Y See outputs.conf.spec for details and more queue settings

Indexer Acknowledgement
Y Guards against loss of data when
forwarding to an indexer
– Forwarder resends any data not
acknowledged as "received" by the indexer
Y Disabled by default
Y Can also be used for forwarders sending to an intermediate forwarder
Y Automatically increases the wait queue to 3x the size of maxQueueSize to
meet larger space requirement for acknowledgement
docs.splunk.com/Documentation/Splunk/latest/Forwarding/
Protectagainstlossofin-flightdata
useACK = true
...

Selectively Forwarding Data to Indexers
Y Example:
– QAteam wants metrics.log sent to the QAteam’s indexer and Ops team wants
runtime.log sent to the operations indexer
Y Universal forwarder can route based on sources
– Define multiple tcpout stanzas in outputs.conf
– Specify a TCP_ROUTING identifying the tcpout stanza names in each source in
inputs.conf
QA

Ops
[tcpout:QA]
server=srv.qa:9997
[tcpout:Ops]
server=srv.ops:9997
[monitor://path/Metrics.log]
_TCP_ROUTING = QA
[monitor://path/Runtime.log]
_TCP_ROUTING = Ops

Configuring Forwarder Inputs
Y You have several options for creating inputs on a new forwarder:
– Use Forwarder Management to automatically deploy input configurations
– Run CLI commands
– Install add-ons
– Manually configure inputs.conf
Y inputs.conf on the forwarder gathers the local logs/system info needed
– Set metadata values for each source for items like sourcetype, host, index, etc.
– Per-event processing (parsing) must be done on the indexer
wiki.splunk.com/Where_do_I_configure_my_Splunk_settings
Y You will configure inputs starting in module 9

Add-ons
Y Apps and add-ons can be installed
on a forwarder in SPLUNK_HOME/
etc/apps
Y Installation is the same on the
forwarder as on an indexer
– However, the GUI options are not
available
Y See the add-on's documentation for
details about its settings for
inputs.conf

Forwarder Monitoring with DMC
Y DMC can provide valuable information on forwarder activity and throughput
– Once enabled, it runs a scheduled search to build a forwarder asset table
– Runs every 15 minutes
by default
– Relies on the internal logs
forwarded by forwarders
– Can affect the search workload
if you have many forwarders
– Can rebuild manually
3
2
1

Forwarder Monitoring with DMC (cont.)
3
2
1

Forwarding Resources
Y Overview of forwarders
docs.splunk.com/Documentation/Splunk/latest/Data/Usingforwardingagents
Y Forwarder deployment overview
docs.splunk.com/Documentation/Splunk/latest/Forwarding/Deploymentoverview
Y Overview of enterprise installation -- link at the bottom of the web page has
example install packages and Windows install script
wiki.splunk.com/Deploying_Splunk_Light_Forwarders

Lab Exercise 7 – Setting up Forwarders
Y Tasks:
– Set up a forwarder on a remote Linux system (10.0.0.50)
– Enable DMC forwarder monitoring
– Verify the forwarder status
Y Lab notes:
– You have a login on a remote Linux host that is your forwarder
– Configure your UF to talk to your own indexer (Linux or Windows)
– This lab exercise only establishes the connection between your UF and Indexer
ê  Youwillconfigureactualinputsinlabexercise9
– See environment topology on the next slide

Lab Exercise 7 – Setting up Forwarders (cont.)
PuTTY > ssh <you>@10.0.0.50
Provision:
1.  Add a receiving port to your indexer
2.  Configure your UF to send data to
your indexer:receiving port
(10.0.0.2xx:9997)
Your UF
splunk add forward-server 10.0.0.2xx:9997
OR
Your
Computer
ssh <you>@10.0.0.50
9997
9997
1
1
2

Lab Exercise 7 – Setting up Forwarders (cont.)
Verification:
1. Enable DMC forwarder monitoring
2. Check forwarder instance
http://{EIP}:8000
Your
Computer

Module 8:
Forwarder Management

Module Objectives
Y Describe Splunk Deployment Server
Y Explain the use of Forwarder Management
Y Configure forwarders to be deployment clients
Y Deploy apps using Forwarder Management

Deployment Management
Y Deployment Server is the tool for distributing configurations, apps, and
content updates to groups of Splunk Enterprise instances
– Allows you to manage remote Splunk instances centrally
Y Forwarder management is a graphical interface built on top of deployment
server
– Handles the job of sending configurations (inputs.conf, outputs.conf, etc.)
– Can automatically restart the remote Splunk instances
Y DMC Forwarders dashboards help you monitor the deployment

Deployment Server
Y Acentralized configuration management tool to manage Splunk
configuration files
Y Deployment server identifies clients and subscribes them to server classes
– Aserver class defines a group of Splunk components (apps) and its member
criteria
– Each deployment client polls the server and then pulls the apps it is subscribed to
Y To enable deployment server component, you need an enterprise license
– Contact Splunk support for a special enterprise license for this capability
Y Cannot be used to install or upgrade Splunk binaries
Y Best practice: dedicate a Splunk instance as a deployment server
– In this class, you will use your indexer as a deployment server

Ports and Configurations
Deployment Clients: deploymentclient.conf
Y  etc/system/local/serverclass.conf defines who gets what
Y  etc/deployment-apps stores the apps for distribution
<app>
outputs.conf defines the recipient(s) of the data
Mgmt. Port
Receiving Port Receiving Port
Indexer 1 Deployment Server Indexer 2
Management
(splunkd port: 8089)
Data
(receiving port: xxxx)
Obtain deployment apps
(configuration bundles)
from the Deployment
Server
1
2
3
4

Implementation Overview
Y To set up Forwarder Management in your implementation:
1.  On the deployment server, add one or more apps in
SPLUNK_HOME/etc/deployment-apps
2.  In the Forwarder Management UI, create one or more server classes
3.  On forwarders, run splunk set deploy-poll <deployServer:port>
ê  Where port is the splunkd port on the deployment server - 8089 is the default
4.  Verify on deployment server:
ê  List of clients phoning home
ê  Deployment status
5.  Verify on forwarders:
ê  etc/apps folder for deployed apps

What's in a Deployment App?
Y Forwarder Management works by deploying one or more apps from the
SPLUNK_HOME/etc/deployment-apps folder to the remote forwarders
– They are deployed to the forwarder's SPLUNK_HOME/etc/apps folder by default
Y An app can have configuration files, scripts, and other resources
– Apps must follow the normal app structure and rules. Required files:
ê  app.conf (in default or local)
ê  local.meta (in metadata)
Y Best practice
– Create small and discrete deployment apps
– Take advantage of .conf file layering
– Use a naming convention
deploymentapps
MyApp
default local metadata
app.conf local.meta

Ways to Group Clients -- Server Classes
Y Aserverclassmapsaclientgrouptooneormore
deploymentapps
Y Asetofclientscanbegroupedbasedon:
–  Clientname,hostname,IPaddress,orDNSname
–  Machinetypes
Y Examples:
–  Windowsserverclass
ê  SystemsrunningWindowsgetApp1
–  Net10serverclass
ê  Hostson10.1.2.*subnetgetApp2andApp3
–  ADserverclass
ê  ADserversgetApp3
Y Noticethatclients(liketheLDAPserver)can
belongtomultipleserverclasses
www1
10.1.2.3
www2
10.1.2.4
Windows
App1
Net10
App2
App3
AD
App3

HR
20.9.8.7
LDAP
20.9.8.6
Server
Classes
Deployment
Clients
(Forwarders)
Deployment Server

Adding a Server Class
Select the Server
Classes tab
1
Enter a name for the
new server class
3
2
2

Selecting Apps for the Server Class
3
Select apps
2
1
stream

Post Deployment Behavior Setting
Click the app's
Edit link
1
Make sure Restart
Splunkd is enabled
2
3

Client Grouping
3
Enter Include, Exclude, and/or Machine Type filters
2
1
Y In addition to include/exclude, you can
further filter based on machine types
Y The list is based on the clients that have
connected to this deployment server
Y Supports wildcards
Y Exclude takes precedence over include

Configuring Deployment Clients
Y Configure your forwarders to be deployment clients
–  Runthisduringforwarderinstallationorlater:
splunk set deploy-poll deployServer:port
ê  deployServer=deploymentserverhostnameorIP
ê  port=splunkdport
ê  Createsdeploymentclient.confin
SPLUNK_HOME/etc/system/local/
–  Restartthedeploymentclients:
splunk restart
Y To override the default attributes,
edit the [deployment-client] stanza
–  Canbeapartofinitialdeploymentapp
–  Theforwarder"phoneshome"onceaminute(bydefault)
[target-broker:deploymentServer]
targetUri = splunk_server:8089
[deployment-client]
clientName = webserver_1
phoneHomeIntervalInSecs = 600
deploymentclient.conf
deploymentclient.conf

Finishing Up (Deployment Server)
Y On the deployment server
– Check the app deployment status, or run
splunk list deploy-clients
ê  Output should be a list of remote forwarders and details about their connections
– Check that the expected data is arriving on the indexer
– To manually force the deployment server to rescan the apps for changes, use
splunk reload deploy-server
ê  Clients retrieve the reloaded apps at their next scheduled check-in
ê  Once the deployment server and clients are set up, this is often easier than using the
Forwarder Management interface

Finishing Up (Deployment Client)
Y On the deployment client (usually a forwarder)
– Confirm that the expected app directories and contents have arrived in
SPLUNK_HOME/etc/apps
ê  Any changes in the selected apps on the deployment server are automatically
updated
ê  If the post deployment behavior option is set, the forwarder is restarted
–  Ifyouwanttochange,usetheapp’sEditmenuassociatedwiththeserverclass
– Use splunk show deploy-poll to check the deployment server settings
– Use splunk list forward-serverto check the indexer destination settings

Lab Exercise 8 – Setting up Forwarder Management
Y Tasks:
– Enable the Forwarder Management UI
– Enable deployment client on the forwarder, 10.0.0.100
– Use Forwarder Management to configure the forwarder
ê  Deploy an app that configures the forwarder’s outputs.conf
– Verify the deployment

Lab Exercise 8 – Setting up Forwarder Management (cont.)
RDC/ssh {EIP}
Provision:
1.  Using Splunk Web, configure
the deployment server
2.  Remote ssh into the forwarder
and configure the client
Your UF
Your Splunk Server
ssh <you>@10.0.0.100
set deploy-poll
Your
Computer
8089
9997
Management channel
Data channel

Lab Exercise 8 – Setting up Forwarder Management (cont.)
Verification:
1.  Check with DMC
2.  Run a search to get the forwarded internal logs
index=_internal host="eng*"
http://{EIP}:8000
Your
Computer

Module 9:
Getting Data In

Module Objectives
Y Identify the types of data you can index
Y Describe the basic settings for an input
Y Configure a file monitor input with Splunk Web

Y  Computers
Y  Network devices
Y  Virtual Machines
Y  Internet of Things (IoT)
Y  Communication devices
Y  Sensors
Y  Databases
Y  Any source
Got Data?
Y  Logs
Y  Configurations
Y  Messages
Y  Call Detail Records
Y  Clickstream
Y  Alerts
Y  Metrics
Y  Scripts
Y  Changes
Y  Tickets
Y  Any data
Indexes any data from any source

Splunk Index Time Process
Y Splunk index time process (data ingestion) can be broken down into three phases:
1.  Input phase: handled at the source (usually a forwarder)
ê  Thedatasourcesarebeingopenedandread
ê  Dataishandledasstreamsandanyconfigurationsettingsareappliedtotheentirestream
2.  Parsing phase: handled by indexers (or heavy forwarders)
ê  Dataisbrokenupintoeventsandadvancedprocessingcanbeperformed
3.  Indexing phase:
ê  Licensemeterrunsasdataisinitiallywrittentodisk,priortocompression
ê  Afterdataiswrittentodisk,itcannot bechanged
Indexer
Source Server
Universal Forwarder
Inputs
Indexing
Forward Parsing
License
Meter
Disk

Input vs. Parsing
Input phase Parsing phase
Y  Most configuration done in
inputs.conf on forwarder
Y  Some configuration is in
props.conf
Y  Acquire data from source
Y  Convert character encoding
Y  Set initial metadata fields: source,
sourcetype, host, index, etc.
Y  Operates on the entire data stream
Y  Most efficient, but low
discrimination
Y  Most configuration done in
props.conf on indexer
Y  Also: transforms.conf
Y  Apply stream-level substitutions
Y  Break data into events with
timestamps
Y  Apply event-level transformations
Y  Fine-tune metadata settings from
inputs phase
Y  Operates on individual events
Y  Less efficient, but fine control

Data Input Types
Y Splunk supports many types of data input
– Files and directories: monitoring text files, either single or entire directories
– Network data: listening on a port for network data
– Script output: executing a script and using the output from the script as input
– Windows logs: monitoring Windows event logs,Active Directory, etc.
– And more...
Y You can add data inputs with:
– Apps and add-ons from Splunk Base
– Splunk Web
– CLI
– Directly editing inputs.conf

Default Metadata Settings
Y When you index a data source, Splunk assigns metadata values
– The metadata is applied to the entire source
– Splunk applies defaults if not specified
– You can also override them at input time or later
Metadata Default
source Path of input file, network hostname:port, or script name
host Splunk hostname of the inputting instance (usually a forwarder)
sourcetype Uses the source filename if Splunk cannot automatically determine
index Defaults to main

Input Staging
Y Production data is usually on a remote system and is not on the indexer
– Normally data comes from one or more Splunk forwarders
Y For testing, you can use Splunk Web to sample a log file on a test server
Y Use Add Data to do this on the test server
– Check to see if sourcetype and other
settings are applied correctly
– If not, delete the test data, change your
test configuration, and try again
Input Index Search

Adding an Input with Splunk Web
Y Splunk admins have a number of ways to start the Add Data page
– Click the Add Data icon
ê  On the admin's Home page
ê  On the Settings panel
– Select Settings > Data inputs > Add new

Add Data Input Options
The location of your source dictates which option to use
Useful for testing Monitor files on
local server
Monitor files on
remote hosts

Select Source
2
1
3
Select the Files &
Directories option to
configure a monitor input
To specify the source:
Y  Enter the absolute path to a
file or directory, or
Y  Use the Browse button
For one-time indexing (or testing);
the Index Once option does not
create a stanza in inputs.conf
For ongoing monitoring

Select Source on Windows Indexer
Y On Windows indexers, there are additional
Windows-specific source options
Y  To monitor a shared network drive, enter the
path manually:
– <host>/<path> on *nix
– <host><path> on Windows
– Make sure Splunk has read access to the
mounted drive

Set Sourcetype
1
2
3
4

Set Sourcetype (cont.)
Splunk automatically determines the source type for major data types
You can choose a different sourcetype from the dropdown list
Or, you can create a new sourcetype name for the specific source
Data preview displays how your processed events will be indexed
– If the events are correctly separated and the right timestamps are highlighted,
you can move ahead
– If not, you can select a different sourcetype from the list or customize
4
3
2
1

Understanding Sourcetypes
Y sourcetype is Splunk’s way of categorizing the type of data
– Splunk indexing processes frequently reference sourcetype
– Many searches, reports, dashboards, apps, etc. also rely on sourcetype
Y Splunk will try to determine the sourcetype for you
– If Splunk recognizes the data, then it assigns one from the pre-trained sourcetypes
– If one is explicitly specified in inputs.conf, then Splunk will not try to determine the
sourcetype
ê  YoucanexplicitlysetsourcetypewithSplunkWeb,CLI,orbymodifying inputs.conf
– Otherwise, Splunk uses the name of the file as the sourcetype
Y You can also add sourcetypes by installing apps, which often define sourcetypes for
their inputs

Pre-trained Source Types
Y Splunk has default settings for many types
of data:
http://docs.splunk.com/Documentation/Splunk/
latest/Data/Listofpretrainedsourcetypes
Y This page also contains a list of sourcetypes
that Splunk automatically recognizes
You can customize your sourcetype configuration by
copying the attributes of the pre-trained sourcetype

Input Settings
Y  The app context determines where your input
configuration is saved
Y  In this example, it will be saved in:
SPLUNK_HOME/etc/apps/search/local
Y  By default, the default host name in General
settings is used
Y  You will learn about other options in later
modules
Y  Select the index where this input should be stored
Y  To store in a new index, first create the new index

Review
Review the input configuration summary and click Submit to finalize

What Happens Next?
Y Indexed events are available
for immediate search
– However, it may take a minute
for Splunk to start indexing the
data
Y You are given other options to
do more with your data
Y The input configuration is
saved in etc/apps/<app>/
local/inputs.conf

Verify your Input
Y Click Start Searching or
search for index=<test_idx>
Y Confirm the host, source, and
sourcetype field values
Y Verify the event timestamps
Y Check the auto-extracted field
names

View Configured Inputs
Inputs handled by remote
instances but configured
from this deployment server
Inputs handled by this server

View Configured Inputs (cont.)
Location of configuration
Indexing destination
Click to edit existing input settings

Inputs.conf
Y To put the input into production, edit the
target index setting in inputs.conf, or
Y Repeat theAdd Data steps
– Inputs you create are
saved in the target app's
local/inputs.conf file
[monitor:///opt/log/www1/access.log]
disabled = false
host = currdev www_ca
index = testinputs web
sourcetype = access_combined_wcookie

What to Monitor with DMC - Snapshot
Note
You will learn more about
detecting problems with this view
in later modules.
After an initial spike, a steady
rate can help you calculate
the daily indexing rate
The snapshot panel shows averages
over the previous 15 minutes

What to Monitor with DMC - Historical
The historical panel exposes trending
and possible decaying health
Filters the view
Roll-over to further
filter the view

Lab Exercise 9 – Add a Data Input
Time: 15 – 20 minutes
Tasks:
– Using Splunk Web, create a test index
– Index a log into the test index with the Index Once option
– Verify the indexed events with their metadata values
– Edit the same input to index it into the websales index
– Locate the saved input stanza in the inputs.conf file
– Monitor the indexing rate of this input with DMC

Module 10:
Monitor Inputs

Module Objectives
Y Create file and directory monitor inputs
Y Use optional settings for monitor inputs
Note
From this point on, you will
assume a typical scenario where
you are collecting data with
Universal Forwarders and sending
the data to an indexer.

Monitoring Files
Y Amonitor input can declare a specific file as the source
– The current content of the file is read
– The file is continuously monitored for new data
– Splunk tracks file status and automatically starts monitoring at the correct file
location after a restart
Y The file monitor supports any text file format, such as:
– Plain text log files
– Structured text files, such as CSV, XML, JSON
– Multi-line logs, such as Log4J
– Splunk can also read files compressed with gzip

Monitoring Directories
Y You can specify directory trees as monitor input sources
– Splunk recursively traverses through the specified directory trees
– All discovered text files are consumed, plus compressed files if possible
ê  Unzipscompressedfilesautomaticallybeforeindexingthem,oneatatime
– Any files added to the directory tree in the future are included
ê  Automaticallydetectsandhandleslogfilerotation
Y The input settings are applied to all files in the tree
– sourcetype, host and index -- if specified -- are applied to all files in the tree
– source= the file name
– Automatic sourcetyping is necessary for directories that contain mixed file types
ê  Canoverrideexceptionsmanually
ê  Discussedinalatermodule

Monitor Input Options in inputs.conf
Y Source (after monitor:// in stanza header) is an
absolute path to a file or directory
– Can contain wildcards
Y All attributes (sourcetype, host, index, etc.) are
optional
Y Defaults apply if omitted
– Default host is defined in etc/system/local/
inputs.conf
– Default source is the fully-qualified file name
– Default sourcetype is automatic
Y There are many possible attributes
– See inputs.conf.spec
[monitor://<path>] 
disabled=[0|1|false|true]
sourcetype=<string>
host=<string>
index=<string>
blacklist=<regular expression>
whitelist=<regular expression>
[monitor:///var/log/secure]
[monitor://C:logssystem.log]
[monitor://C:logs]
[monitor:///var/log/]

File Pathname Wildcards
Monitor stanzas in inputs.conf support two wildcards to help you specify
the files/directories you want to index
Wildcard Description
... The ellipsis wildcard recurses through directories and
subdirectories to match.
* The asterisk wildcard matches anything in that specific
directory path segment but does not go beyond that segment
in the path. Normally it should be used at the end of a path.

File and Directory Matching
[monitor:///var/log/www1/secure.log]
sourcetype = linux_secure
[monitor:///var/log/www1/secure.*] 
[monitor:///var/log/.../secure.*] 
✓ /var/log/www1/secure.log
✓ /var/log/www1/secure.1
✓ /var/log/www1/logs/secure.log
✓ /var/log/www1/secure.1
✗ /var/log/www1/logs/secure.log
✗ /var/log/www1/secure.1
✗ /var/log/www1/logs/secure.log
✓  Matches
✗ Doesn't match

Additional Options
Y Whitelist and Blacklist
– Regular expressions to filter files or directories from the input
docs.splunk.com/Documentation/Splunk/latest/Data/Whitelistorblacklistspecificincomingdata
– In case of a conflict between a whitelist and a blacklist, the blacklist prevails
Y Follow tail (followTail)
– Splunk ignores existing content in the file, but indexes new data as it arrives
– DO NOTleave followTail enabled in an ongoing fashion
– Consider using ignoreOlderThan, if applicable
ê  Afile whose modtime falls outside this time window will not be indexed

Example: Using Whitelist to Include Files
Y Files/directories that match the regular expression are indexed
Y The syntax for blacklists is identical
[monitor:///var/log/www1/]
whitelist = .log$
[monitor:///var/log/www1/] 
whitelist = query.log$|my.log$
[monitor:///var/log/www1/] 
whitelist = /query.log$|/my.log$
✓ /var/log/www1/query.log
✓ /var/log/www1/dbquery.log
✓ /var/log/www1/my.log
✗ /var/log/www1/my.log4j
✓ /var/log/www1/access.log
✓ /var/log/www1/dbaccess.log
✓ /var/log/www1/access.1.log
✗ /var/log/www1/access.log.2
✓ /var/log/www1/query.log
✓ /var/log/www1/my.log
✗ /var/log/www1/dbquery.log
✗ /var/log/www1/my.log4j ✓  Matches
✗ Doesn't match

Overriding the Host Field
Y Normally on a forwarder, the host can be
left to its default value
Y In some cases, the data might be stored on
a different server than its origin
– For example, a web farm where each web
server stores its log file on a centralized file
server
Y You can override the default host value
– Explicitly set the host
– Set the host based on a directory name
– Set the host based on a regular expression
www1 www2 www3
log_server_1

Y host_segment = <integer>
– Setting host_segment to 3 uses the 3rd segment of the directory path as the host name
for files in that directory
Setting the Host: host_segment
[monitor:///var/logs/] 
host_segment=3
3

Y host_regex = <regular expression>
– Setting host_regex to (vmail.+).log$selects the second part of the log file name
as its host name
(vmail.+).log$
Overriding the Host: host_regex
[monitor://C:varlogs] 
host_regex=(vmail.+).log$

Provisioning Remote Data Input
Once deployment clients are working, you can configure inputs on the clients
from the Deployment Server

Provisioning Remote Data Input (cont.)
Y  Configure basic settings only
Y  No data preview

Select Source for Windows UF
Remote data input for Windows UF
Local data input on Windows Indexer

Editing Remote Data Input
1
2
3

Monitoring with DMC: Splunk TCP Inputs
For remoteinputmonitoring, clickIndexing>Inputs>SplunkTCPInputPerformance

Lab Exercise 10 – File Monitors
Y Tasks:
– To test-collect remote data, add a remote directory monitor input to the
testinputs index
– Modify the inputs.conf file of the remote directory monitor and re-deploy
ê  Send the source logs to the websales index
ê  Use host_segment for the host name
ê  To monitor only the www.* sub-directories, use whitelist

Module 11:
Network & Scripted Inputs

Module Objectives
Y Create network (TCP and UDP) inputs
Y Describe optional settings for network inputs
Y Create a basic scripted input

Network Inputs
Y ASplunk instance (forwarder or indexer) can listen on aTCPor
UDPport for incoming data
– Syslog is a good example of network-based data
Y Add theTCPor UDPinput on a forwarder if possible
– Adds a layer of resiliency to your topology
ê  Buffering,loadbalancing,cloning,etc.
ê  IndexerrestartwillnotcausedatalossofTCPorUDPinput
– Minimizes the workload that must be done by the indexer
ê  Managethenetworkconnectionsontheforwarder
ê  Canalsobeusefultobridgenetworksegmentsifneeded Switches Routers Sensors

Adding Network Input
9001
10.1.2.3
dns_10-1-2-3
If not specified, Source defaults to:
Y  TCP: tcp:<port>
Y  UDP: udp:<port>
If specified, only accepts connections
from this host
Y  Otherwise, all hosts are allowed

Optional Network Input Settings
Y You can fine-tune the input settings
by editing the stanza directly
– Metadata override
– Sender filtering options
– Network input queues
ê  Memory queues
ê  Persistent queues
– These settings are described on the
following slides
[udp://[host:]port]
connection_host = dns
sourcetype=<string>
[tcp://[host:]port] 
source=<string>
[udp://514]
sourcetype=syslog
[tcp://10.1.2.3:9001] 
source = dns_10-1-2-3
Examples:

Network Input: Host Field
Y The connection_host attribute controls how the
host field for that input is set:
– dns (UI default)
ê  ThehostissettoaDNSnameusingreverseIPlookup
– ip
ê  Thehostissettotheoriginating
host'sIPaddress
– none (Custom)
ê  Explicitlysetthehostvalue
[tcp://9002] 
sourcetype=auth-data
connection_host=dns
[tcp://9003] 
sourcetype=ops-data
connection_host=ip
[tcp://9001] 
sourcetype=audit-data
connection_host=none
host=dnsserver


Network Input: acceptFrom
Y acceptFrom = <network_acl>
– List address rules separated by commas or spaces
ê  Asingle IPv4 or IPv6 address
ê  ACIDR block of addresses
ê  ADNS name
ê  Awildcard ‘*’and ‘!’
A bunch of network devices are sending syslog reports (UDP 514) to my
Splunk network input but I want to accept UDP inputs more selectively
[udp://514] 
sourcetype=syslog
connection_host=ip
acceptFrom=!10.1/16, 10/8


Network Input
Persistent
Queue
Network Input: Queues
Y Queues provide flow control
– Applies toTCP, UDP, Scripted Input
– Works as the entire input chain ( gets blocked, fills up before )
– Burst in data over network, slow resources, or slow forwarding
Y When memory-queue is full, write to file and keep using until it is empty
Y Persistent queue is preserved across restarts
– Not a solution for input failure
Network Traffic
Memory
Queue
Forward
Output
Queue
Forward
1
2
3
1 2 3

Network Input: Memory Queues
Y The queueSize attribute sets a queue
size for input data in KB, MB, or GB
Y This is a memory-resident queue that
can buffer data before forwarding
Y Defaults to 500KB
Y Useful if the indexer cannot always receive the data as fast as the
forwarder is acquiring it
Y Independent of the forwarder's maxQueueSize attribute
[tcp://9001] 
queueSize=10MB

Network Input: Persistent Queues
Y Provides file-system buffering of data
Y Adds additional buffer space after
memory buffer
– You must set a queueSize first
Y Apersistent queue is written to disk on the forwarder in
SPLUNK_HOME/var/run/splunk/...
Y Useful for high-volume data that must be preserved in situations where it
cannot be forwarded, such as if the network is unavailable, etc.
[tcp://9001] 
queueSize=10MB
persistentQueueSize=5GB

Special Handling and Best Practices
Y UDP – Splunk merges the UDP data until it finds a timestamp by default
– Can override during the parsing phase
Best practices:
Y Syslog – Send data to a syslog collector that writes into a directory structure
– For example, /sourcetype/host/syslog.txt
ê  Monitor the sourcetype directory and use host_segment
– You might need to break out the source types with custom transforms
docs.splunk.com/Documentation/Splunk/latest/Data/HowSplunkEnterprisehandlessyslogdata
Y SNMP traps – Write the traps to a file and use the monitor input
docs.splunk.com/Documentation/Splunk/latest/Data/SendSNMPeventstoSplunk

Scripted Inputs
Y Splunk can execute scheduled scripts and index the generated output
Y Commonly used to collect diagnostic data from OS commands
– For example: top, netstat, vmstat, ps, etc.
– Many Splunk apps use scripted inputs to gather specialized information from the OS or
other applications running on the server
Y Also good for gathering any transient data that cannot be collected with Monitor or
Network inputs
– APIs, message queues, Web services, or any other custom transactions
Y Splunk can run:
– Shell scripts (.sh) on *nix
– Batch (.bat) and PowerShell (.ps1) on Windows
– Python (.py) on any platform

Scripted Input Stanza
[script://<cmd>]
passAuth = <username>
host = <as indicated>
source = <defaults to script name>
sourcetype = <defaults to script name>
interval = <number in seconds or cron syntax>
Splunk only executes scripts from
SPLUNK_HOME/etc/apps/<app_name>/bin,
SPLUNK_HOME/bin/scripts, OR
SPLUNK_HOME/etc/system/bin
Use passAuth to run the script
as the specified OS user –
Splunk passes the auth token
via stdin to the script
Interval is the time period
between script executions –
defaults to 60 seconds

Defining a Scripted Input
1.  Write or obtain the script
2.  Always test your script from the context of an app and make sure it runs correctly
–  On the test/dev server, copy the script to an app’sbin directory
–  To test the script from the Splunk perspective, run splunk cmd scriptname
ê  ./splunk cmd ../etc/apps/<app>/bin/<myscript.sh>
3.  To deploy a remote scripted input via Splunk Web
–  Copy the verified script to the appropriate directory first
–  Configure the scripted input using the Splunk Web UI
4.  Verify the output of the script is being indexed

Scripted Inputs Examples
[script://./bin/myvmstat.sh]
disabled = false
interval = 60.0
source = vmstat
sourcetype = myvmstat
vmstat

Editing Scripted Inputs

Scripted Input Buffering
Y One possible downside to scripted input is potential loss of data
– Example: the forwarder that is running the script is not able to connect to the
indexer due to networking problems
Y You can declare the same queueSize and persistentQueueSize attributes
for a script stanza as for network (TCP and UDP) inputs
Y Buffers data on the forwarder when the network or indexer is not available

Alternate to Scripted Input
Y Set up your script to run as a CRON job and append data to a log file
Y Set up a monitor input to ingest the log file
– Take advantage of the file system and Splunk's robust file monitoring capabilities
– Can easily recover even when forwarder goes down
Y Modular input
– Simple UI for configuring a scripted input
– Appears as its own type of input
– docs.splunk.com/Documentation/Splunk/latest/AdvancedDev/ModInputsScripts
– For more information, take Splunk’s Creating Modular Inputs (eLearning) course

Lab Exercise 11 – Network Input
Y Tasks:
– Create and test a simpleTCP-based network input
ê  On the deployment server (your indexer), add a test network input
ê  Modify the host value for the test network input
ê  Deploy the app to your forwarder
Y Lab notes:
– Your instructor will run a script to sendTCPdata to ports on the forwarder
– Use your assigned port to listen for theTCPdata
Y Optional Task:
– Deploy a remote scripted input

Module 12:
Windows & Agentless Inputs

Module Objectives
Y Identify Windows input types and uses
Y Understand additional options to get data into Splunk
– HTTPEvent Collector
– SplunkApp for Stream

Windows-Specific Inputs
Y Windows OS maintains much of its state data (logs, etc.) in binary format
– Windows providesAPIs that enable programmatic access to this information
Y Splunk provides special input types
to access this data
– All other input types are also supported
– Data can be forwarded to any Splunk
indexer on any OS platform
– Windows Universal Forwarder can run
as domain user without the local
administrator privilege

Windows-Specific Input Types
* Supports both local and remote (WMI) data collection
Input Type Description
Event Log* Consumes data from the Windows OS logs
Performance* Consumes performance monitor data
Active Directory Monitors changes in an Active Directory server
Registry Monitors changes in a Windows registry
Host Collects data about a Windows server
Network Monitors network activity on a Windows server
Print Monitors print server activity

Local Windows Inputs Syntax
Y Configure inputs during the Windows Forwarder installation
Y Or, configure them manually:
– See inputs.conf.spec and inputs.conf.example
for details on setting up each Windows input type
[admon://name]
[perfmon://name]
[WinEventLog://name]
[WinHostMon://name]
[WinNetMon://name]
[WinPrintMon://name]
[WinRegMon://name]
Note
While you can configure Windows
inputs manually, Splunk
recommends that you prepare the
stanza using Splunk Web UI
because it is easy to mistype the
values for event log channels.

Windows Inputs: Using the Manager UI
[WinEventLog://Security]
checkpointInterval = 5
current_only = 0
disabled = 0
start_from = oldest

Windows Input Configuration Options
Y You can filter out non-essential events on the Windows Universal Forwarder
– Set whitelist and blacklist based on event field names and regex
– Allows you to target specific events while filtering out lower value events
ê  whitelist = <list> | key=regex [key=regex]
ê  blacklist = <list> | key=regex [key=regex]
– Can configure up to 10 whitelist and 10 blacklist per stanza
ê  Blacklist overrides whitelist if conflicts occur
[WinEventLog://Security]
disabled=0
whitelist1= EventCode=/^[4|5].*$/ Type=/Error|Warning/
whitelist2= TaskCategory=%^Log.*$%
blacklist = 540

Local vs. Remote Windows Inputs
Y You can configure remote inputs for two types of Windows inputs:
– Event logs
– Performance monitor
Y Advantage:
– You can collect the information from Windows servers without installing Splunk
forwarder
Y Disadvantage:
– Uses WMI as a transport protocol
– Does not scale well beyond a few source machines
– Requires a domain account

WMI Inputs
Y Remote inputs are configured in wmi.conf
Y See wmi.conf.spec and wmi.conf.example for full details
[WMI:remote-logs]
interval = 5
server = server1, server2, server3
event_log_file = Application, Security, System

[WMI:remote-perfmon]
interval = 5
server = server1,server2, server3
wql = Select DatagramsPersec

Special Field Extractions
Y Several Microsoft products use a special multi-line header log format
– For example, IIS/W3C, JSON, and other delimited/structured sources
Y Challenges:
–  These logs often get re-configured by the product administrator
ê  Requires some sort of coordination between the source administrator and the
Splunk administrator to synch up field extraction
Y Solution:
– Use indexed field extraction on the Windows forwarder
ê  Normally the field extraction magic happens on the index/search tier
– You will learn more about this in the Data Preview and Field Extraction modules

Powershell Input
Y Uses built-in powershell.exe scripting facility in Windows
–  Nocustom externallibrary dependencies
[powershell://<name>]
script = <command>
schedule = [<number>|<cron>]
RunningProcesses

Get-Process | Select-Object...

*/10 * * * *
PowerShell v1 or v3
Command or a script file
blank executes once

Windows Inputs Resources
Y About Windows data
docs.splunk.com/Documentation/Splunk/latest/Data/AboutWindowsdataandSplunk
Y Monitoring Event Logs
docs.splunk.com/Documentation/Splunk/latest/Data/MonitorWindowsdata
Y General information about event log
windows.microsoft.com/en-us/windows/what-information-event-logs-event-viewer
Y Performance Monitor
docs.splunk.com/Documentation/Splunk/latest/Data/Real-timeWindowsperformancemonitoring
Y Performance Counters Reference
msdn.microsoft.com/en-us/library/aa373088

Splunk Agentless Inputs
Y HTTPEvent Collector (HEC)
– Atoken-based HTTPinput that is secure and scalable
– Sends events to Splunk without the use of forwarders
ê  Canfacilitateloggingfromdistributed,multi-modal,and/orlegacyenvironments
ê  Logdatafromawebbrowser,automationscripts,ormobileapps
http://dev.splunk.com/view/event-collector/SP-CAAAE6M
Y SplunkApp for Stream (Splunk-supported free app)
– An alternative way to collect “difficult” inputs
ê  NovisibilityintoDBserversbecauseDBAsrefusetoinstallanyagentsonSQLservers
ê  Weblogalonedoesnotprovideenoughvisibilityintonefariouswebtraffic
– Able to grab data off the wire
– Supports Windows, Mac, and Linux
http://docs.splunk.com/Documentation/StreamApp

Distributed HEC Deployment Options
HEC can scale by taking advantage of Splunk distributed deployment
1 2 3 4

Configuring HTTP Event Collector
Y  Enable the HTTPevent collector (disabled by default)
– Navigate to Settings > Data inputs > HTTPInputs
– Click Global Settings > Enabled
Y  Generate a HTTP-input token by clicking New Token
– The Add Data workflow starts
– Name the input token and optionally set the default sourcetype and index
1 2
1
2

Sending HTTP Events from a Device
Y Create a request with its authentication header to include the input token
– While you can send data from any client, you can simplify the process by using
the Splunk logging libraries
ê  Supports JavaScript, Java and .NETlogging libraries
Y POST data in JSON format to the token receiver
curl "http[s]://<splunk_server>:8088/services/collector"
-H "Authorization: Splunk <generated_token>"
-d '{
"host":"xyz",
"sourcetype":"fl01_S2",
"source":"sensor125",
"event": {"message":"ERR", "code":"401"}
}'

HTTP Event Collector Options
Y Enable HEC acknowledgments
Y Send raw payloads
Y Specify metadata at the request level
Y Configure dedicated HTTP settings

Enabling HEC Acknowledgments
Y Example:Application developers want guarantees on the HEC indexing
Y Solution: Enable indexer acknowledgement
Y Configuration Notes:
– ACK is configured at the token level
– Each client request must provide a channel
ê  Achannel is a unique identifier created by the client
– When an event is indexed, the channel gets the ACK ID
– Client polls a separate endpoint using one or moreACK IDs
– Once an ACK has been received, it is released from memory

Enabling HEC Acknowledgments (cont.)
1.  Client POSTprovides a channel
2.  Splunk HEC responds to the channel withACK ID x
3.  Client requestsACK status check forACK ID x (1 in this example)
curl "http[s]://<splunk_server>:8088/services/collector/event?
channel=<client_provided_channel>"
-d '{ "event":"event 1"}'
curl "http[s]://<splunk_server>:8088/services/collector/ack?
-d '{ "acks":[1]}'

Sending Raw Payloads to HEC
Y Example:Application developers want to send data in a proprietary format
Y Solution: HEC allows any arbitrary payloads, not just JSON
– No special configuration required
– Must use channels similar toACK
ê  SupportsACKaswell
– Events MUSTbe bounded within a request
curl "http[s]://<splunk_server>:8088/services/collector/raw?
-d 'ERR,401,-23,15,36'

Request Level Metadata
Y Example:Application developers want to minimize network traffic bloat
Y Solution: Set default metadata in the query and use batch submit
– Apply default metadata to all events within the request
ê  index, timestamp, host, source, sourcetype
– Override any metadata per event (JSON event only)
curl "http[s]://<splunk_server>:8088/services/collector?
source=abc&host=xyz&time=1450302681"
-d '{"event":"401"}{"event:"400"}{"event":"500", "host":"a100"} '

Configuring Dedicated HTTP Settings
Y Example: Splunk admins want to limit who can access the HEC endpoints
Y Solution: Manually add the dedicated server settings in inputs.conf
– Available attributes under the [http] stanza
ê  ConfigureaspecificSSLcertforHECandclientcerts
ê  Enablecross-originresourcesharing(CORS)forHEC
ê  Restrictbasedonnetwork,hostnames,etc.
[http]
enableSSL = 1
crossOriginSharingPolicy = *.splunk.com
acceptFrom = "!45.42.151/24, !57.73.224/19, *"
inputs.conf

Monitoring with DMC: HTTP Event Collector

Lab Exercise 12 – HTTP Event Collector
Y Tasks:
– Enable HTTPevent collector on the indexer
– Create a HTTPevent collector token
– Send HTTPevents from your forwarder #1 (10.0.0.50)
ê  Challenge Exercise: send the events from your computer

Module 13:
Fine-tuning Inputs

Module Objectives
Y Understand the default processing that occurs during input
phase
Y Configure input phase options, such as sourcetype fine-tuning
and character set encoding

Testing New Inputs
Y Every Splunk deployment should have a test environment
– It can be a laptop, virtual machine, or spare server
– It should have the same version of Splunk running in production
Y Test your input for a new source of data and evaluate it in a test environment
Y If not:
– Create a test index and send test inputs to this index
ê  You can delete it when needed
ê  Does not require splunkd restart
– Use Data Preview to evaluate new data sources without actually inputting

Things to Get Right at Index Time
Input phase Y  Host
Y  Sourcetype
Y  Source
Y  Index
Parsing phase Y  Date/timestamp
Y  Line breaking (event boundary)
Y  Any other event level processing

What if I Don't Get It Right?
Y On a testing and development system:
– It’s okay; this is what test/dev Splunk setups are for!
– At any time, clean the indexes out, change your configurations, and try again
– Migrate the configurations to your production Splunk implementation when it is
working correctly
Y On a production server, your choices are:
– Leave the erroneous data in the system until it naturally ages out
ê  Hit the size or time limits
– Attempt to delete the erroneous data
– Only re-index when it is absolutely necessary

What is props.conf?
Y props.conf is a config file that is referenced through all phases of Splunk
data processing and searching, including the inputs phase
Y See props.conf.spec and props.conf.example files in SPLUNK_HOME/
etc/system/README for specifics
docs.splunk.com/Documentation/Splunk/latest/admin/Propsconf

props.conf Stanza
Y All data modifications in props.conf are based on either source,
sourcetype, or host
Y You can use wildcards (*) in source:: and host:: stanzas
[host::host_name]
attribute = value
[source::source_name]
attribute = value
[sourcetype]
attribute = value
[source::/var/log/secure*]
[host::nyc*]
TZ = US/Eastern
[sales_entries]
CHARSET = UTF-8
syntax example

props.conf in the Inputs Phase
Y Some settings in props.conf are applied during the inputs phase:
– Character encoding
– Fine-tuning sourcetypes
– Afew others
Y Some settings in props.conf are applied during the parsing phase:
– Individual event breaking
– Time extraction settings and rules
– Event data transformation
Y Configure props.conf on your forwarders if you have input phase settings
wiki.splunk.com/Where_do_I_configure_my_Splunk_settings

Character Encoding
Y During the input phase, Splunk sets
all input data to UTF-8 encoding by
default
Y This can be overridden, if needed,
by setting the CHARSET attribute
Y Use AUTO to attempt automatic encoding based on language
docs.splunk.com/Documentation/Splunk/latest/Data/Configurecharactersetencoding
[source::/var/log/locale/korea/*]
CHARSET=EUC-KR
[sendmail]
CHARSET=AUTO


Fine-tuning Directory Monitor Sourcetypes
Y When you add a directory monitor and specify a
sourcetype explicitly, it applies to all files in the
directory
Y You can omit the sourcetype attribute
–  Splunkwilltry to use automatic pre-trainedrules
Y You can then selectively override the sourcetype
with props.conf
– Identify the input with a [source::<source>]
stanza and set the sourcetype attribute
– This is an input phase process
[monitor:///var/log/]

inputs.conf
[source::/var/log/mail.log]
sourcetype=sendmail
[source::/var/log/secure/]
sourcetype=secure
...
props.conf

Lab Exercise 13 – Source Type Fine-tuning
Y Tasks:
– Add a test directory monitor to sample the auto-sourcetype behavior
ê  Make note of the sourcetype value
– Override the auto-sourcetyping of a specific source by adding a sourcetype
declaration in props.conf
– Deploy it to your forwarder and check again

Module 14:
Parsing Phase and Data
Preview

Module Objectives
Y Understand the default processing that occurs during parsing
Y Optimize and configure event line breaking
Y Explain how timestamps and time zones are assigned to
events
Y Use Data Preview to validate event creation during the parsing
phase

The Parsing Phase
Y As data arrives at the indexer, it goes through the parsing phase
– The input is broken up into discrete events, each with a timestamp and a time zone
Y The parsing phase is all about creating, modifying, and redirecting events
– Apply additional transformation steps to modify the metadata fields or redirect data
– Both indexers and heavy forwarders parse events
ê  Inthismodule,youwillassumeparsingishappeningonanindexer
Indexer
Source Server
Indexing
Forward Parsing
License
Meter Disk
Universal
Forwarder
Inputs

Event Creation
During the parsing phase, data from input phase is broken up into individual
events, and then event-level processing is performed
"Streams" of data
from inputs phase
Parsed into
individual events
Event-by-event
processing

Event Boundaries
Y Splunk parsing phase determines where one event ends and the next one begins
– Automatically handles line breaking for common source types – even multi-line events
Y This line breaking process involves a series of pipelines
– Each pipeline consists of a set of queues and processors
Y Use Data Preview when on-boarding a new source type

Handling Single Line Events
Y Splunk handles single line event sourcetypes with automatic line breaking
Y It is more efficient to explicitly set:
– SHOULD_LINEMERGE = false
– Default is true and assumes events can span over more than one line
[my_custom_one_event_per_line_sourcetype]
SHOULD_LINEMERGE = false
SPLUNK_HOME/etc/apps/mycustom_addon/local/props.conf

Handling Multi-line Events
Y For multi-line events, Splunk tries to identify event boundaries
– Looks for a new line with a date at the start
BREAK_ONLY_BEFORE_DATE = true (default)
– Allows a maximum of 256 lines per event
MAX_EVENTS = 256 (default)
– Or many other options – for example,
BREAK_ONLY_BEFORE = <REGEX pattern>
docs.splunk.com/Documentation/Splunk/latest/Data/Indexmulti-lineevents

Date/timestamp Extraction
Y Correct date/timestamp extraction is essential
Y Always verify timestamps when setting up new data types
– Pay close attention to timestamps during testing/staging of new data
– Check UNIX time or other non-human readable timestamps
Y Splunk works well with standard date/time format and well-known data
types
Y Custom timestamp extraction is specified in props.conf

Splunk Event Timestamp Processing
Found
a timestamp
in the event?
Found date in
the file name?
Extract using defaults
Yes
Yes
No
Yes
Yes
No
No No
No
Yes
Note
Splunk can only extract dates from
a source. To extract a time from a
source, use a transform.
Use the current system time:
Y  Indexer's OS time
Y  The time the data was
collected on the forwarder
Extract using the rules Use the file modification time
Get date from the file name
Use the most recent one
Any timestamps
from the
same source?
Any explicit rules
in props.conf?
Event contains
only time,
but no date?
http://docs.splunk.com/Documentation/Splunk/latest/Data/HowSplunkextractstimestamps

TIME_PREFIX
Y TIME_PREFIX = <REGEX>
matches characters right BEFORE the date/timestamp
– Use this syntax to specify where the timestamp is located in the event
ê  Example data with "date-like" code at the start of the line
1989/12/31 16:00:00 ed May 23 15:40:21 2015 ERROR UserManager - Exception thrown
[my_custom_source_or_sourcetype]
TIME_PREFIX = d{4}/d{2}/d{2} d{2}:d{2}:d{2} w+s

Start looking here for date/timestamp
props.conf

MAX_TIMESTAMP_LOOKAHEAD
Y MAX_TIMESTAMP_LOOKAHEAD = <integer>
specifies how many characters to look beyond the start of the line for a
timestamp
– Works in conjunction with TIME_PREFIX
ê  If set, it starts counting from the point theTIME_PREFIX indicates Splunk should start
looking for the date/timestamp
– Improves efficiency of timestamp extraction
– The complete timestamp string must be present within the specified range

TIME_FORMAT
Y TIME_FORMAT = <strptime-style format>
specifies the format of the timestamp using a strptime() expression
– For example: 2015-12-31 would be %Y-%m-%d
Y For more detail and other options, check:
– SPLUNK_HOMEetcsystemREADMEprops.conf.spec
–  docs.splunk.com/Documentation/Splunk/latest/Data/ConfigureTimestampRecognition
–  docs.splunk.com/Documentation/Splunk/latest/Data/Handleeventtimestamps

Setting Time Zones – Splunk’s Rules
Y Use time zone offsets to ensure correct event time
Y Splunk applies time zones in this order:
1.  Atime zone indicator in the raw event data
ê  -0800, GMT+8 or PST
2.  The value of aTZ attribute set in props.conf
ê  Checks the host, source, or sourcetype stanzas
ê  If a forwarder is used, the forwarder-provided time zone is used
ê  en.wikipedia.org/wiki/List_of_zoneinfo_timezones
3.  If all else fails, Splunk applies the time zone of the indexer's host server
[host::nyc*]
TZ = America/New York

[source::/mnt/cn_east/*]
TZ = Asia/Shanghai
props.conf

Using Splunk Data Preview
Y Event breaking and date/timestamp settings require testing
– Splunk Data Preview to the rescue!
– Perfect for sandbox environment (or test index) to get your settings right
Y Splunk attempts to auto-detect a sourcetype
– If it doesn’t, you can select from a list or define your own sourcetype
– Supports both unstructured and structured data sources
– CSV, JSON, W3C/IIS, XML, etc.

[167154] 2014-08-21 00:43:26
Received fatal signal 6 (Aborted).
Cause:
Signal sent by PID 4679 running under UID 1687.
Crashing thread: Main Thread
Register
RDI: [0x0000000060000000]
RSI: [0x0000009D00003002]
...

OS: Linux
Arch: x86-64

Backtrace:
[0x00400A6000205000] gsignal + 53 (/lib64/libc.so.6)
[0x0900000F00010000] abort + 373 (/lib64/libc.so.6)
[0x00100D0000000060] ? (/lib64/libc.so.6)
[0x0700500000710000] __assert_perror_fail + 0 (/lib64/libc.so.6)
[0x00B000009003000C] _ZN11XmlDocument8addChildERK7XmlNode + 61 (dcrusherd)
[0x0F00020040600000] _Z18getSearchConfigXMLR11XmlDocumentPKPKc + 544 (dcrusherd)
[0x0003001000500000] _Z22do_search_process_impliPKPKcP12BundlesSetupb + 6141 (dcrusherd)
Linux / usr13.eng.buttercupgames.com / 2.6.32-279.5.2.el6.x86_64 / #1 SMP Fri Aug 24 01:07:11 UTC 2013 / x86_64
/etc/redhat-release: CentOS release 6.3 (Final)
glibc version: 2.12
glibc release: stable
Last errno: 2
...
Previewing Unstructured Data
Splunk will do its best to identify what it
thinks are the event’s boundaries and its
timestamp; however, if you are familiar with
the data, provide more info
1
2
2016-01-21 00:43:25

Previewing Unstructured Data (cont.)
By specifying the timestamp location, Splunk
can update the number for events extracted
Y  Will indicate a warning if Splunk cannot find
a timestamp within the range

Another example – previewing json file as unstructured data

By specifying the timestamp prefix pattern,
Splunk can parse out the proper timestamp

Previewing Structured Data
Splunk automatically identifies structured data and parses the event boundaries and field names
Y  Produces an indexed extraction stanza
Y  If you see a timestamp warning, indicate where to find a timestamp

Previewing Structured Data (cont.)
By specifying a JSON field name, Splunk is able to identify the timestamps

Saving Preview Sourcetype
Or, check the resulting props.conf
stanza for your new sourcetype
You can copy and deploy them
manually to your forwarders
When saved, this becomes a
learned sourcetype that you
can re-use

Source Type Manager
Y Provide a way to view source types configured on the system
–  ClickSettings > Source types
Y Search and access the matching
source types independent of
Add Data workflow
Y Use Clone and edit

Lab Exercise 14 – Create a New Sourcetype
Y Tasks:
– Use preview to evaluate two custom file types:
ê  Anew log sample that contains multiple timestamps
ê  Anew log sample that contains multi-line events in XMLformat
– Apply a custom line breaking rule and custom timestamp rules and save as a
new sourcetype

Module 15:
Manipulating Raw Data

Module Objectives
Y Explain how data transformations are defined and invoked
Y Use transformations with props.conf and transforms.conf to:
– Mask or delete raw data as it is being indexed
– Override sourcetype or host based upon event values
– Route events to specific indexes based on event content
– Prevent unwanted events from being indexed

Modifying the Raw Data
Y Sometimes it’s necessary to modify the underlying log data before it is indexed
Y Examples:
– The case of privacy concerns:
ê  Patientinformationinahealthcareenvironment
ê  Creditcardoraccountnumbersinafinancialenvironment
ê  Databeingtransportedacrossinternationalboundaries
– Event routing according to business use cases (e.g. audit and security):
ê  Alleventsgotothewebindex,exceptcreditcardtransactionswhicharesenttothe credits
index
Y Care should be taken when modifying raw events (_raw)
– Unlike all other modifications discussed, these change the raw data before it is indexed
– Indexed data will not be identical to the original data source

Splunk Transformation Methods
Y When possible, set your values during the inputs phase
– Most efficient to use inputs.conf
Y Splunk provides two methods of raw data transformations:
– SEDCMD
ê  Uses only props.conf
ê  Only used to mask or delete raw data
– TRANSFORMS (REGEX)
ê  Uses props.conf and transforms.conf
ê  More flexible, but more complex to configure
ê  Transforms matching events based on source, sourcetype, or host

SEDCMD
Y Splunk leverages a UNIX "sed-like" syntax for simplified data modifications
– Provides “search and replace” using regular expressions and substitutions
– Works on all Splunk installs including Windows-based
Y Example: Hide the first 5 digits of an account number in the vendor_sales.log source
Y For more examples, see:
docs.splunk.com/Documentation/Splunk/latest/Data/Anonymizedatausingconfigurationfiles
[source::.../vendor_sales.log]
SEDCMD-1acct = s/AcctID=d{5}(d{5})/AcctID=xxxxx1/g
Indicates the capture group
vendor_sales.log
props.conf
[22/Oct/2014:00:46:27] VendorID=9112 Code=B AcctID=4902636948
[22/Oct/2014:00:48:40] VendorID=1004 Code=J AcctID=4236256056
[22/Oct/2014:00:50:02] VendorID=5034 Code=H AcctID=8462999288
Match this and replace with AcctID=xxxxx99288

TRANSFORMS
Y Per event transformation is based on REGEX pattern matches
Y Define the transformation in transforms.conf and invoke it from props.conf
Y Transformation is based on the following attributes:
– SOURCE_KEY indicates which data stream to use as
the source for pattern matching (default: _raw)
– REGEX identifies the events from the SOURCE_KEY
that will be processed (required)
ê  Optionallyspecifiesregexcapturegroups
– DEST_KEY tells where to write the processed
data (required)
– FORMAT controls how REGEX writes the
DEST_KEY (required)
[sourcetype]
TRANSFORMS = stanzaName
[stanzaName]
SOURCE_KEY = ...
REGEX = ...
DEST_KEY = ...
FORMAT = ...
props.conf
transforms.conf

Masking Sensitive Data
[cc_num_anon]
REGEX = (.*CC_Num:s)d{12}(d{4}.*)
DEST_KEY = _raw
FORMAT = $1xxxxxxxxxxxx$2
[source::...storepurchases.log]
TRANSFORMS-1ccnum = cc_num_anon
When SOURCE_KEY is omitted, _raw is used.
This REGEX pattern finds two capture groups
and rewrites the raw data feed with a new
format.
props.conf
transforms.conf
For the purchases.log source, send to the
cc_num_anon transformation processor.
-1ccnum is a label to identify this transform
namespace and is used to determine
sequence.
[22/Apr/2014:00:46:27] VendorID=9112 CC_Num: 4217656647324534 Code=B
[22/Apr/2014:00:48:40] Sent to checkout TransactionID=100763
[22/Apr/2014:00:50:02] VendorID=5034 CC_Num: 6218651647508091 Code=H
[22/Apr/2014:00:46:27] VendorID=9112 CC_Num: xxxxxxxxxxxx4534 Code=B
[22/Apr/2014:00:48:40] Sent to checkout TransactionID=100763
[22/Apr/2014:00:50:02] VendorID=5034 CC_Num: xxxxxxxxxxxx8091 Code=H

Setting Per-Event Sourcetype
Should be your last option because it is more efficient to set the sourcetype
during the inputs phase (using inputs.conf or props.conf)
[source::udp:514]
TRANSFORMS = custom_sourcetyper
[custom_sourcetyper]
SOURCE_KEY = _raw
REGEX = Custom$
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::custom_log
Check events in network input source.
If an event contains “Custom” at the end,
assign the new sourcetype value custom_log
When MetaData: key is used, its FORMAT
value must be prefixed by:
Y host::
Y source::
Y sourcetype::
props.conf
transforms.conf

Setting Per-Event Host Name
[sales_host]
SOURCE_KEY = _raw
REGEX = server:(w+)
DEST_KEY = MetaData:Host
FORMAT = host::$1
Check each event in the _raw source.
If an event contains “server:”, capture the word
and rewrite the value of the MetaData:Host key
with the captured group.
When MetaData: key is used, its FORMAT value
must be prefixed by:
Y host::
Y source::
Y sourcetype::
[22/Apr/2014:00:46:27] sales accepted server:A01R2 SID=107570
[22/Apr/2014:00:48:40] sales rejected server:B13R1 SID=102498
[22/Apr/2014:00:50:02] sales accepted server:A05R1 SID=173560
[sales_entries]
TRANSFORMS-register = sales_host
props.conf
transforms.conf

ff
Per-Event Index Routing
Again, if at all possible, specify the index for your inputs during the inputs
phase (inputs.conf)
[route_errs_warns]
REGEX = (Error|Warning)
DEST_KEY = _MetaData:Index
FORMAT = itops
[mysrctype]
TRANSFORMS-itops = route_errs_warns
If Error or Warning is found in the
incoming _raw, change its index field
value to itops
props.conf
transforms.conf

Filtering Unwanted Events
Y You can route specific unwanted events to the null queue
– Events discarded at this point do NOTcount against your daily license quota
[WinEventLog:System]
TRANSFORMS = null_queue_filter
[null_queue_filter]
REGEX = (?m)^EventCode=(592|593)
DEST_KEY = queue
FORMAT = nullQueue
The (?m)^ in the REGEX indicates a
multiline event that starts with EventCode.
Route to queue and use nullQueue format
to discard events.
props.conf
transforms.conf

Indexing Phase Details
After the parsing phase, Splunk passes the fully processed data to the index
processor
End of the parsing
Y  Null-routed?
Y  Remote server?
Y  Disk?
license meter
_raw is metered for
license usage
index created
Keyword indexed, _raw
is compressed and both
are written to disk
Data Integrity
Control

Persisted to Disk
Y All modifications and extractions are written to disk along with _raw and
metadata
– source, sourcetype, host, timestamp, punct, etc.
Y Indexed data cannot be changed
– Changes to props.conf or transforms.conf only apply to new data
– Indexed data will not be changed without re-indexing
Y Tip:
– When adding or changing stanzas in props.conf, you can call the following
URLend-point to re-load the modified props.conf and transforms.conf
without restarting your indexer:
http://servername:splunkwebport/debug/refresh

Lab Exercise 15 – Manipulating Data
Y Tasks:
– Use transforms.conf to:
ê Mask sensitive data
ê  Redirect events to specific indexes
ê  Drop unwanted events
– Use props.conf to sequence the filtering and redirecting events

Module 16:
Supporting KOs

Module Objectives
Y Create field extractions
Y Configure collections for KV Store
Y Manage Knowledge Object permissions
Y Control automatic field extraction

Search time
transformations
Search Phase: The Big Picture
Normal searches
Real-time searches

Search Time for Admins
Y Splunk extends the ability to create most search time transformations to
user roles, not just admins
– For example, field extractions can be fully administered through Splunk Web’s
Settings view
Y Admins may be asked to:
– Install apps and add-ons
ê  Remember, apps/add-ons are NOTjust views and dashboards
ê  Contain bundles of search time lookups, field extractions, tags, etc.
– Change/disable search time transformations
– Create custom field extractions

Default Search Time Field Extractions
Y For common source types, Splunk has default search time field extractions
Y Most fields are discovered by Splunk from your search results
– Automatically detects key/value pairs (e.g. a=1, or b:2)
Y Additional default extractions are easy to add with add-ons and apps
– The *nix app has many search time fields for standard UNIX logs
ê  For example, secure.log, messages.log, etc.
– The Windows app has similar defaults for Windows data
– For other data, look for an app specifically designed for that type of data on
splunkbase.splunk.com

Custom Search Time Field Extractions
Y Use the rex command, or similar commands, in the search language
– All roles can use this command
– Requires knowledge of regular expressions (REGEX)
Y Use the Field Extractor in Splunk Web
– Handles REGEX-based and delimiter-based extractions
– Knowledge of regular expressions helpful, but not required
Y Edit configuration files
– Available only to admins and provides additional advanced extraction options
– Knowledge of REGEX required

Field Extractions in props.conf
Y Field extraction happens during index-time (indexed fields) and/or search-time
(extracted fields)
Y Whenever possible, use search-time field extractions for better flexibility and performance
Y The search-time extractions can be an inline or a transform
Y Use extraction directives, EXTRACT and REPORT, in props.conf
–  EXTRACT(inlineextraction)isdefinedinprops.confasstandalone
–  REPORT(fieldtransform)isdefinedintransforms.confandinvokedfromprops.conf
Inline extraction
saved as EXTRACT
Saved as REPORT

REPORT Extractions in props.conf
Y REPORT references a transform defined separately in transforms.conf
Y In transforms.conf, you can
– define field extractions using delimiters
– apply other advanced extraction techniques
Y For full details on REPORT, see
docs.splunk.com/Documentation/Splunk/latest/Knowledge/
Createandmaintainsearch-timefieldextractionsthroughconfigurationfiles

props.conf
transforms.conf
Examples
[tradelog]
EXTRACT-1type = type:s(?<acct_type>personal|business)
[sysmonitor]
REPORT-sysmon = sysmon-headers
KV_MODE = none
[sysmon-headers]
DELIMS = ","
FIELDS = Time,EventCode,EventType,Type,ComputerName,LogName,RecordNumber
Captured field name
Y  Applies to this
sourcetype
Y  The REGEX
pattern defines
extracted field
Arbitrary namespace you assign to this extraction
Useful for ordering multiple transactions
Process this stanza in transforms.conf

Managing Extraction Permissions
Y Select Settings> Fields > Field Extractions
Y Extractions are listed by owner and app
Y Use the Permissions link to change permissions by role
– Read access applies the extraction to searches executed by users with that role
– Write access allows users to change the extraction
– Permissions can be private, for one app, or all apps
Y Knowledge object permissions are stored in metadata/local.meta
– Each user/app has its own .meta file
– When you share globally, it is set to
export = system
[props/mysourcetype/EXTRACT-myfield]
export = system
owner = admin
version = 6.4.0
modtime = 1415396457.359661000

Field Extractions at Index Time
Y For known structured data, you can extract the fields during index-time
Y Recommendations:
–  ONLYrecommendedinspecificcircumstances;wheneverpossible,extractfieldsatsearchtime
ê  Forfrequentlyre-configureddelimitedsources,useindexedextractions(example:IIS)
ê  ForstaticCSV,useREPORTandDELIMS,orothersearch-timeextractions
ê  Ifsearch-timeextractionsarenoticeablyimpactingsearchperformance(thisisrare)
–  Useadedicatedindex
PROs CONs
Y  Provision the extraction during
the input phase
Y  Can configure on the universal forwarder
Y  Auto formatting
Y  Can drop useless headers and comments
Y  Increased storage size
Y  Static field names -- additional step required
for late-binding use cases
Y  Possible performance implications

Configuring Indexed Field Extractions
Y Again,indexedextractions are INPUTphaseprops.conf settings
–  Thesesettings belong on the forwarder
–  Checkprops.conf.specformoreoptions
#Software: Microsoft Internet Information Services 7.5
#Version: 1.0
#Date: 2015-06-08 00:00:00
#Fields: date time cs-method cs-uri-stem cs-uri-query c-ip cookie referer cs-host sc-status sc-bytes time-taken
2015-01-08 00:00:00 POST AutoComplete.asmx/GetCompletionList - 10.175.16.79
cApproved=1;+fParticipant=0000000695607440|urn:System-Services:GatewayTokenService_names:tc:SAML:2.0:nameid-
format:persistent|http://www.acme.com/2015/06/attributes/credentialidentifier; &nestedState=;
+WT_id=bd74-10f8-4dfe-bf45-fc2df5;+style=normal https://search.acme.com/Account/Account.aspx?redirect=https://
direct.acme.com/Home.aspx search.acme.com 200 1113 0
...
[my_structured_data]
INDEXED_EXTRACTIONS = w3c
HEADER_FIELD_LINE_NUMBER = 4
TIMESTAMP_FIELDS = date, time

Lookups
Y Alookup is a Splunk data enrichment knowledge object
–  UsedONLYduring search time
–  Thelookupstanzas are defined intransforms.conf
andprops.conf
Y Four types:
–  File-basedusesa csvfilestoredinthe lookupsdirectory
–  KVStorerequires collections.confthatdefinesfields
–  Externalusesapythonscriptoranexecutableinthe bindirectory
–  Geospatial usesa kmzsavedinthe lookupsdirectorytosupportthechoropleth
visualization

KV Store Lookups
Y CSV lookups are for data sets that are small and/or change infrequently
Y KV Store is designed for large key-value data collections that frequently change
– Tracking workflow state changes (an incident-review system)
– Keeping a list of environment assets assigned to users and their metadata
– Controlling a job queue or application state as the user interacts with the app
Y KV Store can:
– Enable per-record CRUD operations using the lookup commands and the RESTAPI
– Access key-value data seamlessly across search head cluster
– Back up and restore KV Store data
– Optional
ê  Allowdatatypeenforcementonwriteoperations
ê  Performfieldaccelerationsandautomaticlookups
ê  Workwithdistributedsearchesonthesearchpeers(indexers)
Note
CRUD:
Create, read, update, or delete

Enabling KV Store Collections
Y Before users can use a KV Store lookup, an admin must create a collection
– KV Store collection is the container of key-value definitions
Y Acollection is defined in collections.conf
– Specify the name of the collection and theschema
ê  Requiresatleasttwofields:matchinglookupfieldandanoutputfield
– Enforcing data types is optional
ê  If enforced, any input that does not match the type is silently dropped
[collection_name]
enforceTypes = [true|false]
field.<name1> = [number|string|bool|time]
field.<name2> = [number|string|bool|time]
accelerated_fields.<xl-name> = <json>
[mykv]
enforceTypes = true
field.x = number
field.y = string
accelerated_fields.xl2 = {"x": 1, "y": 1}

CSV to KV Store Migration
Before you start, decide if KV Store is the right choice
1. Runasearchand get the list of fields includedinthecsvlookup
| inputlookup my_csv_lookup
2. CreateaKV Store collection stanza incollections.conf
– Identifyandincludetheoptionstoenable
3. Update/addlookup definition forKV Store
– ClickSettings>Lookups>Lookupdefinitions
ê  Theresultingconfigurationissavedintransforms.conf
4. WriteexistingCSVdatatotheKVStore
| inputlookup filename.csv | outputlookup lookup_name
Y Ifyouneedtodeleteandresetthecollection,runthisCLI:
splunk clean kvstore –collection <collection>

Monitoring KV Store Activity
Y You can monitor KV Store activity on DMC
– Add the KV Store server role to the instance
– Click Search > KV Store: Instance
Y In addition, you can enable profiling to debug slow
KV Store operations
– Set the profiling threshold in the collection stanza
profilingEnabled = true
profilingThresholdMs = 500
– Logs slow-running operations
Y NOTE:
– For developing and troubleshooting only
– Disable profiling in a production environment

Further Reading: KV Store
Y About KV Store
– docs.splunk.com/Documentation/Splunk/latest/Admin/AboutKVstore
Y Tutorial: Use KV Store with a simple app
– dev.splunk.com/view/SP-CAAAEZT
Y KV Store backup and restore
– docs.splunk.com/Documentation/Splunk/latest/Admin/BackupKVstore

Other Search Time Knowledge Objects
Y Knowledgeobjectsarestoredinconfigurationfiles:
–  macros.conf,tags.conf,eventtypes.conf,savedsearches.conf,etc.
–  Seespecific.specfilesinSPLUNK_HOME/etc/system/READMEandthedocsfordetails
Y Whenuserscreateormodifyknowledgeobjects,SplunkWebautomaticallyupdatesthe.conffiles
Y UseSplunkWebUIasmuchaspossible
–  Adminscanusebtoolandeditthe.conffilesdirectly
Y SomesystemsettingscanbecheckedandchangedwithAdvancededit

Mitigating Possible Data Access Risks
Y SPLsafeguards
– Warn users when they click a link that attempts to execute risky SPLcommands:
ê  collect, crawl, delete, dump, input, outputcsv, outputlookup,
runshellscript, script, sendalert, sendemail, stash, tscollect
Y Displays a warning dialog:
– Investigate – open but do not
execute
– Run – execute the search
– Cancel – clear the search
Y How to audit
index=_internal sourcetype=splunkd component=HandleJobsDataProvider

Mitigating Possible Data Access Risks (cont.)
Y Example:Admins want to limit a role ability to export results via the UI
– Only disables the UI access
– Still can export via the search command
– Remove export_results_is_visible from the selected role capabilities
– Disables the UI element in Search, Report, Dashboard, and Pivot

Lab Exercise 16 – Migrate CSV Lookup to KV Store
Y Tasks:
– Identify what fields are in the CSV lookup
– Define a KV Store collection
– Migrate the CSV lookup data to KV Store
– Verify the migration with Splunk search and DMC

Module 17:
Distributed Search

Module Objectives
Y Describe how distributed search works
Y Explain the roles of the search head and search peers
Y Configure a distributed search group
Y List search head scaling options

Distributed Search
Y Productionservers with universalforwarderssenddata
toindexersusingloadbalancing
Y Indexers(peers)storetheirportionofthedata
Y Userslogontothe search head and runreports
–  Thesearchhead dispatches searchestothepeers
–  Peersrunsearches in paralleland returntheirportion
ofresults
–  Thesearchhead consolidates the individualresults
andpreparesreports
Search
Peers
(Indexers)
Auto-balancing Forwarder
Search Head

Setting Up Distributed Search
Y Install Splunk on each search head and peers using an enterprise license
Y Set up the same indexes on all peers (indexers)
–  Fornon-clustered peers, use the deploymentservertomanageconfigurations
–  Forclustered peers, use the cluster mastertomanagepeers’configurations
Y Configure forwarders to load-balance outputs across all peers (indexers)
Y Add a user to each peer with a role that has the edit_user capability
–  Usedonlyfor authenticating a searchheadtothepeers
Y On the search head, configure search peers
by selecting Settings > Distributed search
– Distributed search is turned on by default, so
just add search peers

Search Peers
Y Select Settings > Distributed search
> Search peers >Add new
Y Enter the servername:port for a
search peer
Y Enter a username and password of
an account on the search peer
–  Theaccountmust have theedit_user
capability
–  Youshouldcreate an account on each
peerforthispurpose

Distributed Search Best Practice
Y Forward all search head indexes to the search peer (indexer) layer
– Simplifies the process of managing indexes
– Can diagnose from other search heads if one goes down
– Allows other search heads to access all summary indexes
ê  By default, summary indexes are on the instance that generates them
[indexAndForward]
index = false

[tcpout]
defaultGroup = default-autolb-group
forwardedindex.filter.disable = true
indexAndForward = false

[tcpout:default-autolb-group]
server=idx1:9997,idx2:9997
outputs.conf
2
1

Peer Failure
Y When an indexer goes down, the forwarder
automatically uses only the available indexers
–  Theofflineindexerdoesnotparticipateinsearches
–  Theremainingindexershandleallindexingandsearches
Y If a peer goes down during a job, a notification is sent to
the user that the job is potentially incomplete
Y If a peer is already down, a message indicates which
peer is down
Search
Peers
(Indexers)
Auto-balancing Forwarder
Search Head

Use Cases for Multiple Search Heads
Y Access control
– Control who can access which indexes using what apps
– Dedicate a search head for each functional area – ITOps, Security, or BI
Y Manage geo-dispersed data
– Allow local offices to access their own data while maintaining centralized indexers
Y Performance enhancement
– Distribute indexing and search loads across multiple servers
ê  Facilitates horizontal scaling

How Many Search Heads?
Y One dedicated search head can handle around 8 to 12 simultaneous users
or scheduled searches/alerts
– Exact numbers depend on types of searches being executed and the hardware
of the server the search head is installed on—especially number of CPU cores
Y Search heads can be added to the distributed group at any time
– Install Splunk on the new server and configure as per previous procedure
Y Search heads can be dedicated or clustered
– Dedicated search heads don't share knowledge objects (separate small teams)
– Search head cluster shares a common set of knowledge objects (large teams)
ê  Discussed in detail with hands-on labs in Splunk ClusterAdministration class
ê  http://docs.splunk.com/Documentation/Splunk/latest/DistSearch/AboutSHC

Dedicated Search Heads
Y More than one search head can be
configured for the same set of search
peers
Y Each search head
–  containsitsown unique set of reports,
dashboards,etc.
–  isdedicatedto one teamof users whowantto
haveuniqueknowledge objects fortheirown
use
Y Good when you have teams of different
people who don't share knowledge objects Indexer1 Indexer2 Indexer3
Search Heads
Search Peers

Search Head Cluster
Y Accommodates large enterprise use cases
– Search head high-availability
– Unified user experience across SHs
– Search scaling foundation
ê  Configurationsharing
ê  Artifactreplication
ê  Jobdistribution
ê  Alertmanagement
ê  Loadbalancing
Y Can configure an external (non-Splunk) load
balancer to provide transparent access to the
cluster
Load Balancer
Search Head Cluster
Deployer

Lab Exercise 17 – Distributed Search
Task:
Add a search peer to your search head

Module 18:
Basic Performance Tuning

Module Objectives
Y List Splunk monitoring tools
Y Identify indexing performance with DMC dashboards
Y Investigate and improve search performance
Y Optimize Splunk resources with limits.conf, ui-prefs.conf, and user quota
Y Manage Splunk summarization options

Monitoring Splunk
Y Splunk provides Distributed Management Consoleto monitor the health, status,
and activities of various Splunk components
– System overview
ê  AdditionalTopologyviewanddrill-downsifrunningin Distributedmode
– Indexing
ê  Indexingperformance,volumeusage,HECinputs&SplunkTCPinputactivities,and
licenseusage
– Search
ê  Searchactivities,usagestats,KVstoreactivities,andScheduleractivity
– Resource usage
ê  Perinstanceandpermachine
– Forwarders
ê  Forwarderactivitiesperinstanceanddeployment

Indexer
Splunk Data Pipeline
Y By default, Splunk dedicates a pipelineSet to process an input
– PipelineSet is a series of pipelines
– Each pipeline consists of a set of queues and processors
– Aprocessor takes in data from a queue, performs a task, and pushes the result
to the next queue in the pipelineSet
PipelineSet
Forwarder
Pipeline 1 Pipeline n Output
Input Pipeline 1 Pipeline 2 Pipeline n
Input

Index Parallelization
Y If indexers have high number of under-utilized CPU cores, you can enable multiple
pipelineSets for faster data ingestion
– Requires additional IOPS and RAM to scale properly
– Each pipelineSet handles one source at a time and
maintains its own state
– Data is written to buckets in parallel per pipelineSet, thus
buckets can have overlapping time ranges
Y To enable, edit server.conf
Warning
Index parallelization increases the
CPU utilization and consumes
additional RAM, therefore less is
available for other tasks such as
searching.
[general]
parallelIngestionPipelines = 2
NOTE: For most installs, the default
setting of 1 is optimal.
Forwarder
Forwarder
Indexer
server.conf

Index Storage Optimization
Y Example:
– You are running out of indexer storage space
– Or, you want to lower the overall storage cost
Y Assumption:
– Majority of Splunk searches execute over a set range
ê  90% of the scheduled searches run over the last 24 hours
– You are willing to trade search performance on older data with space saving
ê  No delete command support
– Use storage optimization to remove tsidx files for the older buckets

Index Storage Optimization (cont.)
Y Index reduction is triggered based on the latest time in
the bucket controlled by settings in indexes.conf
Y Job inspector notifies the user about executing
a search over minified buckets
[mini]
...
homePath = $SPLUNK_DB/mini/db
coldPath = $SPLUNK_DB/mini/colddb
thawedPath = $SPLUNK_DB/mini/thaweddb
enableTsidxReduction = 1
timePeriodInSecBeforeTsidxReduction = 7776000
tsidxReductionCheckPeriodInSec = 900
indexes.conf
Warning
Do not Disable Reduction after
the policy has been in-use.
90

Search Performance
Y Search performance can be impacted by many factors
Y Searches can be categorized as:
– Dense: many results from a given time range—CPU bound
– Sparse: few results from a given time range—CPU bound
– Super-sparse: few results from many buckets—I/O bound
– Rare: extremely few results from a given time range—I/O bound
docs.splunk.com/Documentation/Splunk/latest/Capacity/HowsearchtypesaffectSplunkEnterpriseperformance

Improving Search Performance
Y Best practice:
– Make sure disk I/O is as good as you can get
– Increase CPU hardware if needed
– Ensure search peer hardware meets recommended levels:
docs.splunk.com/Documentation/Splunk/latest/Capacity/Referencehardware
Y Most search performance issues can be addressed by adding additional search
peers
– How deployments scale:
docs.splunk.com/Documentation/Splunk/latest/Deploy/Deploymentcharacteristics
Y Splunk search performs better with more specific search criteria
– Monitor jobs regularly to identify the expensive searches
– Search performance debugging:
wiki.splunk.com/Community:PerformanceTroubleshooting

Improving Search Performance (cont.)
Y For a large number of simultaneous searches, you can take several actions
to mitigate performance issues:
– Spread scheduled searches to run on non-overlapping schedules
ê  Specify the search scheduling window (user) and the priority factors (admin)
– Make sure only necessary real-time searches are executed
ê  Configure a role-based concurrent real-time search limit
ê  Or, switch to run indexed real-time search mode (set in limits.conf)
– Limit the time range of end-user searches
– Add more search heads for user-interactive searches
–  UseSearchHead Cluster

Managing Search Jobs
Y Each search running on a search head creates a job
– Search jobs run as child processes
Y Click Activity > Jobs to view, save, delete, or inspect search jobs
Click Inspect to open
the job inspector

Job Status
Y Jobs in the job list are in one of several states:
– Running: the job is still executing
– Done: the job has completed and the results will be available (TTL) for 10 minutes
– Paused: the job has not completed, but has been paused by a user; could be restarted
– Finalized: the job did not run to completion because a user stopped it
– Saved: a user saved the job after running it—theTTLincreases to 7 days by default
Y Job artifacts are saved in SPLUNK_HOME/var/run/splunk/dispatch/<job_id>
– This job directory exists for theTTLof the job
– If you manually kill a job, you may need to delete orphan job directories
– To identify the process ID and the search job ID, use the Job Inspector
Y TTLfor normal or saved states can be modified in limits.conf

What is the Search Job Inspector?
Y The Job Inspector provides
a large amount of information
about a search, including
many performance metrics
and job properties including:
– User
– App
– Process ID
– Search ID
docs.splunk.com/Documentation/Splunk/latest/Knowledge/ViewsearchjobpropertieswiththeJobInspector

limits.conf
Y limits.conf defines many settings that affect search
performance and optional capabilities
– Control memory allocated to jobs
– Use search parallelization
– Disconnect slow peers
– Restrict real-time searching or switch to indexed real-time search
– Control the maximum size of results that can be generated by a search
– Control the time-to-live (TTL) of a completed search job
Y limits.conf has settings for both search heads and indexers
– For distributed environment, settings must be on all relevant peers and search head for
consistent results
docs.splunk.com/Documentation/Splunk/latest/admin/Limitsconf
Note
Be careful when changing this
configuration. There are many
inter-dependent attributes. For
most, the default setting is optimal.

limits.conf Examples
...
[search]
ttl = 300
default_save_ttl = 172800
...
batch_search_max_pipeline = 2
...

[slow_peer_disconnect]
disabled = false
batch_search_activation_fraction = 0.9
threshold_connection_life_time = 60
...
Number of seconds a search job is stored on disk after
the search is complete
Y  Default = 600 (10 minutes)
Number of seconds a job is saved if a user clicks the
save option
Y  Default = 604800 (one week)
Y  0 = indefinite (NOT recommended)
Use more CPU cores to help search performance
Y  Default = 1
Y  Will increase threads and memory usage
In a large search peer environment, a slow (downed)
peer can result in poor search performance
Y  When enabled, Splunk finalizes the search once
90% of the peers have returned their results after 60
seconds by default

Switching to Indexed Real-time Search Mode
Y Real-time search searches the indexing pipeline before events are indexed
Y Indexed real-time search is a historical search that continually updates its event set as the
new events appear on disk
Y The number of concurrent real-time searches can greatly affect indexing performance
Y If up-to-the-second accuracy is not crucial, switch to indexed real-time search and lessen
the performance impact on the indexer
[realtime]
indexed_realtime_use_by_default = true
indexed_realtime_disk_sync_delay = 60
indexed_realtime_default_span = 1
indexed_realtime_maximum_span = 0
limits.conf

Changing Default Search UI
Y ui-prefs.conf contains attributes for Splunk Web view preferences
– This preference can be set globally, per app, or per user
[default]
dispatch.earliest_time = @d
dispatch.latest_time = now
[search]
display.prefs.enableMetaData = 0
display.prefs.showDataSummary = 0
display.page.search.searchHistoryTimeFilter = @d

Splunk Data Summary Options
Y Search execution time can be reduced by storing frequently used summary data
Y Splunk provides three data summary creation methods:
–  Datamodelacceleration–acceleratesallofthefieldsdefinedinadatamodel
–  Reportacceleration–acceleratesindividualreports
–  Summaryindexing–acceleratesreportsthatdon'tqualifyforreportacceleration
docs.splunk.com/Documentation/Splunk/latest/Knowledge/Aboutsummaryindexing
Y Summaryindexes live on search head (unlessforwardedtoindexingtier)
Y Datamodeland report accelerations aresavedonindexers
–  Youmustconsideradditionaldiskspaceneeds
Note
Choosing a summary option is
further discussed in the
Architecting and Deploying
Splunk course.

Identifying Acceleration
In Splunk Web, acceleration is marked with a lightning symbol
Report Acceleration Data Model Acceleration

Report Acceleration Summaries
Y Select Settings > ReportAcceleration Summaries
– Multiple reports can leverage the same summary
– If the summary load is high and it is rarely used, consider deleting it to reduce the strain
on the server
Y Will not accelerate if it involves less than 100K hot bucket events and 10% of total
bucket size
Y Scheduled to run every 10 minutes but can edit via Advanced Edit
Y Summarization load reflects the background effort required to keep summaries up to
date
1
2
1
2

Data Model Acceleration
Y All data models are accelerated
– Ad hoc pivots temporarily accelerate the model on search head
ê  Lasts for the duration of a user's pivot session
– Persistent accelerated data models live on indexers
ê  The acceleration window is scoped to a specific time range
ê  Old accelerated data that is outside of the scope gets reaped regularly
Y Only admins can persistently accelerate the data models
– Test model permissions and efficacy before accelerating
– Users need access to an index for which accelerations exist
– Data model editing requires turning off acceleration and re-building

DMC – Search Activity and Usage Statistics
Y Search activity
– Search concurrency, activity by app, user, mode, type, and role
– Median search concurrency and resource usage
– Top memory-consuming searches
– Aggregate search runtime
– Searches Started Per Minute by Dispatcher
Y Usage statistics
– Search activity by user and common searches by user
– Long-running searches
– Common search commands
Y SchedulerActivity
– Concurrency count, execution latency, and skip ratio
– Count of skipped reports, skipped report names, and reason
Note
The snapshots are taken every 10
seconds by default. A blank
snapshot panel means no
searches ran within the window or
they were very short lived.
The historical panels get data from
introspection logs.

Lab Exercise 18 – Monitoring Data Pipeline
Tasks:
– Check the current pipeline health with DMC
– To simulate an indexing issue, configure the forwarding host
– Check the pipeline dashboard again and observe the failing queues
– To restore the proper services, remove the forwarding host

Module 19:
Problem Isolation Overview

Module Objectives
Y Configure Splunk's internal logging settings
Y Troubleshoot issues and work with Splunk Support
Y Enable an alert to monitor your Splunk environment

Splunk Monitoring Tools
Y Various tools exist to help you investigate Splunk problems
Y We've already discussed:
– btool, SystemActivity Reports, and Distributed Management Console
Y Splunk log levels - Splunk collects a lot of data about itself
– _internal
– _audit
– _introspection
Y Another useful tool is diag

Splunk Log Levels
Y Log levels from lowest to highest: crit, fatal, error, warn, info, debug
Y By default all subsystems are set to info or warn
Y All of Splunk’s logs can be set to debug by restarting Splunk in debug mode
– Generally not recommended since it’s burdensome on production systems and
creates a lot of unwanted "noise" in the logs
– Better to set to debug granularly on the individual subsystem(s) you are
troubleshooting (see next slide)
– Splunk Support may ask for overall debug mode in certain cases

Setting Granularity of Log Levels
Y Adjust subsystem log levels to
debug and troubleshoot issues
Y In Splunk Web:
– Settings > System settings >
System logging
Y In config file:
– You can customize logging in
SPLUNK_HOME/etc/log-local.cfg
– Default logging should not be
changed in SPLUNK_HOME/etc/
log.cfg

Metrics.log
Y metrics.log provides useful insights into indexing processes:
– Which hosts are connecting to Splunk as inputs and how many times?
index=_internal source=*metrics.log* tcpin_connections
| stats count by sourceIP
– Where is Splunk trying to forward data to?
index=_internal source=*metrics.log* destHost | dedup destHost
– What output queues are set up?
index=_internal source=*metrics.log* group=queue tcpout
| stats count by name

index=_internal
source=*metric*
group=tcpout*
index=_internal
source=*metric*
series=<source>
Splunk Search Problem Isolation
lic. usage OK?
splunk_server=
<indexer>?
time range correct?
index=_internal
host=<forwarder>
Get reset key Check SH connection Search all time
Check forwarder setup
Check forwarder target
Check forwarder input
No No No
No
No
No

./splunk diag
Ensuring clean temp dir...
Selected diag name of:
diag-splunk.acme.com-2016-03-29
Starting splunk diag...
Copying Splunk configuration files...
Copying Splunk log files...
Copying index worddata files...
Copying Splunk log files in the dispatch dir...
Creating archive file...
Cleaning up...
Splunk diagnosis file created:/opt/splunk/diag-
splunk.acme.com-2016-03-29.tar.gz
What’s a Splunk Diag?
Y Gathers useful troubleshooting
information
– Splunk logs
– Splunk configuration files
– System information
-  No customer data retrieved
Y Serves as a quick backup of
configuration files
Y Run splunk diag before and
after upgrades to track release
differences

Inside a Diag
Y composite.xml – the unified config file that governs all processing
Y etc subdirectory – contains the diag system’s etc directory
Y log subdirectory – contains the diag system’s Splunk logs
Y var subdirectory – contains information about the indexes and index
structure
Y dispatch subdirectory – directory of search dispatches
Y systeminfo.txt – contains OS and hardware info as well as Splunk version
and build

Using Splunk diag
1. Access the Splunk server having the problem
2. Run SPLUNK_HOME/bin/splunk diag
- Diag can run with Splunk running or stopped
- Creates the file diag-<servername>-<date>.tar.gzin SPLUNK_HOME
- Retrieve the diag-<servername>-<date>.tar.gz file
– Additional diag command options:
http://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/Generateadiag
3. Unpack and examine the contents—Splunk it if you like!
4. If you are working with Splunk Support, send the file to them

DMC Platform Alerts
Y Effective operation of your Splunk
environment is timely identification and
notification of critical conditions
– Any view over 80% mark can’t be good
Y DMC Alerts Setup provides a number
of pre-configured platform alerts
– Platform alerts are disabled by default
– Tweak parameters such as alert
schedule, suppression time, and alert
actions

Lab Exercise 19 – Enable a DMC Alert
Tasks:
Enable an alert to monitor the physical memory usage

Module 20:
Introduction to Large-scale
Splunk Deployments

Module Objectives
Y Install universal forwarder on remote systems using scripts
Y Understand the role of heavy forwarders and index-and-forward
Y Understand basic Splunk index clustering concepts

Large-scale Roll-out of Splunk Forwarders
Y Manually installing forwarders on hundreds or thousands of systems is not
feasible
Y Two common options:
– Include the forwarder in your standard new server image
ê  Good for new servers, but not for existing servers
– Use an automated script to install the forwarder on remote machines
ê  Works for existing servers
Y *NIX and Windows systems have different requirements for automating
installation

Including the Forwarder in a Server Image
Y When configuring a new server image, you can also pre-install universal
forwarder
Y Do a normal install of Universal Forwarder and also:
– Enable boot-start (*NIX)
– Change the default admin password:
splunk edit user admin –password newPwd –auth admin:changeme
– Configure the deploymentclient.conf file to reference your deployment server
Y On the deployment server, make sure the new UF has been added to the
appropriate server classes
– The deployment server should deploy one or more apps with all required
inputs.conf and outputs.conf settings

Scripted Install Overview
1. Execute the script on the remote machine
–  Depending on operating system, various methods exist
–  Pass server-specific values
–  Your script needs to know at least the location to install the forwarder to, and the
hostname:port of the Splunk deployment server
2. The script downloads the universal forwarder install file
–  Normally done using a wget call, direct to the splunk.com download site
ê  Requires a user name and password
3. Unpack/execute the install file
4. Set the deployment server host and port, and start the forwarder

Docs for Deploying Forwarders
Y Example Install Scripts - *NIX
answers.splunk.com/answers/34896/simple-installation-script-for-universal-forwarder
docs.splunk.com/Documentation/Splunk/latest/Forwarding/Remotelydeployanixdfwithastaticconfiguration
Y Windows installer flags
docs.splunk.com/Documentation/Splunk/latest/Forwarding/DeployaWindowsdfviathecommandline
Y Windows forwarder deployment
docs.splunk.com/Documentation/Splunk/latest/Forwarding/RemotelydeployaWindowsdfwithastaticconfiguration
Y Powershell Script
answers.splunk.com/answers/60934/powershell-unattended-installation

Index
Inputs
Parsing/Route
Heavy Forwarders
Y Full Splunk instance
– Does everything as Splunk Enterprise but indexing
– Set the license group to Forwarder License
Y Accepts all input types and can parse raw data
Y Can route data to different indexers or 3rd party
receivers
Y Often used as an intermediary receiver for one or
more universal forwarders
– As a mid-tier component in a multi-stage data
routing design
– To aggregate data from universal forwarders, parse,
and route them to the indexers
– Can create a single point of failure
External
Server

Forwarder Deployment Options
Y An organization can deploy thousands of forwarders on various systems
– Types of deployment scenarios can introduce more forwarder concepts
ê  Intermediate forwarder
ê  Gateway forwarder
ê  Can be either a universal forwarder
or a heavy forwarder
Y Managing many forwarders can be
complex
– Use Deployment Server
Intermediate
Forwarder
Gateway
Forwarder
Indexer
Splunk Cloud
HF
UF

Introducing Splunk Clustering
Y Configure indexers to replicate indexes and group search heads to coordinate
their search loads and activities using commodity hardware
– Allows you to balance growth, speed of recovery, and overall disk usage
– Splunk Clustering is discussed in detail in Splunk ClusterAdministration class
High Availability (HA) Disaster Recovery (DR)
Indexing
Tier
Single-site cluster (index replication)
Y Available since Splunk 5
Y Flexible replication policies
Multisite cluster
Y Can withstand entire site failure
Y Supports active-passive and
active-active configurations
Search
Tier
Y  Independent search head
Y  Search head cluster
Search affinity (site-aware)
Y Independent search head
Y Search head cluster

Splunk Cluster Overview
Load-balanced
Forwarders
Indexers
Search Heads
Clustered
Clustered Additional Components
Y  Cluster Master
Y  Distributed Management
Console
Y  Deployment Server
Y  Deployer
Y  License Master
Site 1 Site 2

Auto Indexer Discovery
Y With indexer clustering, forwarders can get the list of indexers from Cluster
Master for elastic deployments
Y Discussed in detail in Splunk ClusterAdministration class
[tcpout]
server = idx1:9997,idx2:9997,idx3:9997
useACK = true
Manual indexer configuration: Auto indexer discovery:
[tcpout]
indexerDiscovery = mycluster
[indexer_discovery:mycluster]
master_uri = <cluster_master>

Further Reading: Clustering
Y Basic clustering concepts for advanced users
docs.splunk.com/Documentation/Splunk/latest/Indexer/Basicconcepts
Y Configure the search head
docs.splunk.com/Documentation/Splunk/latest/DistSearch/AboutSHC
Y Indexer discovery
docs.splunk.com/Documentation/Splunk/latest/Indexer/indexerdiscovery

Lab Exercise 20 -- Remote Forwarder Install
Tasks:
– Use a supplied script to execute a remote forwarder installation
ê  Not for production use
ê  Simplified for this training environment only

Course Wrap-up

Support Programs
Y Community
–  Splunk Answers: answers.splunk.com
Post specific questions and get them answered by Splunk community experts.
–  Splunk Docs: docs.splunk.com
These are constantly updated. Be sure to select the version of Splunk you are using.
–  Wiki: wiki.splunk.com
A community space where you can share what you know with other Splunk users.
–  IRC Channel: #splunk on the EFNet IRC server Many well-informed Splunk users “hang out” here.
Y Global Support
Support for critical issues, a dedicated resource to manage your account – 24 x 7 x 365.
–  Phone: (855) SPLUNK-S or (855) 775-8657
–  Web: http://www.splunk.com/index.php/submit_issue
Y Enterprise Support
Access your customer support team by phone and manage your cases online 24 x 7
(depending on support contract).

Thank You
Y Complete the Class Evaluation to be in this month's drawing for a $100 Splunk
Store voucher
– Look for the invitation email, What did you think of your Splunk Education class, in your
inbox
– Click the link or go to the specified URLin the email

What’s Next?
For detailed course and certification information go to: http://www.splunk.com/view/education/SP-CAAAAH9
If you have further questions, send an email to: certification@splunk.com
Power User
Certification
Using
Splunk
Searching
And Reporting
Creating
Splunk
Knowledge
Objects
Infrastructure
Overview
(e-learning)
Certified Power
User
Online Test
Advanced
Searching and
Reporting
Analytics and
Data Science
Administrator
Certification
Using
Splunk
Searching
And Reporting
Creating
Splunk
Knowledge
Objects
Infrastructure
Overview
(e-learning)
Certified Power
User
Online Test
Splunk
Administration
Certified
Administrator
Online Test
Splunk
Cluster
Administration
Architect
Certification
Using
Splunk
Searching
And Reporting
Creating
Splunk
Knowledge
Objects
Splunk
Administration
Advanced
Dashboards
and
Visualizations
Architecting
and Deploying
Splunk
Architect
Certification
Lab
Splunk for App
Developers
Using
Splunk
Searching And
Reporting
Creating
Splunk
Knowledge
Objects
Advanced
Searching and
Reporting
Advanced
Dashboards
and
Visualizations
Building
Splunk Apps
Developing
with Splunk
Java and
Python SDKs
Required
Required E-learning
Exam
Recommended
You are here

.conf2016: The 7th Annual
Splunk Worldwide Users’ Conference
Y  September 26-29, 2016
Y  The Disney Swan and Dolphin, Orlando
Y  4400+ IT & Business Professionals

Y  3 days of technical content
Y  175+ sessions

Y  3 days of Splunk University
Y  Sept 24-26, 2016
Y  Get Splunk Cer�ﬁed!
Y  Get CPE credits for CISSP, CAP, SSCP
Y  80+ Customer Speakers
Y  40+ Apps in Splunk Apps Showcase
Y  70+ Technology Partners
Y  1:1 networking: Ask The Experts and
Security Experts, Birds of a Feather and
Chalk Talks
Y  NEW hands-on labs!
Y  Expanded show ﬂoor, Dashboards Control
Room & Clinic, and MORE!
Visit conf.splunk.com for more information

Splunk 6.4 Administration.pdf

More Related Content

What's hot (20)

Similar to Splunk 6.4 Administration.pdf (20)

More from nitinscribd (8)

Recently uploaded (20)

Splunk 6.4 Administration.pdf