Elastic Security for SIEM

An Elastic Training Course

elastic.co/training

v1.0.2-on-demand

1
Welcome to Elastic Training
● Visit learn.elastic.co and log in
○ follow instructions from registration email to get access
● Go to "My Enrollments" and click on today's training
● Download the PDF file from the "Content" tab
○ this contains all the slides and lab instructions
● Click on "Access your virtual class here" to access the Lab
Environment

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
About Elastic Training
● Environment
○ Strigo test: app.strigo.io/system-test
● Introductions
● Code of Conduct
○ www.elastic.co/community/codeofconduct

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Elastic Security for SIEM
1. Elastic Stack Overview
2. Elastic Common Schema
3. Discover
4. Visualizations
5. Lens
6. Dashboards
7. Security App

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
 Elastic Security for SIEM Agenda
1. Elastic Stack Overview
2. Elastic Common Schema
3. Discover
4. Visualizations
5. Lens
6. Dashboards
7. Security App

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited

5
Elastic Stack Overview
Topics
● Elastic Stack Components
● Fleet and Elastic Agent

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Learning Objectives
Elastic Stack Overview

1 Describe basic Stack Architecture

1.1 Recognize the components of the Elastic Stack

1.2 Identify how data flows through the Elastic Stack

1.3 Discuss data sources for security use cases

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Learning Objectives
Fleet and Elastic Agent

2 Demonstrate use of Fleet and Elastic Agents

2.1 Explain the configuration and architecture of Fleet

2.2 Describe the management of Elastic Agents with Fleet

2.3 Describe integrations and the types available

2.4 Configure policies for an Elastic Agent

2.5 Deploy an Elastic Agent

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Elastic Stack Components
 Elastic Stack
What is the stack?

Kibana Visualize &

Manage

Elasticsearch Store, Search, &

Analyze
Elastic Stack

Beats Agent Logstash Ingest

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Elasticsearch
● Heart of the Elastic Stack
● Storage, search and
Log Monitoring
analytics layer
○ Fully scalable Infrastructure
Monitoring
○ Capable of solving data problems
Enterprise Search
at any scale
○ Handles common data Maps
transformations
SIEM
○ All data types are compatible
Endpoint Security

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Logstash
● Logstash is used to ingest
data into the Elastic Stack
● Server-side component
● Functions include:
○ Ingest data of all shapes and sizes
○ Normalizes and parses data
○ Transport data to any output
○ Connects stream data from other
storage layers

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Beats
● Beats ship data to
Elasticsearch or Logstash
● Functions include:
○ Sit on servers with containers or
available as a binary
○ Ship data to Elasticsearch
○ Forward data to Logstash for
additional parsing

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Beats Overview
● Filebeat
○ Logs and other data
Filebeat Metricbeat Packetbeat
● Metricbeat
○ Metric data
● Packetbeat
○ Network data
● Winlogbeat
○ Windows event logs
● Auditbeat
Winlogbeat Auditbeat Heartbeat
○ Audit data
● Heartbeat
○ Uptime monitoring
© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Beats Overview
• Auditbeat
‒ Groups related messages into a single event
‒ Ships with some default audit daemon rules and accepts existing audit
daemon rules
‒ Automatically maps usernames and process ids
‒ The audit daemon (auditd) can monitor syscalls
‒ The file integrity monitoring FIM) module is extremely efficient
• Winlogbeat
‒ Can parse and ship any Windows event log
‒ Monitors Active Directory interaction
‒ 3rd party Windows Event Log applications supported
‒ Integration with Sysmon
© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Elastic Agent
● A single, unified way to add
monitoring for Agent
○ Logs
○ Metrics
○ Other host data
Agent
● Managed via Kibana
● Ship to Logstash Agent
● Over 200+ integrations
○ Configure Elastic Defend for Agent
Security as an integration

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Kibana
● Kibana is the window into the
Search
Elastic Stack
● User-facing component of
Elastic Stack
● Browser-based UI layer
○ Visualize and explore data in
Elasticsearch
○ Investigate and respond to threats
with the Security App
○ Manage the Elastic Stack

Protect

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Elasticsearch Data Journey

Elastic Agent

Elastic Defend

Data Source Ingest Store Analyze

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Fleet and Elastic Agent
Introduction to Fleet
● Fleet is a web-based UI in Kibana for centrally managing Elastic Agents
and their policies
Manage Elastic Agents
and their policies
through Kibana

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Fleet Prerequisites
● In order to use Fleet in Kibana you need:
○ A Fleet server deployed
○ Internet access for Kibana to reach out to internet for Integrations updates
and downloads
■ Air-gapped environments and offline endpoints are supported
○ Kibana user with all privileges

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Fleet Configuration

Get
Enroll Policy

Kibana
Return Return Policy Fleet UI
Policy Policy
Elastic Agent
Elasticsearch
Policy Package
Fleet Server Builder Manager

Data

Artifact Package
Registry Registry

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Fleet UI Package Manager
● The Fleet package manager is used behind the scenes to both
manage and explain integrations provided by Elastic via Beats and
the Elastic Agent
● Typically requires an internet connection because integrations are
updated and released periodically
○ Air-gapped environments can be supported via a local package repository

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Fleet UI Integrations page
● Connects Elastic to
external services and Add and manage
integrations through
systems Kibana
● Can collect new sources
of data
● Supports air-gapped
environments
● Often ships with
out-of-the-box assets
○ Dashboards
○ Visualizations
○ Pipelines
© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Integrations: Elastic Defend
● Prevent complex attacks
○ Prevent malware
○ Prevent ransomware execution
○ Stop advanced malicious behavior
○ Memory threats
● High fidelity detections and alerts
● Rapid triage and response
● Secure cloud workloads Elastic Defend
● Replay terminal sessions

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Additional Integrations
● Osquery
○ Query operating systems like a database
○ Provides visibility into infrastructure and operating systems
○ Live queries across entire enterprise for incident response, threat
hunting, and vulnerability detection
● AbuseCH
○ Ingests threat intelligence indicators from multiple feeds
○ Provides malicious URL-based indicators
○ Supports malware-based indicators
○ Utilizes known indicators of compromise

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Fleet Configuration
Fleet UI Policy Builder
● Create an Agent policy to enroll Agents into
● An Agent policy is a set of individual integrations
● Each integration provides several settings

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Fleet UI Agent Policies
● Attach integrations to the Agent policy
● Multiple agents can be assigned to one policy
● Configure settings under each integration attached to the policy

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Fleet UI Agents
● Centrally manage your deployed Agents
● Monitor/Troubleshoot Agents status
● Deploy new Agents

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Fleet Configuration

Get
Enroll Policy

Kibana
Return Return Policy Fleet UI
Policy Policy
Elastic Agent
Elasticsearch
Policy Package
Fleet Server Builder Manager

Data

Artifact Package
Registry Registry

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Elastic Package Registry
● An online package hosting service for the Elastic Agent integrations
available in Kibana
● This capability allows integrations to be updated and released periodically
within the Elastic release cadence and as the packages are updated
○ Kibana connects to the package repository at epr.elastic.co
○ Air-gapped environments can be supported by hosting a local package
repository

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Fleet Configuration

Get
Enroll Policy

Kibana
Return Return Policy Fleet UI
Policy Policy
Elastic Agent
Elasticsearch
Policy Package
Fleet Server Builder Manager

Data

Artifact Package
Registry Registry

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Elastic Artifact Registry
● An online package hosting service that provides necessary binaries for
Elastic Agent installation and upgrades.
○ Deployed Elastic Agents download binaries from artifacts.elastic.co
○ Elastic Defend requires additional access to updates and security artifacts
from security.artifacts.elastic.co
○ Air-gapped environments can be supported by hosting a local artifact
repository

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Fleet Configuration

Get
Enroll Policy

Kibana
Return Return Policy Fleet UI
Policy Policy
Elastic Agent
Elasticsearch
Policy Package
Fleet Server Builder Manager

Data

Artifact Package
Registry Registry

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Fleet Server
● Used to centrally manage Elastic Agents, it allows remote Elastic Agents
to communicate with Elasticsearch
● Deployed as a special agent policy via Elastic Agent
● Allows for a scalable infrastructure
● Supported in Elastic Cloud and self-managed

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Lab Environment ROEs
● Students WILL NOT
○ Interfere with other students/elements
○ Deny/degrade access to shared resources
○ Respond to systems from other students/elements
○ Access investigations created by other students
○ Modify or alter administrative configurations
○ Create or modify accounts or permissions
○ Carve, copy, or execute malicious samples, files, or malware found

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Accessing Your Kibana Instance
● Credentials:
○ Directory: /home/student
○ File: CREDENTIALS

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
 Kibana Access
and Fleet Configuration Demo
Follow along with your instructor as they
guide you through
Accessing Kibana and configuring Fleet
 Lab 1
Elastic Stack Overview
Lab 1.1 Elastic Agent Configuration
Lab 1.2 Configuring Agent Policies and
Integrations
Summary: Elastic Stack Overview
● The Elastic Stack efficiently collects, stores, and analyzes large
volumes of data for log management, monitoring, and data
visualization
● Fleet manages Elastic Agents, which collect and forward various
types of data to Elasticsearch, which simplifies data management in
the Stack

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Elastic Security for SIEM Agenda
1. Elastic Stack Overview
2. Elastic Common Schema
3. Discover
4. Visualizations
5. Lens
6. Dashboards
7. Security App

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Elastic Common Schema
Topics
● Normalizing Data
● Logs and Data Structures

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Learning Objectives
Elastic Common Schema

3 Examine the application of Elastic Common Schema ECS

3.1 Discuss the fundamentals of logging and data structures

3.2 Recognize issues with aggregating data from disparate data sources

3.3 Identify how ECS normalizes data

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Normalizing Data
Aggregating Disparate Data Sources
● Like sensors, data comes in all
shapes and sizes Analysis
● Ingesting data from multiple
Archiving
sources could create a
bottleneck
● May require multiple systems or Schemaless
Repository
tools to analyze data Monitoring

Alerting

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Data Normalization
● Data normalization is the process of accepting data from multiple
schemas and converting it into a consistent schema
Host Address
Client Address
Host IP
Client IP
Source Address
Source IP

destination.port 80

Host Port source.ip 172.16.100.54

Source Port
Client Port source.port 56690

Destination Port
Server Port
Remote Port

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Normalize data with ECS

Searching without ECS Searching with ECS

host.address:172.16.100.54 source.ip:172.16.100.54
OR client_ip:172.16.100.54
OR host_ip: 172.16.100.54
OR source_address:172.16.100.54
OR src_ip:172.16.100.54

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
What is Elastic Common Schema
● Elastic Common Schema ECS) is designed as an open source
specification that supports uniform data modeling
● ECS defines a common set of fields to be used when storing event data
in Elasticsearch
● ECS enables users of Elasticsearch to normalize event data
● ECS provides the ability to analyze, visualize, and correlate events from
any data source

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Normalization Benefits
● Simplified field naming with few rules and exceptions
● Increases search speed efficiency
● Ability to:
○ Simply correlate data from different sources
○ Reuse analysis content across multiple data sources
○ Incorporate future Elastic/partner provided analysis in an established environment
without modifications
● Helps uniformly examine data in:
○ Interactive searches
○ Visualizations
○ Automated analysis
© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
ECS Normalized Data
● Elastic provided integrations
○ Hundreds of out of box integrations
○ Custom data sources
● Logstash `` Agent
○ Server-side data processing
○ Ingests data and transforms it
● Beats
○ Normalized data sent to Elasticsearch
○ Disparate data sent to Logstash
● Agent
○ Normalized data sent to Elasticsearch
© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Logging and Data Structures
Elasticsearch Data Structure
● Data in Elasticsearch is stored in an index
● Elasticsearch indices contain a collection of documents
● Documents are a collection of data field names and field values for a
specific log event

Names Values
Index

"source.ip": "192.168.1.10",
"network.protocol": "http",
"network.bytes": 232

Document Document Document

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Elasticsearch Fields - Data Types
● Each field also has a data type
● The data type dictates which kinds of searches can be used on that field
● There are many different data types recognized by Elasticsearch
● Commonly used data types:
○ Date: allows for searching by ranges of time
○ Numbers: numeric values including long (integers) and double (decimals)
○ Strings: a sequence of characters used to store normal text
○ IPs: allows searching for ranges of IP addresses using CIDR notation

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
ECS Field Types
● ECS defines common fields for ingesting data into Elasticsearch
● Common data fields support the uniform data modeling used in ECS
● Three types of data fields: core, extended, and custom

Level Description Recommendation

Core Fields Fully defined set of field names that exists These fields are common across most use
under a defined set of ECS top-level objects cases, so work should begin here

Extended Fields Partially defined set of field names that exists Extended fields may apply to narrower use
under the same set of ECS top-level objects cases or be open to interpretation depending
on the use case

Custom Fields Undefined set of user-supplied non-ECS This is where you can add fields.
top-level objects. Must not conflict with ECS
fields or objects

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Summary: ECS
● Elastic Common Schema ECS) defines a common set of fields for
ingesting data into Elasticsearch
● Data comes in all shapes and sizes. Be aware of the implications
when data comes from multiple data sources
● Data normalization keeps your data the same while increasing search
speed and efficiency
● Elasticsearch uses indices to store data. Each index contains
documents with fields and values associated to events
● Fields also have associated data types which dictate the types of
searches can be performed on that field

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Elastic Security for SIEM Agenda
1. Elastic Stack Overview
2. Elastic Common Schema
3. Discover
4. Visualizations
5. Lens
6. Dashboards
7. Security App

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Discover
Topics
● Index Patterns and Data Views
● Discover Fundamentals
● Querying data

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Learning Objectives
Discover

3 Customize the Discover interface to search for data

3.1 Identify how to access data from different Elasticsearch indices

3.2 Discuss how Kibana displays data in Discover

3.3 Choose the appropriate settings in Discover to view relevant data

3.4 Construct queries using KQL/Lucene to view relevant data

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Index Patterns and Data Views
Index Patterns and Data Views
● Your data is stored in Elasticsearch indices
● Kibana requires an index pattern/Data view to access the Elasticsearch
data that you want to explore
○ Index pattern is the legacy terminology replaced by Data View starting with Kibana
version 8.0
● You can point to one or more indices, data stream, or index aliases

All ECS

Network IDS logs

Network Traffic
Metadata
© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Index Patterns - All ECS Indices Index Pattern: ecs-*

ecs-suricata-network-2018.09.01 ecs-zeek-network-2018.09.01

ecs-suricata-network-2018.09.02 ecs-zeek-network-2018.09.02

ecs-suricata-network-2018.09.03 ecs-zeek-network-2018.09.03

ecs-suricata-network-2018.09.04 ecs-zeek-network-2018.09.04

ecs-suricata-network-2018.09.05 ecs-zeek-network-2018.09.05

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Index Patterns - Suricata Only Index Pattern: ecs-suricata*

ecs-suricata-network-2018.09.01 ecs-zeek-network-2018.09.01

ecs-suricata-network-2018.09.02 ecs-zeek-network-2018.09.02

ecs-suricata-network-2018.09.03 ecs-zeek-network-2018.09.03

ecs-suricata-network-2018.09.04 ecs-zeek-network-2018.09.04

ecs-suricata-network-2018.09.05 ecs-zeek-network-2018.09.05

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Index Patterns - All Zeek Data Index Pattern: ecs-zeek*

ecs-suricata-network-2018.09.01 ecs-zeek-network-2018.09.01

ecs-suricata-network-2018.09.02 ecs-zeek-network-2018.09.02

ecs-suricata-network-2018.09.03 ecs-zeek-network-2018.09.03

ecs-suricata-network-2018.09.04 ecs-zeek-network-2018.09.04

ecs-suricata-network-2018.09.05 ecs-zeek-network-2018.09.05

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Discover Fundamentals
Where to Start
● Start with the home page
● Provides access to the solutions, and everything you need to visualize
and analyze your data

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Discover
● Use Discover to search your data for hidden insights and relationships

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
 Discover Components
Demo
Follow along with your instructor as they
guide you through
Accessing Kibana and Features of the
Discover Application
Discover Index pattern
/ Data view
Time filter

Histogram

Doc table

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
 Query
Fields list Toolbar
bar
Discover

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Analyze Your Data with Discover
● Search through the list of available fields
● Click on a specific field in the Field list pane to see its 10 most popular
values available in the documents explorer

Type your
field name

Visualize the
data in Lens

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Customize the Discover Interface
● By default, the document table includes a column for the time field and a
column that lists all other fields in the document
● Youʼll modify the document table to display your fields of interest
● Click on the + icon to add a field to the table
● To rearrange the table columns, click a column header, and then select
Move left or Move right

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Searching for Data
● You can use the query bar to write a query for results using a variety of
methods
○ Many times, the same results can be achieved as using a filter

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Filtering your Data
● You can filter results to include or exclude specific fields, filter for a
value in a range, and more
● Edit or remove your filter any time

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
 Lab 3
Discover
Lab 3.0 Getting Started with Kibana
CTFd flag: “acquisitionˮ
Lucene and KQL Query Syntax
Kibana Query Language
● Introduced as the default language in version 7.0 for filtering
Elasticsearch data
● Search capabilities include:
○ Search where a field exists
○ Field Matching
○ Search within a range of values
○ Negation
○ Combine queries using boolean operators
○ Multiple field matching
○ Nested and scripted field search
● Syntax meant to be easier to use and offers field suggestions

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Lucene Query Syntax
● Original search language of Kibana
● Also used for filtering Elasticsearch data
● Many of the same search capabilities as KQL with slightly different
syntax
● Additional search capabilities include:
○ Fuzzy searching
○ Proximity searching
○ Regular expressions

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Elasticsearch Fields - Data Types
● Each field has a data type
● Commonly used data types:
○ Date: allows for searching by ranges of time
○ Numbers: numeric values including long (integers) and double (decimals)
○ Strings: a sequence of characters used to store normal text
○ IPs: allows searching for ranges of IP addresses using CIDR notation

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Elasticsearch Fields - Text and Keyword
● Strings in Elasticsearch are broken into either a text or keyword type
● Text
○ Analyzed
○ Broken into a list of terms (tokens) before being indexed
○ Enables searching of individual words within the string
○ More index resources required
● Keyword
○ Not analyzed
○ Will only match searches when the entire string is matched exactly
○ Can be aggregated and sorted

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Elasticsearch Fields - Multi-fields
● Index data using more than one data type
● Enables searching of the same data in different ways
○ Use the text type for full-text search
○ Use the keyword type for aggregating and sorting the field values
● The same data is indexed in more than one field
○ Strings are normally indexed as the keyword type
○ The text type is indexed as a multi-field, denoted by the additional .text portion in the
field name

user_agent.original vs user_agent.original.text

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Full-text Search
● Search for any phrase or word within an analyzed field
● Elasticsearch performs analysis on text fields during indexing
● Useful when field values are bodies of text that contain individual values
we might be interested in
● Examples of fields that are analyzed:
○ User-agents
■ Search for specific parts of a user-agent string

○ DNS queries
■ Search for a specific substring of a domain in a DNS query

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Field Matching
● Searching for a specific field name and a specific value
● Leverages Elasticsearch indexing to optimize search speed
● Much more efficient than full-text search

<FIELD>: <VALUE>

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Field Matching
● Entries from the Zeek conn.log

event.dataset: conn

● Log entries where the server/responding port is 5353

destination.port: 5353

● All Zeek logs associated with the event with UID CaYzxD4PdT2Dtz2J8d

zeek.session_id: CaYzxD4PdT2Dtz2J8d

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Boolean Queries
● Uses AND, OR and NOT operators
● Allows for more complicated searches by returning results based on if:
○ More than one condition has been met
○ One condition OR another condition has been met
● Group values using parentheses:
○ When searching for multiple values for the same field
○ To set search precedence
■ Boolean operators inside parentheses are evaluated before operators outside parentheses

● Best practice to always uppercase Boolean operators

○ Case insensitive in KQL
○ Must be uppercase in Lucene
© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Boolean Queries

NOT NOT <FIELD>:<VALUE>

AND <FIELD>:<VALUE> AND <FIELD>:<VALUE>

OR <FIELD>:<VALUE> OR <FIELD>:<VALUE>

Field Grouping

<FIELD>: (<VALUE> OR <VALUE>)

(<FIELD>: <VALUE> OR <FIELD>: <VALUE>) AND <FIELD>:<VALUE>

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Boolean Queries
● Entries from the Zeek conn.log OR the http.log

event.dataset: (conn OR http)

● Log entries where the protocol is http and the port is not 80

network.protocol: http AND NOT destination.port: 80

● Logs where the protocol is http AND NOT port 80, OR where the port is 80
AND the protocol is NOT http
(network.protocol: http AND NOT destination.port: 80) OR
(destination.port: 80 AND NOT network.protocol: http)

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Boolean Queries
(network.protocol: http AND NOT destination.port: 80)
OR (destination.port: 80 AND NOT network.protocol: http)

network.protocol : server.port :
http 80

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Wildcards
● Basic wildcard searching Lucene
● Resource intensive
dns.\*: google*
● Can wildcard part of a field name or a
field value ● Escape wildcards in field names

<FIELD>.*: *<VALUE>*

● Search "url.domain" for any value that

ends with ".com"

url.domain: *.com

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Field Exists
● Searching for a specific field name to Lucene
see if it exists in the data
_exists_: <FIELD>
<FIELD>: *

● Results from the dns.log where the

DNS answers field is empty

event.dataset: dns AND NOT dns.answers.name: *

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
IP Addresses
● Fields must have the IP data type Lucene
● Search using CIDR notation
source.ip: 172.16.100.0\/24
● Exact IP match
● Escape CIDR notation
source.ip: 172.16.100.54

● Range of IPs
source.ip: 172.16.100.0/24

● Internal to external traffic

source.ip: 172.16.100.0/24 AND NOT destination.ip: 172.16.100.0/24

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Numbers and Ranges
● Search for an exact match Lucene
source.port: 56689
source.port:>56689

● Comparison Operators source.port:[56689 TO 56692]

○ No colon with comparison operators
source.port:{56689 TO *}
source.port > 56689
● Include a colon with comparison
source.port <= 56689 operators
● Range syntax uses [ ] inclusively
● Range syntax uses { } exclusively
● Define a numerical range
source.port > 20 AND source.port < 100
© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Lucene Specific Search Types
Lucene Queries - Regular Expressions
● Apache Lucene regex
○ Similar to PCRE
● Encapsulate the query in forward slashes

<FIELD>: /[PATTERN]/

● Case insensitive HTTP GET requests

http.request.method: /[Gg][Ee][Tt]/

● DNS answers formatted like IP addresses

dns.answers.name: /([0-9]{1,3}\.){3}[0-9]{1,3}/
© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Lucene Queries - Fuzzy
● Find terms that are similar to the search term
● Leverages full-text search, use on .text fields
● Use a ~ at the end of the search term
<FIELD>: <VALUE>~

● Find terms that are similar to "Mozilla" but are not exactly "Mozilla"
user_agent.original.text: Mozilla~ AND NOT
user_agent.original.text: Mozilla*

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Lucene Queries - Proximity
● Search for terms within a defined distance from each other
○ Distance is number of tokenized terms after first match
● Leverages full-text search, use on .text fields
● Encapsulate search terms in double quotes " "
● Use a ~ and a numeric value after the search terms

<FIELD>: "<VALUE1> <VALUE2>"~<NUMBER>

● Find user-agent string where "Windows" and "x64" appear within 5

tokens of each other

user_agent.original.text: "Windows x64"~5

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
 Lucene and KQL
Demo
Follow along with your instructor as they
guide you through
searching with Lucene and KQL
 Lab 3
Discover
Lab 3.1 Searching with KQL/Lucene
CTFd: “pursueˮ
Summary: Discover
● Discover provides access to every document in every index in the
cluster
● By querying, filtering, and parsing information we are able to better
analyze our data in Discover

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Elastic Security for SIEM Agenda
1. Elastic Stack Overview
2. Elastic Common Schema
3. Discover
4. Visualizations
5. Lens
6. Dashboards
7. Security App

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Visualizations
Topics
● Aggregation-based

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Learning Objectives
Visualizations

4 Create aggregation-based visualizations for security use cases.

4.1 Recognize the different types of aggregations and visualizations.

4.2 Identify best practices for creating aggregation-based visualizations.

4.3 Given parameters, construct an aggregation-based visualization.

4.4 Interpret data within visualizations.

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Aggregation-based
Core of Kibana: Aggregation Based Visualizations
● Data is often complex and involves many dimensions
● Often, we want summarized insights:
○ Slices based on specific attributes
○ Calculations based on specific attributes
● Two types of aggregations:
○ Metrics: Compute a numeric value
○ Bucket: Slice your data

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Metrics Aggregation
● Calculates numerical values over a set of documents
● Similar to how values are summarized in a pivot table for a specific
column
● Mathematical operation that outputs
○ Single value (eg., avg, sum, min, max, unique count)
○ Multiple values (eg., percentiles, percentile_ranks)

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
A Simple Example: Spreadsheet

id user age country category

1 Bill 30 FR A

2 Marie 32 US A

3 Claire 32 US A

4 Tom 44 DE B

5 John 40 US B

6 Emma 26 US B

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
A Simple Average Using a Pivot Table
id user age country category

1 Bill 30 FR A

2 Marie 32 US A

3 Claire 32 US A

4 Tom 44 DE B

5 John 40 US B

6 Emma 26 US B

Pivot table definition Pivot table

Rows Values AVG of age

AVG of age 34

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Buckets Aggregation
● A way of slicing data
● Similar to grouping by values in rows or columns in a pivot table
● Creates buckets
○ Collection of documents that share a common criterion
○ Can have one or more metrics associated with it
○ Number of documents (doc count) per bucket is default metric

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
A Simple Bucket Using a Pivot Table
id user age country category

1 Bill 30 FR A

2 Marie 32 US A

3 Claire 32 US A

4 Tom 44 DE B

5 John 40 US B

6 Emma 26 US B

Pivot table
Pivot table definition
category COUNT of id
Rows Values
A 3
Order ASC by category COUNT of id
B 3

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Bucket Aggregation

Metrics
Aggregation

Bucket
Aggregation
© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Sub-Bucket Aggregation

Metrics
Aggregation

Bucket
Aggregation
© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Visualization Types

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Aggregation Based Visualization
Demo
Follow along with your instructor as they
guide you through
aggregation based visualizations
Bucket Aggregation
● Buckets allow you to split your data based on the value in a field

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Metric Aggregation - Count
● Default metric for visualizations
● Counts the total number of records/documents

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Metric Aggregation - Math Functions
● More than just counting records
● Math functions against the values of certain fields
● User has to specify the field to run calculations against

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Sub-Bucket Aggregation
● Subdivides buckets into small groupings
● Groups records based on different
combinations of values in specified
fields

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
 Lab 4
Visualizations
Lab 4.1 Data Table
CTFd flag: “accessˮ
Elastic Security for SIEM Agenda
1. Elastic Stack Overview
2. Elastic Common Schema
3. Discover
4. Visualizations
5. Lens
6. Dashboards
7. Security App

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Visualizations
Topics
● Lens

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Learning Objectives
Visualizations

5 Construct Lens visualizations for security use cases

5.1 Discuss the relevance of Lens

5.2 Demonstrate the use of additional features in Lens

5.3 Convert aggregation-based visualizations to Lens

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Lens
Lens
● Lens is the recommended visualization editor
● Provides a modern interface for quick visualization creation and editing
● Default visualization editor from within Kibana modules

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Lens Features
● Field names available in Fields List
○ Provides quick access to all available fields in index pattern
● Drag & Drop
● Suggestions for alternate visualizations
● Quick edit and customization
○ Easily switch between visualization types & index patterns
○ Swap fields and axes values with drag/drop and single click
○ Visualization updates automatically applied

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
 Lens Components
Index
Pattern/Data Visualization type
View
Layer pane

Fields List

Workspace

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Customize Visualizations with
Lens Demo
Follow along with your instructor as they
guide you through
customizing visualizations with Lens
Fields List
● List is populated based on index
pattern/data view
● Interface similar to Discover
○ Search & Filter fields
○ Display field data types
○ Interactive
■ Click fields for popup visualization
● Drag & Drop

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Drag & Drop
● Fields dragged onto workspace automatically transformed into a chart
○ Lens evaluates the field data type to choose appropriate Metric
■ Text: Top Values
■ Number: Median
■ Timestamp: Count of Records

● Fields can also be dropped on specific axis and break down by settings

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Layer Pane
● Settings for data elements of the visualization
● Data elements can be dragged between areas
● Settings adjust based on visualization type
● Supports multiple layers
○ Different visualizations can be combined through
layers
■ Independent type & index pattern

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Suggestions
● Use the suggestions to quickly apply a different visualization type

Try a
suggested
visualization

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Chart Settings
Customize the look & feel of full-chart elements
– Labels, Legends, Axes, Gridlines
Available settings adjust based on visualization type
Visual options Legend Axes

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Field Settings
● Each field group has different settings
○ Display names and other cosmetic settings
○ Ordering, grouping, field selections and other
functional settings

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Metrics Settings
● Default functions are applied by Lens
● Modify data format, display value
● Different settings available for different
data and visualization types

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Layers
● Lens visualizations support multiple layers
○ Multiple charts can be displayed in the same interface
● Layers are independent
○ Visualization type
○ Index pattern

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
 Lab 5
Lens
Lab 5.1 Create a Visualization
Lab 5.2 Data Table
Lab 5.3 Multi-layer Date Histogram
CTFd flag: “ancestorˮ
Summary: Visualizations
● Aggregation based visualizations help us visualize our data, find
outliers, and track metrics over time
● To create a visualization: decide your criteria, group data by buckets,
and perform a metric
● Lens simplifies the visualization creation process and has additional
functionality, such as layering indices and charts

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Elastic Security for SIEM Agenda
1. Elastic Stack Overview
2. Elastic Common Schema
3. Discover
4. Visualizations
5. Lens
6. Dashboards
7. Security App

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Dashboards
Topics
● Dashboard Fundamentals

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Learning Objectives
Dashboards

6 Construct Dashboards for security use cases

6.1 Identify best practices for creating Dashboards

6.2 Given parameters, build a Dashboard

7 Demonstrate the use of Dashboards

7.1 Identify best practices for pivoting between apps in Kibana

7.2 Given a scenario, interpret data using relevant Dashboards

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Dashboard Fundamentals
Dashboards
• Collection of visualizations & saved searches
• Requires data that is indexed and a data view

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
 Dashboards
• Dashboards are simply a collection of visualizations
• Focus on one data type/activity at a time Word Cloud

Bar Chart

Metric

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Dashboards - conn.log data
• Different logs sometimes have different fields
• Focus for this dashboard is on the Zeek conn log

Zeek Log
Navigation

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Dashboards - http.log data
• Tailored to HTTP dataset
• Utilizes a saved search in the backend for each visualization
Top HTTP
Request
Methods

Top
HTTP
Request
Bytes

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Filters
• Appear right below the search bar
• Filter for any value from any visualization
• Impact all visualizations/data
• Can be renamed for clarity

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Filters - Modification Options

● Pinning a filter will allow

it to persist across all
● Useful for removing known Kibana modules
elements ● Indicated by slim grey
bar

● Allows for quick

analysis without having
to rebuild filters
© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Filters - Pin to Pivot

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Time Range
• Use the pre-built date
ranges we have provided
• Be careful to reset your
time filter after you have
exhausted a lead

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Time Range - Granularity
• Changing the date/time range will
impact the granularity of the
Histogram
• The smaller the time range, the
more detail you get
• Use caution when trying to
determine when a spike occurred

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Time Range - Granularity Impact
Aug 1, 2018 @ 11:02:37

Aug 1, 2018 @ 11:13:32

Aug 1, 2018 @ 11:26:10

Aug 1, 2018 @ 11:27:32

Aug 1, 2018 @ 11:40:32

Aug 1, 2018 @ 11:41:35

Aug 1, 2018 @ 11:45:10

11:00 - 11:30 -
11:30 12:00

@Timestamp per 30 Minutes © Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Time Range - Granularity Impact
Aug 1, 2018 @ 11:02:37

Aug 1, 2018 @ 11:13:32

Aug 1, 2018 @ 11:26:10

Aug 1, 2018 @ 11:27:32

Aug 1, 2018 @ 11:40:32

Aug 1, 2018 @ 11:41:35

Aug 1, 2018 @ 11:45:10

11:00 11:10 11:20 11:30 11:40 11:50

@Timestamp per 10 Minutes © Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
 Dashboards Demo
Follow along with your instructor as they
guide you through
dashboards
 Lab 6
Dashboards
Lab 6.1 Creation
Lab 6.2 Analysis
CTFd flag: “arrangementˮ
Summary: Dashboards
● Dashboards allow us to place visualizations and queries in a single
display
● The visualizations within a Dashboard are interactive
○ Use filters and queries to drill down into the results

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Elastic Security for SIEM Agenda
1. Elastic Stack Overview
2. Elastic Common Schema
3. Discover
4. Visualizations
5. Lens
6. Dashboards
7. Security App

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Security Apps
Topics
● Getting Started
● Elastic AI Assistant for Security
● Attack Discovery
● Detection Engine
● Alerts
● Timeline and Cases

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Learning Objectives
Security Application

8 Recognize the capabilities of the Security App.

8.1 Describe the capabilities of the Security App.

9
Use Explore within the Security App to view security related
events.

9.1 Summarize relevant information within the Host, Network, and User pages.

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Learning Objectives
Security Application

10
Describe how the Detection Engine searches activity and
generates alerts.
10.1 Identify the anatomy of a rule.

10.2 Distinguish between pre-built and custom rules.

10.
3 Construct custom detection rules with KQL, EQL, Lucene, and ES|QL.

11 Analyze Alerts that are generated from Detection rules.

11.1 Assess Alerts via Event Visual Analyzer and Tables for relevant data.

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Learning Objectives
Security Application

12 Correlate relevant data using Timeline.

12.1 Construct Timelines using KQL/Lucene/EQL to view relevant data.

13 Track security events using Cases.

13.
1 Import Lens, Timelines, and Hyperlinks using comments.

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Getting Started
What is the Elastic Security App?
● The Elastic Security App is a suite of tools that equips teams to prevent,
detect, investigate, and respond to evolving threats

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
The Main Components
Automatically searches for suspicious host and
Detection network activity in indices, it is responsible for alerts in
Engine the Security app and allows for reactive response to an
alert firing

Timelines Workspace for investigating alerts and events

An internal system to open, track and share security

Cases issues directly in the Security app

Manage A dedicated space to view and manage hosts

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
How to Get Started
● Easiest way to get started using the Security App is to ingest some data
using Beats or Agent
● Data shipped to Elasticsearch is organized in different pages based on
category:
○ Filebeat Zeek logs, Syslog, Threat Intelligence)
○ Packetbeat
○ Auditbeat File Integrity Monitoring)
○ Winlogbeat Windows Logs)

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
 Security App Demo
Follow along with your instructor as they
guide you through
Security App
 Lab 7
Security App
Lab 7.1 Getting Started with the Security App
CTFd flag: “analysisˮ
Explore
● Data shipped to Elasticsearch is organized in different pages:
○ Host: comprehensive overview of all hosts and host-related security events. Data is
presented in within interactive Charts, Data table and Widgets and linked to Timelines
for further investigation.
○ Network: key network activity metrics in an interactive map and provides network event
tables. You can drag and drop items of interest from the Network view to Timeline for
further investigation.
○ Users: comprehensive overview of user data to help understand authentication and user
behavior

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
 Explore Demo
Follow along with your instructor as they
guide you through
Security App Explore
 Lab 7
Security App
Lab 7.2 Explore Pages
CTFd flag: “analysisˮ
Elastic AI Assistant for Security
Elastic AI Assistant
● Elastic AI Assistant is designed to enhance analysis.
● Utilizes generative AI for interaction with Elastic Security
○ Alert investigation
○ Incident response
○ Query generation
● Elastic does not store or examine prompts or results used by Elastic AI
Assistant
○ Data will be processed by the third-party large language model LLM) provider you
connected to as part of setup

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Elastic AI Assistant for Security - Interaction
● System prompts at the beginning of a conversation establish how
detailed and technical you want answers to be
● Two built-in prompts are provided
○ Default system prompt
○ Enhanced system prompt
● Also an option to provide a custom prompt

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Elastic AI Assistant for Security - Interaction
● There are also quick prompt options at the bottom of the flyout
○ Intended to aid in writing prompts for a specific purpose
● Quick prompt availability varies based on context
○ Summarizing alerts
○ Workflow suggestions
● As with system prompts, you can also create custom quick prompts

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Elastic AI Assistant for Security - Knowledge base
● Elastic AI Assistant can help you write ES|QL queries or answer general
questions about ES|QL syntax and usage
● This can include:
○ Writing new queries
○ Providing feedback to optimize existing queries
○ Customizing queries for your environment
○ Troubleshooting

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Elastic AI Assistant for Security - Alert Triage
● Elastic AI Assistant can help your alert triage workflows by:
○ Assessing multiple recent alerts in your environment
○ Helping you interpret an alert and its context
● AI Assistant can
○ Answer questions about data
○ Offer insights and actionable recommendations to remediate the issue
■ For example: “What ES|QL query would isolate actions taken by this user?ˮ

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Elastic AI Assistant for Security - Document
● Can also streamline the documentation and report generation process
by:
○ Providing clear records of security incidents
■ Their scope, impact, and remediation efforts
● You can add the following to Elastic Securityʼs case management system
to generate incident reports
○ Add data
○ Narrative summaries
○ Other information from Security Assistant responses

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Elastic AI Assistant for Security - Propagate
● Elastic AI Assistant can translate its findings into other human languages
○ Helping to enable collaboration among global security teams
○ Making it easier to operate within multilingual organizations
● After Elastic AI Assistant provides information in one language, you can
ask it to translate its responses
○ You can then add the translated output to a case

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Attack Discovery
Attack Discovery
● Attack discovery utilizes Large language models LLMs
● Generative AI provides the ability to identify and describe attacks
● Discoveries are analyzed against MITRE ATT&CK
● Attack discovery provides the following benefits:
○ Improves analyst time efficiency
○ Helps combat alert fatigue
○ Reduces mean response time

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Attack Discovery
● Attack discovery uses LLM connectors from AI Assistant
● Attack discovery is compatible with many models
○ Models with larger context windows are more effective

Claude 3 Opus Claude 3.5 Claude 3 GPT4o GPT4 Turbo

Sonnet Haiku

Assistant - General Excellent Excellent Excellent Excellent Excellent

Assistant - ES|QL
Great Great Poor Excellent Poor
generation

Assistant - Alert
Excellent Excellent Excellent Excellent Poor
questions

Attack discovery Excellent Excellent Poor Poor Good

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Attack Discovery
● Only alerts generated within past 24 hours are analyzed
● Attack discovery analyzes 20 alerts by default
○ Can be customized up to 100 alerts
● Attack discovery uses AI Assistant settings for configuration

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Attack Discovery
● Attack discovery is found in the Security navigation menu
● Select an existing connector or add a new one from the dropdown menu
● Click Generate to start analysis

Connector Start analysis

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Attack Discovery
● Threats that are identified appear as discoveries
● Discoveries are generated between a few seconds and several minutes
○ Time depends on number of alerts and the LLM selected
● Only alerts that are opened and acknowledged will be analyzed

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Attack Discovery
● Each discovery includes the following information:
○ Descriptive title and summary of the potential threat
○ Number of associated alerts and MITRE ATT&CK matrix relation
○ Users, hosts, and what suspicious activity was observed

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Attack Discovery
● Discovery results can be used in other workflows
● Additional host and user information
● Add host and user details to a Timeline
● Add discovery information to a new or existing case
● Investigate the discovery in a Timeline
● Utilize AI Assistant to ask questions about a discovery or associated
alerts

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Event Query Language
Event Query Language EQL
● Used for event-based time series data and designed for security use
cases
● Does not require a schema, however EQL is designed to work with ECS
by default
● Supports case-insensitive search
● Requires data that contains:
○ timestamp field
■ Uses "@timestamp" from ECS by default
○ event category field
■ Uses "event.category" from ECS by default

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
EQL  Parts of a Query
● An EQL query contains an event.category and a condition
● An event.category is used to define which types of documents to search
○ network - Most often contains data related to IP addresses, ports, and protocols
○ file - Contains information related to the creation, access, and deletion of files
○ intrusion_detection - Relating to intrusion detection logs from IDS systems
● A condition is used to define the criteria that an event must match to return
results
○ Can be defined and combined using a set of operators

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
EQL  Basic Query Format
● The basic format of an EQL query:
○ Search by event category where some condition matches, exists, or is true

<event.category> where <CONDITION>

● Match documents with an event.category of network that contain a field

name "destination.port" with a field value of "80"
network where destination.port == 80

● Match documents with an event.category of file that contain a field name

"file.name" with a field value of "malware.exe"
file where file.name == "malware.exe"

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
EQL  Comparison Operators
● == (equal, case-sensitive)
○ Returns true if the values to the left and right of the operator are equal
file where file.name == "malware.exe"

● ! (not equal, case-sensitive)

○ Returns true if the values to the left and right of the operator are not equal
file where file.name != "notepad.exe"

● : (equal, case-insensitive)
○ Returns true if strings to the left and right of the operator are equal
file where file.name : "PrestoChangeo.exe"

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
EQL  Boolean Operators
● and
○ Returns true if the condition on the left and right are both true
network where network.protocol == "http" and destination.port == 80

● or
○ Returns true if one of the conditions on the left or right is true
network where network.transport == "udp" or network.transport == "tcp"

● not
○ Returns true if the condition to the right is false
network where network.protocol == "dns" and not network.transport == "udp"

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
EQL  Sequences
● Match an ordered series of events
● Each event category and condition is surrounded by square brackets [ ]
● Listed in chronological order with most recent event listed last

sequence
[ event.category where condition ]
[ event.category where condition ]
[ event.category where condition ]

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
 EQL Demo
Follow along with your instructor as they
guide you through
EQL
Elasticsearch Query Language
Elasticsearch Query Language ES|QL
● Flexible searches with the ability to define fields at query time
● Accurate detection rules that help reduce alert fatigue
● Work with summarized data using aggregations in queries
● Providing data enrichment at query time
● Used in Security App Timelines) and Discover

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
ES|QL Syntax
● An ES|QL query is composed of a series of commands chained together by pipes
● Command terms are case insensitive
○ Source commands retrieve or generate data in the form of tables

Processing commands take a table as input and produce a table as output

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
ES|QL Commands
● FROM source command returns a table with data from Elasticsearch
● Each row represents a document
● Each column represents a field

FROM logs-network*

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
ES|QL Commands
● WHERE filters rows based on a specified condition

FROM logs-network*
| WHERE destination.port == 80

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
ES|QL Commands
● KEEP specifies which columns to display in the output table
● The order of the fields listed in the keep command determines the order
in the table

FROM logs-network*
| WHERE destination.port == 80
| KEEP source.ip, source.bytes, destination.ip

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
ES|QL Commands
● STATS .. BY used to group and apply metrics to data
● Creates a column that represents a measurement of a column by some
other column or columns

FROM logs-network*
| WHERE destination.port == 80
| KEEP source.ip, source.bytes, destination.ip
| STATS total_source_bytes = sum(source.bytes) BY source.ip,
destination.ip

● This example creates a column called total_source_bytes that represents

the sum of source.bytes for each source.ip across all documents

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
ES|QL Commands
● SORT used to sort data by a column in Ascending or Descending order
● Nulls are included at the top of the table by default, specify nulls first or
last if needed

FROM logs-network*
| WHERE destination.port == 80
| KEEP source.ip, source.bytes, destination.ip
| STATS total_source_bytes = sum(source.bytes) by source.ip,
destination.ip
| SORT total_source_bytes desc nulls last

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
ES|QL Commands
● LIMIT specifies the maximum number of rows in the output table
● Useful to show top or bottom matches

FROM logs-network*
| WHERE destination.port == 80
| KEEP source.ip, source.bytes, destination.ip
| STATS total_source_bytes = sum(source.bytes) by source.ip,
destination.ip
| SORT total_source_bytes desc nulls last
| LIMIT 10

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
 ES|QL Demo
Follow along with your instructor as they
guide you through
ES|QL
Detection Engine
Detection Engine
● The Detection Engine is at the core of the Security Application
○ Automatically searches for suspicious host and network activity and
generates alerts based on matching events
○ Provides hundreds of prebuilt rules to automate threat detection at a scale
○ Allows to create custom rules with the support of multiple query languages:
KQL, Lucene, EQL, ES|QL
○ Enables analysts to tune each rule and provide exceptions to optimize the
alert generation and reduce the noise

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Prebuilt Rules
● Over 1000 prebuilt rules available
○ Not enabled by default

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
 Anatomy of a Rule
Rule Description

Query

Schedule

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Rule Types
● Custom Query
○ KQL or Lucene to detect issues across indices
● Machine Learning
○ ML jobs to detect anomalous activity
● Threshold
○ Specified fieldʼs value meets a threshold during an execution
● Event Correlation
○ Utilizes EQL to match events and sequences
● Indicator Match
○ Use intelligence source to detect matching events and alerts
● New Terms
○ When a value appears for the first time
● ES|QL
○ Use Elasticsearch Query Language to find events and
aggregate results
© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Custom Query Example
● KQL | Lucene-based
event.action:"Process Create (rule: ProcessCreate)" and
process.name:"vssadmin.exe" and process.args:("delete" and "shadows")

● Searches winlogbeat-* indices for vssadmin.exe executions with the

strings delete AND shadows present
○ Deleting the volume shadow copies
© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Machine Learning
● Pre-built rules
○ Hovering over job gives a description
● User created custom jobs require the use of assigned group “Securityˮ
● User defines the Anomaly score threshold

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Threshold
● Searches defined indices and creates an alert when a defined
number of events occur
● Utilize the GROUP BY and COUNT field to create Boolean logic to
alert on
○ Great for finding bruteforcing attempts Okta example)

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Event Correlation
● Alerts based on EQL queries
○ Assumes the @timestamp is present
○ Uses the event.category field
● Great for complex rules that trigger on multiple events in a sequence
● Allows for highly tuned effective rules

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Event Correlation Rule
● Relies on EQL to match events or
sequences and generates an Alert
○ Select the source indices
○ Specify the EQL query to run
○ Configure the rule settings and
activate it

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Indicator Match
● Creates an alert when Elastic Security index field values match field values
defined in the specified indicator index patterns.
● ECS compliant data provides the best results
○ Can be used for threat intelligence or monitoring critical infrastructure
○ Multiple uses for when a match of two index patterns occur
○ Elasticsearch indices can be created using the Create Index API or Machine Learning Data
Visualizer for CSV | JSON files

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Rule Tuning: Exceptions
● Provide a convenient way of allowing trusted processes and network
activity to function without producing unnecessary noise
○ Available for both detection rules or Agent running Endpoint Security

Detection Rule
Exception

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Rule Tuning: Value Lists
● Define and import custom value lists
○ IP Address, hostnames, etc
● Reuse it to define exceptions

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Watch Your Rule
● Check the execution state of each rules and its related metric
○ If the rule fails, check the Failure History associated with the rule

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Detection Engine Demo
Follow along with your instructor as they
guide you through
Detection Engine
 Lab 7
Security App
Lab 7.3 Detection Rule Types
CTFd flag: “engineˮ
Alerts
Investigate: Alerts
● Alerts page shows all alerts generated by the Detection rules
○ Use this as a starting point for Investigation
● Manage your Alerts and share the workload with your team

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Investigate: Event Visual Analyzer
● Shows the Timeline of events that led up to the Alert generation
● Available for all events detected by the Endpoint Integration or
Winlogbeat Sysmon
● Shows all related event to each process
● Observe child process events that spawned from the parent process

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
 Alerts Demo
Follow along with your instructor as they
guide you through
Alerts
 Lab 7
Security App
Lab 7.4 Alerts
CTFd flag: “signalˮ
Timeline and Cases
Investigate: Timeline
● Workspace to investigate Alerts
● Search with KQL/Lucene, EQL and ES|QL
● Supports Drag & Drop

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Investigate: Cases
● Cases are used to open and track security issues directly in the Elastic
Security App
○ “One pane of glassˮ
○ Built to keep Security Analyst in the place they work
● Comments = Markdown syntax
○ Link saved Timelines and reports/threat intel/ web pages
● Allows to import Lens visualizations, Timelines, hyperlinks

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Investigate: Cases
● Once a case is created you can:
○ Add comments Markdown)
■ Links
■ Threat Intelligence reports
■ Timelines
○ Add Lens Visualizations
○ Edit comments
○ Close and Reopen cases
○ Refresh the case - get new data
● Add connectors
● Send updates to external systems

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Adding Lens Visualizations
● Once a visualization is added to the case, it can only be deleted by
removing the markdown within the comment
○ You can interact with it via the Open Visualization option
○ Set an Absolute Time so the time doesnʼt change after you save it to the case

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Import and Export Cases
● Import cases via a saved object in a NDJSON file
● Export cases to other Kibana instances

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Cases and External Systems
● You can send Cases to external systems using External Connectors
● External Connectors:
○ ServiceNow ITSM
○ ServiceNow SecOps
○ Jira (including Jira Service Desk)
○ IBM Resilient
○ Swimlane
● Refer to documentation for the configuration

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited
Timeline and Cases Demo
Follow along with your instructor as they
guide you through
Timeline and Cases
 Lab 7
Security App
Lab 7.5 Timelines
Lab 7.6 Cases
CTFd flag: “historyˮ
Summary: Security App
● The Security app allows you and your team to detect, investigate,
and respond to threats to secure your operations
● Enable and create rules to detect suspicious activity so Alerts can be
triggered
● Organize your analysis efforts using Timelines and Cases

© Copyright Elasticsearch BV 20152025 Copying, publishing and/or distributing without written permission is strictly prohibited