FortiAnalyzer
SQL and Datasets (supplementary material)
FortiAnalyzer 6.4
© Copyright Fortinet Inc. All rights reserved.
Last Modified:
Wednesday, February 16, 2022
Lesson Overview
Datasets and SQL
SQL Functions and Operators
FortiAnalyzer Functions and
Macros
Datasets and SQL
Datasets
• Datasets are SQL SELECT queries to the database
• Data populates a chart
ADOM specific!
Reports > Report Definitions > Datasets
Dataset (example App-Sessions-By
Category)
4
Designing SQL Queries
• FortiAnalyzer uses SQL as the local database
Test that queries are well-
• Proper query syntax required formed and all keywords are
spelled correctly
SQL queries are
not case-sensitive
5
SQL – The Declarative Language
SELECT dstip as destination_ip, count(*) as Session
FROM $log WHERE $filter and dstip is not null GROUP BY
dstip ORDER BY session desc LIMIT 7
• Declarative language: Describes what needs to be done rather than how to do it
• All information in the database is represented as tables
• Each table consists of a set of rows and columns
• Two types of tables: User tables and System tables
6
Basic Data Manipulation Constructs (DML)
• SELECT
• Retrieve and display data from one This is the only query
or more database tables (read-only query)
• SELECT … FROM … WHERE
statement used by
FortiAnalyzer for reports
• INSERT
• Add new rows of data into a table
• INSERT INTO … VALUES …
• UPDATE
• Modify existing data in a table
• UPDATE … SET … WHERE
• DELETE
• Remove rows of data from a table
• DELETE FROM … WHERE
7
SELECT Statement
• The SELECT statement retrieves the log data you want from the database
• Must specify criteria using a recognized/supported clause
Clauses must be coded in a
specific sequence!
Clause Definition
FROM Selects the table or views.
WHERE Sets the conditions (all rows that do not satisfy the condition are eliminated)
GROUP BY Collects data across multiple records and groups the results by one or more columns.
ORDER BY Orders the results by rows.
LIMIT Limits the number of records returned based on a limit value. OFFSET clause can be used with the
LIMIT clause to offset the results by a set value.
8
SELECT and FROM
• Use the SELECT query to ask specific questions of the database
SELECT column FROM log_type
Column from database that contains The log type under which the data is
the value(s) you want to retrieve contained (ie. Traffic, Web filter, etc.)
• When designing queries for SQL reports on the FortiAnalyzer device, the Log Type
is assigned to a variable called $log
SELECT dstip as destination_ip FROM $log
SELECT *
returns all
data
9
Multiple Log Types
• Search multiple log types
• Combine the data so that you can compare and contrast information
SELECT dstip, hostname FROM $log-traffic, $log-webfilter
Log type syntax Log type
$log-attack Attack log
$log-dlp DLP log
$log-event Event log
$log-netscan NetScan log
$log-app-ctrl Application control log
$log-emailfilter Email filter log
$log-traffic Traffic log
$log-virus Anti-virus log
$log-webfilter Web filter log
10
WHERE
• The WHERE clause requests data with certain characteristics
• The expression specifies a stored value in the database
SELECT column FROM log_type WHERE expression1 and expression2 not in
expression3
Criteria you want to Can use multiple expressions
specify separated by AND/OR/NOT statements
SELECT dstip as destination_ip FROM $log WHERE $filter and dstip is
not null
11
GROUP BY
• GROUP BY statement is usually used in conjunction with aggregate functions to
group data by one or more columns.
• Returns one output row for each group
• Can form groups within groups
• Each item in the SELECT list produces a single value per set
SELECT column, aggregate_function FROM log_type WHERE
expression1 and expression2 not in expression3 GROUP BY column
If GROUP BY is used without aggregates,
it is similar to the DISTINCT clause.
SELECT dstip as destination_ip, count(*) as session FROM $log
WHERE $filter and dstip is not null GROUP BY dstip
12
ORDER BY
• By default, rows of an SQL query result table are not arranged in a particular order
SELECT column, aggregate_function FROM log_type WHERE expression1
and expression2 not in expression3 GROUP BY column ORDER BY
column_name|column_number asc|desc
Can sort data by Can sort data in ascending (asc)
column name or or descending (desc) order. By
column number default, sorts in ascending order.
SELECT dstip as destination_ip, count(*) as session FROM $log WHERE
$filter and dstip is not null GROUP BY dstip ORDER BY session desc
13
LIMIT and OFFSET
• The LIMIT clause limits the number of records retrieved from the query result
• Useful in large deployments to help limit the CPU/memory usage for reports
• Can be combined with ORDER BY asc to get the “top <x> results”
SELECT column, aggregate_function FROM log_type WHERE expression1
and expression2 not in expression3 GROUP BY column ORDER BY
column_name|column_number asc|desc LIMIT number OFFSET number
Specify how many records to return Specify how many records to skip
SELECT dstip as destination_ip, count(*) as session FROM $log WHERE
$filter and dstip is not null GROUP BY dstip ORDER BY session desc
LIMIT 7 OFFSET 1
14
Creating a Dataset in FortiAnalyzer
Reports > Report Definitions > Datasets
15
Analyzing the Dataset Test Results
Reports > Report Definitions > Datasets
16
Lesson Progress
Datasets and SQL
SQL Functions and Operators
FortiAnalyzer Functions and
Macros
SQL Functions and Operators
“Normal” Functions vs. Aggregate Functions
Aggregate functions “Normal” functions
Use the entire column of data as their input and Operate on each element in the column of data
produce a single output
19
NULLIF
• NULLIF function takes two arguments: if the first two arguments are equal, then
NULL is returned. Otherwise, the first argument is returned.
SELECT NULLIF(expression1, expression2)
Must be values that are of the
same datatype
• NULL represents unknown data—it is not equal to zero
20
COALESCE
• Returns the first of its arguments that is not NULL. NULL is returned only if all
arguments are NULL
SELECT coalesce(catdesc,‘unknown’) as category,
coalesce(root_domain(hostname), ‘unknown’) as domain FROM $log
GROUP BY category, domain
21
Aggregate Functions
• Aggregate functions perform a calculation on a set of values in a column and return
a single value
Aggregate functions
AVG(expression) Returns the average value
COUNT(expression) Returns the number of rows
COUNT(*) Returns all rows, even if some columns contain a NULL value
FIRST(expression) Returns the first value
LAST(expression) Returns the last value
MAX(expression) Returns the largest value
MIN(expression) Returns the smallest value
SUM(expression) Returns the sum
22
Operators
• Reserved word or character used primarily in the WHERE clause to perform various
operations
• Arithmetic operators
• Comparison operators
• Logical operators
23
Arithmetic Operators
• Perform mathematical operations on two expressions of one or more of the data
types of the numeric data type category
Operator Description
+ Addition – adds values on either side of the operator
- Subtraction – Subtracts right hand operand from left hand operand
* Multiplication – Multiplies values on either side of the operator
/ Division – Divides left hand operand by right hand operand
% Modulus – Divides left hand operand by right hand operand and returns
remainder
24
Comparison Operators
• Test whether two expressions are the same
• Can be used on all expressions except text, ntext, or image data types
Operator Description
= Equal to
> Greater than
< Less than
>= Greater than or equal to
<= Less than or equal to
<> Not equal to
!= Not equal to (not ISO standard)
!< Not less than (not ISO standard)
!> Not greater than (not ISO standard)
25
Logical Operators
• Test for the truth of some condition
• Return a boolean data type with a value of TRUE, FALSE, or UNKNOWN
Operator Description
ALL TRUE if all of a set of comparisons are TRUE.
AND TRUE if both Boolean expressions are TRUE.
ANY TRUE if any one of a set of comparisons are TRUE.
BETWEEN TRUE if the operand is within a range.
EXISTS TRUE if a subquery contains any rows.
IN TRUE if the operand is equal to one of a list of expressions.
LIKE TRUE if the operand matches a pattern.
NOT Reverses the value of any other Boolean operator.
OR TRUE if either Boolean expression is TRUE.
SOME TRUE if some of a set of comparisons are TRUE.
26
Lesson Progress
Datasets and SQL
SQL Functions and Operators
FortiAnalyzer Functions and
Macros
FortiAnalyzer Functions and Macros
root_domain
• root_domain(hostname)
• Retrieves the root domain of the fully qualified domain name (FQDN)
SELECT devid, root_domain(hostname) as website FROM
$log WHERE‘user’=‘USER01’ GROUP BY devid, hostname
ORDER BY hostname LIMIT 7
29
nullifa
• nullifna(expression)
• Inverse operation of COALESCE
• Can be used to filter out values with N/A and n/a from logs
• SQL syntax SELECT NULLIF(NULLIF(<value>, 'N/A'), 'n/a')
SELECT coalesce(nullifna(‘user’), ‘srcip’) as user_src,
coalesce(nullifna(root_domain(hostname)),’unknown’) as domain FROM
$log WHERE dstport=‘80’ GROUP BY user_src, domain ORDER BY
user_src LIMIT 7
If user is n/a, the source IP is
displayed, otherwise it returns the
user name
30
FortiAnalyzer Functions: email_domain, email_user
• email_domain: Retrieves anything after the @ symbol in an email address
• email_user: Retrieves anything before the @ symbol in an email address
SELECT ‘from’ as source, email_user(‘from’) as e_user,
email_domain(‘from’) as e_domain FROM $log LIMIT 5 OFFSET 10
Source e_user e_domain
[email protected] user11 example.com
[email protected] user12 hostname.com
[email protected] user13 exampleXYZ.com
[email protected] user14 hostnameXYZ.com
[email protected] user15 example.com
31
FortiAnalyzer Functions: from_dtime, from_itime
• from_dtime(bigint): Returns device timestamp without time zone
• from_itime(bigint): Returns FortiAnalyzer’s timestamp without time zone
SELECT itime, from_itime(itime) as faz_local_time, dtime,
from_dtime(dtime) as dev_local_time FROM $log LIMIT 3
32
Macros
• FortiAnalyzer Date/Time macros
Macros PostgreSQL syntax Result
$hour_of_day to_char(from_itime("itime"), 'HH24:00') 18:00
$HOUR_OF_DAY to_char(from_itime("itime"), 'YYYY-MM-DD 2015-01-01 18:00
HH24:00')
$day_of_week to_char(from_itime("itime"), '"WDAY" D-Dy') WDAY 2-Mon
$DAY_OF_WEEK XXX XXX
$day_of_month to_char(from_itime("itime"), 'DD') 01
$DAY_OF_MONTH to_char(from_itime("itime"), 'YYYY-MM-DD') 2015-01-01
$month_of_year to_char(from_itime("itime"), 'YYYY-MM') 2015-01
$MONTH_OF_YEAR XXX XXX
33
Lesson Progress
Datasets and SQL
SQL Functions and Operators
FortiAnalyzer Functions and
Macros
