BIG IP ASM V12
DDOS PROFILE
Global Service Tech Summit, Seattle
Sep, 2015, v3
Lior Rotkovitch, NPI
ASM, L7 DDoS & Analytics
lior@f5.com
ASM DDoS Profile
DDOS - HTTP FLOODS ATTACKS
DDOS - BOTS
From single IP to single URL
Simple bots
From multiple IPs to single fixed URL
Impersonating Bots
From multiple IPs to multiple fixed URLs
Bots with cookies & JS capabilities
From multiple IPs to multiple random URLs
Bots acting as full browser
From multiple IPs from a specific country
Fine Tune your Thresholds & Reporting
F5 Networks, Inc
Reporting
HTTP Floods facts:
Users Or Bots
Legitimate Layer 7 requests
Asking a web page thousands of time instead of one (few) times
Exhausting backend servers resources: memory, CPU, Disk etc
Relatively easy to execute with simple tools
Not easy to detect the offending source nor to prevent it
Wrong identification will prevent valid users from accessing the site (false positive )
User
Source IPs
Unidentified User
Hacktivism
Web Site
Servers
Database
Google Web Bot
Web Bot
F5 Networks, Inc
HTTP Floods types
Users Or Bots
Requests increase from or/and to URLs inside the web site
From single IP to single URL
From multiple IPs to single fixed URL
From multiple IPs to multiple fixed URLs
From multiple IPs to multiple random URLs
From multiple IPs originating from a specific country
User
Source IPs
Unidentified User
Hacktivism
Web Site
Servers
Database
Google Web Bot
Web Bot
F5 Networks, Inc
ASM Detection & Mitigation concept - HTTP Floods
ASM process:
Users Or Bots
1.
Monitoring entities: RPS Latency IPs URLs
2.
Detecting Increase
3.
Activating Mitigation
Source IPs
User
Source IPs
Unidentified User
Hacktivism
RPS
Web Site
Latency
App URLs &
objects
Servers
Database
Google Web Bot
Web Bot
F5 Networks, Inc
ASM Detection & Mitigation concept DoS Profile
Location: Security DoS Protection DoS Profiles dos
F5 Networks, Inc
TPS Based Detection: Transaction Per Second based detection
and mitigation
Client: Hey server, can I get the web page ?
Server
ASM: no, you are sending too many requests.
You will have to
Answer CSID
Answer CAPTCHA
Be Rate Limited / Blocked
F5 Networks, Inc
TPS Based Detection
Monitoring Request Per Second increase form source IP, Geo, URL, Site Wide.
Then apply one of the mitigation policies: CSID, CAPTCHA, Rate limit
F5 Networks, Inc
TPS Based Detection
1.
BY Source IP (Detection & Mitigation Polices)
2.
Mitigation polices:
a)
Client Side Integrity Defense
b)
CAPTCHA challenge
c)
Request Blocking
3.
By Geolocation (Detection & Mitigation Polices)
4.
By URL (Detection & Mitigation Polices)
5.
By Site Wide (Detection & Mitigation Polices)
6.
Prevention Duration
a
2b
c
3
6
F5 Networks, Inc
By Source IP: Detection Criteria
Detection: thresholds for determining DDoS attack - by source IP increase
Mitigation: which mitigation will apply on the offending source IP
Detection
Mitigation
F5 Networks, Inc
10
By Source IP: Detection Criteria
Detection Ratio
Ratio thresholds - measuring ratio with two time interval:
Long (History Interval): Measure the last 1 hour RPS averages every 10 seconds
Short (Detection Interval): Measure the last 10 seconds RPS averages every 10 seconds
F5 Networks, Inc
11
By Source IP: Detection Criteria
Detection Ratio
Example:
Long (History Interval): 50 TPS
Short (Detection Interval): 370 TPS
F5 Networks, Inc
TPS increased by: ((370 - 50) /50)*100 = 640%
640% > 500% = True
12
By Source IP: Detection Criteria
At least X Transactions:
Detection Ratio
A minimum condition to
prevent false positive
increase (source IP
starts browsing the site
and goes from 0 to 30
RPS)
TPS increased by %
Example:
F5 Networks, Inc
640%
AND
minimum fix number of transactions
AND
40 =
True
13
By Source IP: Detection Criteria
Detection Ratio
or
Fixed
TPS reached:
Ratio thresholds
ORed with fixed TPS
(TPS increased by %
Example:
F5 Networks, Inc
640%
AND
minimum fix number of transactions) OR TPS reached
AND
40
OR
200
True
14
TPS Based Detection
1.
BY Source IP (Detection & Mitigation Polices)
2.
Mitigation polices:
a)
Client Side Integrity Defense
b)
CAPTCHA challenge
c)
Request Blocking
3.
By Geolocation (Detection & Mitigation Polices)
4.
By URL (Detection & Mitigation Polices)
5.
By Site Wide (Detection & Mitigation Polices)
6.
Prevention Duration
a
2b
c
3
6
F5 Networks, Inc
15
Client Side Integrity Defense Concept
Client: Hey server, can I get the web page ?
Server
if a browser:
Yes, Im a browser
User
Web Bot
F5 Networks, Inc
If a bot:
*^lkjdfg@#$
ASM: no, you are sending too many
requests. Are you a browser ?
ASM: ok, you are allowed. Here is the web
page you asked for.
ASM: Bye Bye Blocked
16
Client Side Integrity Defense Concept
Checking JavaScript capabilities
A client is considered legitimate if it meets the following criteria:
The client support JavaScript
The client support HTTP cookies
The client should calculate a challenge inside the JS
If satisfied = legitimate client that can access the site
F5 Networks, Inc
17
Client Side Integrity Defense - Flow
User
First main page access
Browser
DoS Profile
App
HTTP Request (no cookie)
Send JS test
Computational challenge
Solve challenge/
set cookie with time stamp
HTTP Request (cookie)
Reconstruct request
Original HTTP Request
HTTP Response (main page)
HTTP Response (main page)
More object requests (cookie)
Validate cookie: format & time stamp
More object requests
Deliver page
F5 Networks, Inc
More responses
More responses
This is the flow and timeline
of events.
Transparent to the user,
done under the hood
Note that request is held at
the ASM and not arriving the
app until checks are
satisfied
Not all checks are described
here, some are internal IP.
18
Client Side Integrity Defense JavaScript sample
The JS is obfuscated
From user perceptive this
is transparent action.
F5 Networks, Inc
19
Client Side Integrity Defense Mitigation summary
If no reply No problem for us
If didnt solve the challenge but still sending request Block (RST)
If did solve the challenge but:
Cookie is wrong format Block (RST)
Time stamp expired Block (RST)
If client access a resource (image) without getting the cookie first Block (RST)
F5 Networks, Inc
20
TPS Based Detection
1.
BY Source IP (Detection & Mitigation Polices)
2.
Mitigation polices:
a)
Client Side Integrity Defense
b)
CAPTCHA challenge
c)
Request Blocking
3.
By Geolocation (Detection & Mitigation Polices)
4.
By URL (Detection & Mitigation Polices)
5.
By Site Wide (Detection & Mitigation Polices)
6.
Prevention Duration
a
2b
c
3
6
F5 Networks, Inc
21
CAPTCHA Challenge - Concept
Client: Hey server, can I get the web page ?
Server
If a user:
OK, I answered
User
If none user:
Ha ? *^lkjdfg@#$
Web Bot
F5 Networks, Inc
ASM: no, you are sending too many
requests. Please answer this CAPTCHA
challenge, show me your human !:
ASM: ok, you are allowed. Here is the web
page you asked for.
ASM: Bye Bye Block him dood !
22
CAPTCHA Challenge
Completely Automated Public Turing test to tell
Computers and Humans Apart
Ultimate solution for identifying human or bot
Send challenge to every IP that reached IP detection criteria thresholds
To CAPTCHA or not to CAPTCHA ?
Some argues that CAPTCHA is not a good usability because an innocent user gets
CAPTCHA and he will not know why. So, remember that a valid user should pass
browser tests. i.e. if a user is blocked (or gets a CAPTCHA) there is a reason and
maybe he is not innocent (infected ?) .
F5 Networks, Inc
23
CAPTCHA customize response
Can be customize to the web site look and feel colors via css
Failure Response page is served if the first attempted fails
F5 Networks, Inc
24
CAPTCHA Challenge - Flow
User
Request mypage.php
Browser
DoS Profile
App
GET /mypage.php (no cookie)
CAPTCHA HTML +JS response
CAPTCHA rendered
Send CAPTCHA
Cookie with time stamp
Solve CAPTCHA
Submit CAPTCHA
solution
GET /mypage.php + CAPTCHA
cookie
mypage.php
rendered
F5 Networks, Inc
HTML of mypage.php
While the system is still in a
state of attack the offending
source will be presented with
another CAPTCHA every 5 min.
Same as CSID, request is held
at the ASM until CAPTCHA is
solved
Verify CAPTCHA solution
Validate cookie
GET /mypage.php
HTML of mypage.php
25
CAPTCHA mitigation summary
If didnt submit the challenge - no request DOSing us
If didnt solve the challenge but still sending us attacks Blocked
If did solve the challenge but:
Cookie is wrong format RST
Time stamp expired 5 min RST
F5 Networks, Inc
26
TPS Based Detection
1.
BY Source IP (Detection & Mitigation Polices)
2.
Mitigation polices:
a)
Client Side Integrity Defense
b)
CAPTCHA challenge
c)
Request Blocking
3.
By Geolocation (Detection & Mitigation Polices)
4.
By URL (Detection & Mitigation Polices)
5.
By Site Wide (Detection & Mitigation Polices)
6.
Prevention Duration
a
2b
c
3
6
F5 Networks, Inc
27
Request Blocking / Rate limit
Client: Hey server, can I get the web page ?
ASM: no, you are sending too many requests.
Im limiting your requests sending rate
While CSID and CAPTCHA try to understand who is the offending source (bots or browsers)
request limiting is indifferent to the identity and limits the offending sources.
F5 Networks, Inc
28
Request Blocking
Request Blocking:
Blocking: block all IPs from the offending source if a source IP reached thresholds I dont
want him on my site at this point
Rate Limit: limit the amount of allowed request from the offending source if reached
thresholds I can sustain only some of the traffic at this point
F5 Networks, Inc
29
Request Blocking Mitigation Summary
Block all blocking all traffic from the offending
source (i.e. I dont want to see any more traffic from
this source)
Rate Limit rate limit the offending source
Rate limit will limit to long (history) TPS rate
Example
If long was 50 TPS And increase in short is 150 TPS
Rate limit to 50 TPS
F5 Networks, Inc
30
TPS based: by source IP Summary
Measuring source IP increase
All source IPs that reached the thresholds will be presented with the enabled mitigation
If still increasing , fall back according to the order in the GUI (switching mitigation)
Client Side Integrity Check
CAPTCHA Challenge
Request Blocking
F5 Networks, Inc
31
TPS Based Detection
1.
BY Source IP (Detection & Mitigation Polices)
2.
Mitigation polices:
a)
Client Side Integrity Defense
b)
CAPTCHA challenge
c)
Request Blocking
3.
By Geolocation (Detection & Mitigation Polices)
4.
By URL (Detection & Mitigation Polices)
5.
By Site Wide (Detection & Mitigation Polices)
6.
Prevention Duration
a
2b
c
3
6
F5 Networks, Inc
32
HTTP Floods Geolocation detection and Mitigation
Users Or Bots
http floods type: From multiple source
IPs originating from a specific country
User
Source IPs
Unidentified User
Hacktivism
Web Site
Servers
Database
Google Web Bot
Web Bot
F5 Networks, Inc
33
Geolocation - Detection
Geolocation Relative to the whole traffic of the site:
500 % request increase of the whole site from a specific country
AND
At least 10 % of the whole site traffic
F5 Networks, Inc
34
Geolocation Mitigation
All clients requests arriving from the specific country will be presented with mitigation:
Client Side Integrity Check
CAPTCHA Challenge
Request Blocking
(note that blocking will block
all users from this country)
F5 Networks, Inc
35
Geolocation Black n White listing
Specifies the countries that the system always blocks
whenever the system is in a state of DDoS detection.
Done regardless of the thresholds set in the DDoS
profile
F5 Networks, Inc
Allows access to the web site regardless of
geolocation detection criteria thresholds only
i.e. other thresholds still apply
36
TPS Based Detection
1.
BY Source IP (Detection & Mitigation Polices)
2.
Mitigation polices:
a)
Client Side Integrity Defense
b)
CAPTCHA challenge
c)
Request Blocking
3.
By Geolocation (Detection & Mitigation Polices)
4.
By URL (Detection & Mitigation Polices)
5.
By Site Wide (Detection & Mitigation Polices)
6.
Prevention Duration
a
2b
c
3
6
F5 Networks, Inc
37
HTTP Floods URL Detection and Mitigation
Measuring requests increase on a URL
Floods types:
Users Or Bots
From multiple IPs to multiple fixed URLs
From multiple IPs to multiple random URLs
Source IPs
User
Source IPs
Unidentified User
RPS
Web Site
App URLs &
objects
Hacktivism
Servers
Database
http://site.com/sell.php
Google Web Bot
Web Bot
F5 Networks, Inc
38
URL Detection Criteria
Collecting RPS on URLs
TPS increase by* AND at least X TPS **
Calculation:
*Ration of long and short
**Minimum TPS thresholds for detection
F5 Networks, Inc
OR TPS reached
39
URL Detection Criteria Mitigation
All clients that access the URL: Client Side Integrity Check
CAPTCHA Challenge
Request Blocking Rate limit (No block all)
F5 Networks, Inc
40
TPS Based Detection
1.
BY Source IP (Detection & Mitigation Polices)
2.
Mitigation polices:
a)
Client Side Integrity Defense
b)
CAPTCHA challenge
c)
Request Blocking
3.
By Geolocation (Detection & Mitigation Polices)
4.
By URL (Detection & Mitigation Polices)
5.
By Site Wide (Detection & Mitigation Polices)
6.
Prevention Duration
a
2b
c
3
6
F5 Networks, Inc
41
HTTP Floods Site Wide Detection and Mitigation
Floods types:
Users Or Bots
From multiple IPs to multiple random URLs
Cases where DDoS attack is under the radar
Source IPs
User
Source IPs
Unidentified User
RPS
Web Site
App URLs &
objects
Hacktivism
Servers
Database
Monitoring: all entities
Google Web Bot
Web Bot
F5 Networks, Inc
42
Site-Wide Detection Criteria
Collecting RPS on the entire website (all entities URLs, IPs)
In some cases the floods will avoid thresholds for IP based or URL based.
Site wide provide another layer of detection and prevention
Detection:
Ratio
Fixed
*TPS increase by AND Minimum TPS thresholds for detection
OR TPS reached
F5 Networks, Inc
43
Site-Wide Detection Criteria Mitigation
Prevention polices
All clients that access the site:
F5 Networks, Inc
Client Side Integrity Check
CAPTCHA Challenge
Request Blocking - only rate limit no blocking
44
TPS Based Detection
1.
BY Source IP (Detection & Mitigation Polices)
2.
Mitigation polices:
a)
Client Side Integrity Defense
b)
CAPTCHA challenge
c)
Request Blocking
3.
By Geolocation (Detection & Mitigation Polices)
4.
By URL (Detection & Mitigation Polices)
5.
By Site Wide (Detection & Mitigation Polices)
6.
Prevention Duration
a
2b
c
3
6
F5 Networks, Inc
45
Prevention duration
Escalate top down every 120 second if thresholds are still increasing
Client Side Integrity Check
CAPTCHA Challenge
Request Blocking
F5 Networks, Inc
De escalate - start from the top
Every 7200 seconds
46
Stress Based detection
Predictive Latency predict how long it will take to
serve a new incoming request
ASM: Hey server, how many more requests
can you handle ?
Im the server
Server: Im fine, keep on sending them
F5 Networks, Inc
47
Stress Based Detection and prevention concept
Client: Hey server , can I get the web page ?
Im the server
.
.
After a while
ASM: mmm let me check. The Server can take
additional incoming requests. you are allowed
Client: Hey server, can I get web pages again now ?
ASM: no, my backend latency is now too high
and you are sending too many request.
You will have to:
Answer CSID or
Answer CAPTCHA or
Be rate limit
F5 Networks, Inc
48
Stress Based GUI
Same concept as TPS
based: source IP, Geo, URL,
Site wide and their
mitigation policies.
Addition condition of
backend latency. i.e. only
when the two conditions
reach thresholds, then apply
mitigation policy.
Note: Can work together (operate in
parallel) with TPS based and act as
layers of protection (e.g. TPS based
does only CSID in alert mode and
Stress based does request blocking in
case of latency increase)
F5 Networks, Inc
49
Stress Based Detection & Mitigation
Similar to TPS based,
Quiz yourself, what each item
means ?
1. By Source IP
a) CSID
b) CAPTCHA
c) Request Blocking
2. By Geolocation
3. By URL
4. Site Wide
F5 Networks, Inc
50
Stress Based Detection thresholds condition
TPS
Latency
threshold AND threshold
exceeded?
exceeded?
Mitigation Is activated when
two types of thresholds are
reached :
Latency thresholds
AND
Then:
Activate
Mitigation Policy
F5 Networks, Inc
TPS thresholds
51
Stress Based Detection thresholds condition
TPS thresholds
stress detection
prevention
in order to apply a prevention policy, both TPS and
Latency thresholds must be exceeded, then the enabled
prevention policy is activated.
Latency thresholds are not visible in the GUI, they are part
of automatic detection.
F5 Networks, Inc
Example:
Automatic stress detection enters a
state of exceeding thresholds. This by
itself will not active the prevention.
Only when the TPS thresholds will
exceed then the prevention policy is
activated.
52
TPS based
VS
Quick way to protect against DDoS. Im in trouble
and I want to block now !
Fixed number on the TPS reached is very easy
and useful. Also easy to detect offending sources
Stress based
Allows the option to activate the Mitigation only
when the backed experiencing latency AND RPS
increase (I only want to block when the attack is
causing backend latency)
Provide Layers of defense and notify about backend
issues (not just DDoS)
Conclusion:
TPS based is quick while latency based
allows more granular approach
F5 Networks, Inc
53
Heavy URLs
Not all URLs are equal
Some are more attractive than others
F5 Networks, Inc
54
Heavy URLs
Heavy URLs are URLs that consume more processing resources
from the server
Are good application DoS point - Even few requests can DoS the app
Typical heavy URLs are search box, product IDs
http://site.com/serach.php?q=a
Heavy URL
Servers
Database
Ho wow, this will take a while
Searching
..
hold on
..
Almost there.
F5 Networks, Inc
55
Heavy URLs concept
Automatically measures latency on URLs for 48 hours
and decide who is heavy
When any URL based mitigation is active, the heavy URLs
that was detected as heavy will also get the active
mitigation
F5 Networks, Inc
56
Heavy URLs concept
Heavy URL is another detection capability. Once it is reached the thresholds AND one of the By URL detection
thresholds are reached Then the URLs that consider heavy URLs will be applied with the active mitigation policy
F5 Networks, Inc
57
Heavy URLs configuration
Example: By URL TPS reached 1000 TPS and is currently applying CSID mitigation.
Heavy URL is enabled
F5 Networks, Inc
58
Heavy URLs Configuration
1
1. Automatic Detection - Automatically add URL
that will be considered as heavy
2. Manual Heavy URLs Manually add URL that
will be considered as heavy
3. Ignored URL Exclude those URL from
heaviness
4. Latency Threshold Above this threshold ->
heavy URL
F5 Networks, Inc
59
Heavy URL Reporting
Security Reporting DoS Application URL Latencies
Example:
If search.php is defined as heavy and if index URL is currently being mitigated with CSID because it exceeded the
thresholds of URL reached (or fixed) then every source IP that is accessing search.php will also get the CSID check.
F5 Networks, Inc
60
Reporting first
Remember, security is a process
Fine Tune your thresholds
Before DDoS Attack
During DDoS Attack
F5 Networks, Inc
61
First rule of detection - AVR Reporting
Know your web site metrics
AVR graphs help you understand the site metrics:
Statistics Analytics Overview
Statistics Analytics Transactions View by
AVR graphs inside ASM tab:
Security Reporting DoS Overview
Security Reporting DoS Application Transaction outcome
F5 Networks, Inc
Sources
IPs
URLs
Site Wide
Geolocation
RPS
TPS
Latency
L7 DDoS measurements
62
Why Fine Tune Thresholds ?
Out of the box thresholds are good for most web sites
Depending on the web site traffic fine tuning thresholds might be needed.
Fine tuning thresholds can be divided into:
Before DDoS Attack
During DDoS Attack
Good for me ??
F5 Networks, Inc
63
Fine Tune Thresholds Before attack
Process:
Pre requisite: Enable DDoS Profile on the desired virtual
1) White list IPs, geolocations countries, URLs (admin) etc
2) Get visibility with transparent mode write down metrics*
3) Test and decide which prevention will apply thresholds exceeds (TPS bases/
Latency based , heavy URL config etc)
4) Fine tune thresholds for fixed and ratio based
5) Switch to blocking When needed
*good list for L7 DDoS metrics
F5 Networks, Inc
64
Fine Tune Thresholds Before DDoS for Source IP
Go to Statistics Analytics HTTP Transaction
View by: Client IP address
List top TPS Avg IPs
F5 Networks, Inc
65
Fine Tune Thresholds Before DDoS for Source IP
By examining the client IP Address you can
conclude the averages of normal traffic you
expect to see from the top source IPs.
Knowing normal averages can help defining
the TPS increase by ratio.
The idea is that you can determine how much
traffic is allowed until assumed a ddos attack.
F5 Networks, Inc
66
Fine Tune Thresholds Before DDoS for Geolocation
Go to Security Reporting dos Application transaction outcome
The same concept works for the
geolocation thresholds graph.
From the drilldown choose Countries on
AVR reports
Which countries you expect to see traffic ?
F5 Networks, Inc
67
Fine Tune Thresholds Before DDoS for URL
The same idea applies to URLs.
Sort graph by URLs
Which URL should have to highest RPS ?
F5 Networks, Inc
68
Fine Tune Thresholds Before DDoS for URL
F5 Networks, Inc
69
Fine Tune Thresholds Before DDoS for Site Wide
On the drilldown choose
Virtual Server
This will help us understand
the over all traffic load that we
have when there is no DDoS
attack.
F5 Networks, Inc
70
Fine Tune Thresholds Before DDoS for Site Wide
Site wide = Virtual server
The overall traffic should be much higher than the other thresholds.
The values reflect the total amount of TPS that the virtual can handle.
F5 Networks, Inc
71
Fine Tune Thresholds During attack
Process:
1) Fine tune white list source if needed
2) Identify sources that exceed thresholds (source IPs, URLs, Geo, SiteWide) by
looking at reporting.
3) Determine the attack type: from fixed/random source IP to fixed/ random URL.
Conclude which of the detection types you need (source IP only ? Source IP and URL
based only ? etc. )
4) Fine tune thresholds according to the exceeding sources (ratio / fixed)
5) Apply mitigation and decide what is working and what is not. Uncheck the
mitigations that are not effective
6) Go to step 1 and repeat
F5 Networks, Inc
72
Fine Tune Thresholds During attack Source IP
Security Reporting dos
Application transaction
outcome
On the drilldown choose
Client IP Address
F5 Networks, Inc
73
Fine Tune Thresholds During attack Geolocation
Security Reporting dos
Application transaction
outcome
On the drilldown choose
Countries
F5 Networks, Inc
74
Fine Tune Thresholds During attack URLs
Security Reporting dos
Application transaction
outcome
On the drilldown choose URLs
F5 Networks, Inc
75
Fine Tune Thresholds During attack - Site Wide
Security Reporting dos
Application transaction
outcome
On the drilldown choose
Virtual Servers
F5 Networks, Inc
76
AVR reports and graphs
Security Event Logs DoS Application Events
Host IP
Time line
F5 Networks, Inc
Attack start / end
Mitigation type can help
understand which of the
mitigation is effective and
when switching mitigation
occurred
Number
of TPS
Attack ID : Clicking
will show graph
77
AVR reports and graphs
Security Reporting DoS Application Transaction Outcomes
High, medium and low
impact allow to filter
high impact attacks
and deal with them
first
impact is the latency
on the backend for all
entities. The higher
the latency the higher
the impact is.
F5 Networks, Inc
78
AVR reports and graphs
Security Reporting DoS Application Transaction Outcomes
Start and End points - red flags indicate the start of an attack and the green flags indicate the end of
an attack. Switching mitigation can occure several time over the DDoS attack.
F5 Networks, Inc
79
AVR reports and graphs
Incomplete Indicates traffic that was dropped by the server because the connection was
incomplete or the server did not respond.
Blocked Indicates traffic that was blocked as a result of the mitigation policy (any of the
prevention policies including bots blocking)
Proactive Mitigation Indicates the amount of time that the proactive bot defense mechanism
was severed
CAPTCHA mitigation Indicates the amount of time that the CAPTCHA challenge was severed
to offending sources
CS integrity mitigation Indicates the amount of time that the client-side integrity defense
challenge was severed to offending sources
BIG IP Response Indicates traffic that is a response to the client from the BIG-IP system.
Cache by BIG IP Indicates traffic that is served from cache configured (WA, RamCache)
Whitelisted Indicates traffic from IP Address that are in the whitelist of the DoS profile
Pass through - Indicates traffic that is pass to the application trough ASM to the server
F5 Networks, Inc
80
AVR reports and graphs
The AVR DoS graph now
shows the thresholds
that are set in the TPS
detection tab.
The Display Thresholds
check box will display
them or clear them from
the graph.
F5 Networks, Inc
81
Fine Tune Thresholds Summary
Before DDoS:
Write the normal thresholds for the web site: (IPs, Geolocation, URLs, Site Wide)
Set the ratio and the fix threshold for each of the above detection criteria (How much the web
site can take 2 times the traffic , 5 times etc)
Test the configuration and the prevention policy, then conclude which one is good for you
During DDoS:
Identify the source IPs, URLs and entire site traffic increase and determine the attack type
Set the fixed TPS number in each of the above criteria and apply mitigation
Verify the results in the Transaction outcome graph
F5 Networks, Inc
82
DDoS Bots - Detection & Mitigation
F5 Networks, Inc
83
Layers of defense against Bots
Simple Bots
Gohogle
Impersonating Bots
Bots with cookies / JS capabilities
This bot section is mostly about
bots that DoS / DDoS. However,
Bots detection and prevention
can be used for various bot
problems the site is experiencing.
Bots acting as full browser
F5 Networks, Inc
84
DDoS Bots
Users Or Bots
Google Web Bot
Bots can be classify in many ways, mostly there are:
1. Simple bots
2. Impersonating Bots
3. Bots with cookies & JS capabilities
4. Bots acting as full browser
Web Site
Web Bot
Unidentified
User
Servers
Database
User
F5 Networks, Inc
85
Enabling Bot signatures protection
F5 Networks, Inc
86
Bots Simple Bot
Simple bot can be any command line tool such as:
curl , wget , ab
Im a simple Bot
Server
ASM: yes, I have your signature. Sorry
mate you are blocked.
F5 Networks, Inc
87
Categorizing Bots
Bad Bots aka Malicious
are well know command
line tools we want
them out
Good Bots aka Benign are
well know search engine and
monitoring tools we want
them in
F5 Networks, Inc
Bad Bots
Good Bots
88
Bot Signatures -
Each categories include:
None ignore
Report report only used for monitoring
Block block
None
Report
Block
None
Report
Block
F5 Networks, Inc
89
Excluding specific bot signatures from category settings
Specific signature can be excluded from the
category setting
Search the signature in Available signature list
and move it to the left pane.
In this example ab tool will not be blocked
even if the category that includes it is in
blocking mode
F5 Networks, Inc
90
First - White list good Bots
3
DNS Server
Google
1
Im a google Bot
Web Server
2
ASM: lets see if you really are. Im doing
Reverse DNS lookup.
Google
Thanks
F5 Networks, Inc
Yes, I see that, please continue.
91
White list good Bots with their domain name
3. The signature includes domain name.
ASM issue Reverse DNS query to verify
the origin of the request
1. Request arrive with User
Agent : Googlebot/2.1
2. ASM search the
google bot signature
4. Once approved ASM will allow the
google bot to access the web site
User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
F5 Networks, Inc
DNS Server
Web Server
92
Bot Signature Repository
Security Options DoS Protection Bot Signatures List
Bot Signature repository for
the entire system is under
Options.
Bot signatures repository is
update with the ASM
signature update
F5 Networks, Inc
93
Bot Signature List: general signatures repository
Signatures can be sort by:
Signature Category
Signature Type:
Malicious / Benign
User Define signatures
Yes / No
Partition: signature can
be assign to a specific
partition
Clicking on any of the sorting
will change the order.
F5 Networks, Inc
94
Sorting the Bot Signature Repository
Various filtering
Create new Bot Signature
F5 Networks, Inc
95
Bot Signature Categories
Creating new category
for Malicious or Benign
F5 Networks, Inc
96
Create a new bot signature: simple edit mode
Bot Signature name
Domain name execute
reverser DNS query to
verify origin. Add the
domain if the Bot has one
Category
Simple edit mode: inside
a user agent header or
in a URL.
Create when done
F5 Networks, Inc
97
Create a new bot signature - advanced edit mode
Advanced Edit Mode - rule granularity
For full details consult F5 document
Signature syntax example:
headercontent: "sample_text"; useragentonly;
F5 Networks, Inc
98
Bot signature facts
Signatures associated with a domain name are
validated with reverse DNS lookup.
Blocking and reporting :
Block flag - resets the connection and reports the
action as "bot signature block" with the bot
signature name.
Report flag - Report bot name and categories
(AVR)
Updating bot signature as part of the ASM
signature update
F5 Networks, Inc
99
Bots Impersonating Bot
Gohogle
DNS Server
Im a google Bot
ha ha ha
Gohogle
Bummer
F5 Networks, Inc
Web Server
ASM: lets see if you are. Im doing Reverse
DNS lookup.
ASM: Hey DNS, whos this guy ?
DNS: no one important
ASM: you are not google bot
Bye Bye -> block this creature !
100
Bots Impersonating Bot
3 The real google bot includes domain
name. ASM issue Reverse DNS query to
verify the origin of the request
1 Request arrive with User
Agent : Googlebot/2.1
2 ASM search the
google bot signature
DNS Server
4 If the source IP is not the expected one
according to the DNS query ASM will block
the impersonating bot
Web Server
F5 Networks, Inc
101
Bots with cookies & JS capability
Im a bot that can
understand JS and
support cookies
Ha ?
Web Servers
ASM: prove it, answer my challenges
No you are not, bye bye -> block this bot.
Bummer
F5 Networks, Inc
102
Proactive Bot Defense
PBD is good for:
Bots that can handle JS
Bots that can handle JS and cookies
Bots floods
Under the radar bots
Block any bot accessing the site (humans only web site)
F5 Networks, Inc
103
Proactive Bot Defense and Bot Signature
Proactive Bot
Defense is now
integrated with
the bot
signatures.
When enabling
proactive bot
defense the
bot signature
feature will be
enabled as
well
F5 Networks, Inc
104
Proactive Bot Defense
Send Client side challenges to ALL client and thus mitigate bots all the time
Various challenges are sent and then validate by PBD blocked or allowed
F5 Networks, Inc
105
PBD - Client side integrity defense - flow
User
First main page access
Browser
DoS Profile
App
HTTP Request (no cookie)
Computational challenge
Solve challenge/
set cookie with time stamp
HTTP Request (cookie)
Reconstruct request
Original HTTP Request
HTTP Response (main page)
HTTP Response (main page)
More object requests (cookie)
Validate cookie: format & time stamp
More object requests
Deliver page
F5 Networks, Inc
More responses
More responses
106
Proactive Bot Defense configuration
Always sending CS all the time
During attack only if other component of the dos profile is in dos mode PBD
will send the CS challenge (acting as two layers of mitigation)
This allows second layer of protection (rate limit and PBD)
Grace period - cookie expiration time 300 = 5min
White list exclude PBD on those IPs
F5 Networks, Inc
107
Bots acting as full browser
DNS Server
Im a Bot that
simulate browser
Web Server
ASM: ok, what are your capability ? If you will not
answer right you will have to answer a CAPTCHA
Capability ?
CAPTCHA ?
You are not human, byyyye -> block this unhuman !
Bummer
F5 Networks, Inc
108
PBD Additional bots identification with capabilities script
Bots: Bots acting as full browsers - Browser Simulation
F5 Networks, Inc
109
How bots that simulate
browsers are evaluated ?
Block Suspicious Browsers addition tests are done to understand if this is a bot or a browser. ASM
will evaluate the source and will give it a score:
if the score indicates that the source is a bot it will block it.
If the score indicate uncertainty and if CAPTCHA challenge is checked, then CAPTCHA will be presented
to the source. If answer it is a human if not, blocked.
F5 Networks, Inc
110
Block Suspicious Browsers
If Block Suspicious Browsers is unchecked send CS Challenge
If Block Suspicious Browsers is checked and CAPTCHA is checked send Client
Capabilities challenge and give it a score: If score in doubt send a CAPTCHA for
human verification
If Block Suspicious Browsers is checked but CAPTCHA Challenge is unchecked do
not send CAPTCHA and only block if the score is more than a human
F5 Networks, Inc
111
Client Capabilities -challenge script flow
User
First request GET /sell.php
Browser
DoS Profile
App
GET /sell.php (no cookie)
Client Capabilities Challenge response
Return Client Capabilities
verification
1.
2.
3.
Authenticate and decrypted JS results,
Verify capabilities and set a score
Determine an action based on score
Blank page & Set cookie
Original HTTP Request + cookie
Reconstruct request
HTTP Response (cookie)
HTTP Response
GET /img.png (cookie)
Validate cookie: format
& time stamp
GET /img.png (cookie)
F5 Networks, Inc
112
DoS Bots Reporting
F5 Networks, Inc
113
Bot signatures simulation
Reporting dos Application Transaction outcomes
Transaction outcomes
is very useful for
monitoring traffic and
indicates various
measurements
F5 Networks, Inc
114
Bot signatures simulation
Analytics HTTP throughput request throughput
AVR will provide details on DoS bot signatures (use drill downs )
F5 Networks, Inc
115
Summary
Simple bots can easily be detected and blocked
White listing of bots = visibility to bot access and keep other bots out
Impersonating bots can be monitored / blocked
Bots that support JavaScript and cookies can now be noticed and be blocked
Reporting on the visiting bots to your web site is available via AVR
Custom bots signature is powerful tool to deal with bots
Bots signature is updating via the ASM signatures update
F5 Networks, Inc
116
Resources
Our documentation is free for all. Read and learn more:
BIG-IP Application Security Manager Operations Guide
BIG-IP Application Security Manager: Getting Started
BIG-IP Application Security Manager: Implementations
BIG-IP Application Security Manager: Custom Signature Reference
BIG-IP Analytics: Implementations
F5 Networks, Inc
117
