How To Protect Your Website From Bot Attacks

e-Commerce Bot Attacks!
How To Protect Your Website From Price Scraping
Section 1
What is a Botnet, Purposes and a CaseStudy
What is a Botnet?
Botnets are networks made up of remote-controlled computers, or “bots.” These computers have been infected with
malware that allows them to be remotely controlled. Some botnets consist of hundreds of thousands — or even
millions — of computers.
If your computer is part of a botnet, it’s infected with a type of
malware. The bot contacts a remote server — or just gets into
contact with other nearby bots — and waits for instructions
from whoever is controlling the botnet. This allows an attacker
to control a large number of computers for malicious purposes.

Section 1
Purposes of a Botnet
Botnets can be used for many different purposes, including: distributed denial-of-service (DDoS) attack on a web
server, sending spam emails, “click fraud” and even mining Bitcoins.
Botnets can also just be used to distribute other malware — the bot software essentially functions as a Trojan,
downloading other nasty stuff onto your computer after it gets in. The people in charge of a botnet might direct the
computers to download additional malware, such as keyloggers, adware, and even nasty ransomware like
CryptoLocker.

Section 1
Case Study: The ZeroAccess botnet
The ZeroAccess botnet is one of the largest known botnets in existence today with a population upwards of 1.9
million computers, on any given day.
A key feature of the ZeroAccess botnet is its use of a peer-to-peer (P2P) command-and-control (C&C)
communications architecture, which gives the botnet a high degree of availability and redundancy.
Since no central C&C server exists, you cannot simply disable a set of attack servers to neuter the botnet. Whenever
a computer becomes infected with ZeroAccess, it first reaches out to a number of its peers to exchange details about
other peers in its known P2P network.
This way, bots become aware of other peers and can propagate instructions and files throughout the network quickly
and efficiently. In the ZeroAccess botnet, there is constant communication between peers. Each peer continuously
connects with other peers to exchange peer lists and check for updated files, making it highly resistant to any
take-down attempts.

Section 1
ZeroAccess: the courier service
Given its construction and behavior, ZeroAccess appears to be primarily
designed to deliver payloads to infected computers.
In a ZeroAccess botnet, the productive activity (from an attacker’s point of
view) is performed by the payloads downloaded to compromised
computers, which boil down to two basic types, both aimed at revenue
generating activities.

Section 2
Click fraud and Bitcoin mining
Click fraud
One type of payload we’ve seen is the click fraud Trojan.
The Trojan downloads online advertisements onto the computer and then
generates artificial clicks on the ads as if they were generated by legitimate
users.
These false clicks count for pay-outs in pay-per-click (PPC) affiliate schemes.

Section 2
Click fraud and Bitcoin mining
Bitcoin mining
The virtual currency holds a number of attractions for cybercriminals.
The way each bitcoin comes into existence is based on the carrying out of
mathematical operations known as “mining” on computing hardware.
This activity has a direct value to the botmaster and a cost to unsuspecting victims.

Section 3
Bot Detection Reality Check
Today, bots have become the scourge of the Web. Making up nearly 50% of site visitor traffic, they inflict damage on
a broad range of online businesses, from data providers and publishers, to eCommerce and travel sites.*
In fact, a recent cross-industry study of 900 organizations in 62 countries found that 63% of organizations were in-
fected by at least one bot. Most were infected by a variety of bots.**
As the problem of malicious bots has intensified, a growing number of hardware and software vendors have thrown
their hats into the bot protection arena. However, most solutions that advertise bot protection were never designed
specifically for bot mitigation. And as a result, they lack sophisticated and/ or comprehensive capabilities for bot de-
tection.
Consulting and services firms are attempting to tap into the growing demand for bot defense solutions arising from
inadequate bot detection capabilities found in web application firewalls (WAF) and other products.

Section 3
Bot Detection Reality Check continued
Unfortunately, human monitoring by services firms works similarly to in-house attempts to spot and stop bots. They
are simply too resource intensive and slow at detecting the latest dynamic bot threats. Website owners end up in-
vesting heavily in a high-priced game of Whack-a-Mole that misses some bots and does little to solve the problem
permanently.
While web application firewall (WAFs) vendors gained some initial sales by positioning their products with “bot pro-
tection included” website owners have come to realize that WAFs lack proactive defenses against bots.
WAFs are designed to wait until a website visitor surpasses a rate limit or performs devious actions before triggering
a response.
This reactive stance lets damage occur before action is taken. By the time site owners recognize they have an issue,
bots have made off with significant amounts of data and caused other irreparable harm at the site’s expense.
Add-on products and modules are even less effective than in-house and service-based solutions. Add-on functionality
to firewalls, load balancers, and other products fail to detect bots at a high rate, because the solutions operate with
a lack of agility.

Section 3
Bot Detection Reality Check continued
The people who develop and launch bots continuously change their bot-cloaking techniques to avoid detection.
Therefore, any solution that cannot adjust to new bots in real time will fail to deliver acceptable levels of security.
The Web is littered with ‘How to’ blog posts and ads from obscure website data scraping vendors that demonstrate
how quickly bot-protection solutions become obsolete. A simple Google search yields multiple results with details of
ways to circumvent “detection by IP address” and “detection by request signatures” - both major elements of com-
mon anti-bot security products.
At the same time, product vendors continue to tout these detection methods as if they represent the leading edge in
the bot defense industry. This disconnect leads many website owners to have a false sense of security when it comes
to preventing bots on their sites.

Section 4
Existing Solutions
Existing Solutions: Automation over Consultation:
Completely Automated Public Turing Tests to Tell Computers and Humans Apart
(CAPTCHAs) exist to ensure that computers do not generate user input.
CAPTCHAs are simple to integrate and were highly effective for years, making
them the most common bot defense.
However, in recent years, bots have evolved to easily beat CAPTCHAs. Business
line executives also dislike the loss in customer conversions that result from CAPTCHAs.
CAPTCHA’s

Section 4
Existing Solutions
Rate Limiting
Advanced scraping utilities mimic normal browsing behavior, but most hastily written scripts are not.
Bots will follow links and make web requests at a much higher and consistent rate than normal human
users.
Limiting IPs that make several requests per second would allow a company to catch basic bot behavior
in an automated manner. However, all but the most basic bots go undetected and often users sharing
an IP will trigger a false positive.

Section 4
Existing Solutions
IP Blacklists
Subscribing to lists of known botnets and anonymous proxies, then uploading them to a firewall access control list
provides a baseline of protection. Yet, many scrapers employ botnets and Tor nodes to hide their true location and
identity.
Moreover, bots are often deployed across residential IP ranges and blacklisting those IPs results in legitimate traffic
being blocked.
Blacklist...

Section 4
Existing Solutions
Honeypots
Bots and scraping tools generally follow links blindly, so online data
companies sometimes deploy hidden links leading to a dead page.
This helps to identify simple scraping scripts, but commercial scraping
tools can detect honeypots quite easily. This method stops the amateur
scrapers, but not the pros.

Section 4
Existing Solutions
Hardware and Add-on Modules
Many companies already own hardware that offers some layer of site security. These were never designed to fend
off bots that scrape data, so vendors are offering add-on modules to fill the defense gap. These bolt on solutions
catch only the simplest of bots and leave a large gap in protection.
Those who deploy bots have benefitted greatly from website owners having varied and uncoordinated responses to
bot threats. With each site using its own bot detection and prevention methodology (or none at all), a bot can simply
move from one site to the next and eventually obtain the information or inflict the damage for which it was
developed.
Countless posts to the pro-bot blog http://websitescraper.blogspot.com/ point out that most websites only use
stagnant or basic bot detection techniques, leaving them vulnerable to simple bot cloaking.

Section 4
Existing Solutions
SaaS-based bot detection
To have an effective solution, online companies need a dynamic system that constantly adapts to new bot types and
their associated cloaking techniques. The only realistic way to cost effectively achieve this on an ongoing basis is
through SaaS-based bot detection.
The effort required to identify, track and manage the millions of current and future bots is simply too
resource-intensive and cost-prohibitive for most websites to justify. Yet, this is the ideal scenario for which SaaS
exists.
In fact, the bot defense industry mirrors the circumstances that gave rise to the anti-virus solution market. SaaS-
based solutions have clearly proven most effective, as they are doing now in the bot detection and mitigation indus-
try.
A SaaS model of bot detection provides shared knowledge and spreads costs across a community of sites. This
provides each site with the most protection at the lowest possible cost. Additionally, SaaS-based solutions can
continuously update customer sites with new information and functionality to stay ahead of evolving bot threats.

Section 4
Existing Solutions
The Need for Bot Fingerprinting
To effectively fend off bots, website owners must use defenses that evolve as quickly as the
changing bot threats and their associated cloaking schemes.
This means incorporating bot detection techniques that far surpass those of WAFs and
packaged products.
Moreover, to maintain cost efficiency, the human element must be removed from the
detection part of the equation. Forward-thinking website managers that want automated
and comprehensive bot detection are using fingerprinting techniques to stop bots.
As in the offline world, fingerprints are unique to each site visitor. First put to use in the
multi-billion-dollar online advertising industry, fingerprinting technology helps defend
against bots by identifying repeat offenders despite cloaking techniques like changing their
IPs.
Fingerprints are based on multiple metrics, such as connection properties, header values and website/data request
behavior.

Section 5
SaaS Solutions
SaaS Providers Bring More Advanced Detection Technologies
An advanced SaaS provider will feature the latest technologies for detecting and remediating bots, keeping websites
in the community protected from newer bot tactics. For example, fingerprinting of bots, instead of relying on IP’s, is
the latest innovation that SaaS providers leverage for detecting and tracking malicious attackers.
Having fingerprint information enables the SaaS provider to build a database of bot fingerprints that includes mul-
tiple types of associated data needed to track bots and bot networks as they attempt to shift tactics and launch loca-
tions. As noted earlier, while IP address data can help with basic bots, fingerprint data proves far more valuable by
detecting all bots.

Section 5
SaaS Solutions
SaaS Providers Bring More Advanced Detection Technologies
continued
Website owners need to be careful when evaluating the capabilities of a bot protection provider. The question on
each website manager’s mind when considering a vendor is, “What is your level of bot fingerprint data, and how do
you keep it updated in real time?”
A purpose-built solution specific to bots must have at least 33 different properties that make up a bot fingerprint in
order to uniquely identify all bots before they enter a site.
When combined with machine learning algorithms, each website visitor’s session is automatically analyzed for
behavioral anomalies, making it extremely difficult for a bot to go undetected. The SaaS provider adds the
fingerprints of detected bots to its centralized tracking database, then instantly shares the data across the
community of customer sites.

Section 5
SaaS Solutions
SaaS Offers Substantial Cost Savings and Resource Efficiency
Another major advantage of SaaS-based bot protection comes from shifting
costs from the individual website owners to the SaaS provider.
Websites drastically reduce costs for personnel, consulting services and
infrastructure while vastly improving their level of bot detection.

Section 5
SaaS Solutions
SaaS-Based Bot Mitigation
SaaS also removes the risk of vendor lock-in. Once a company has purchased software or hardware for their site, they
are committed, no matter how painful the costs of initial outlay, integration and product upgrades. SaaS minimizes
risk, because it requires no infrastructure changes and reduces website infrastructure and support costs in three
specific categories:
Initial costs:
Monthly subscription fees replace substantial initial cash outlay for products and services.
Upgrades:
The SaaS provider manages all updates and upgrades to the bot remediation service, so customers have no need to
install patches. The SaaS provider also manages availability of the system.
Integrations:
SaaS-based bot protection integrates with virtually no coding or software integrations. This holds true for
deployments in both private cloud and public cloud formats.

Section 5
SaaS Solutions
SaaS is Faster with ‘Instant-On’ Bot Protection
Since they face no requirements for modifying infrastructure or integrating solutions, Websites
that join a SaaS-based community of bot protection gain instant access to a wealth of bot
intelligence that has been gathered on behalf of other member sites in the community. In this
way, newer customers in the community achieve higher return on their investment from the
outset. There is little need to justify and calculate long-term ROI, as there are no significant
product and service investments.
Unlike the traditional argument by product vendors that their solutions overtake SaaS in ROI after several years, that
is not the case when it comes to bot protection. The sharing of intelligence that occurs across a bot protection
community consistently outpaces the intelligence that one website can gather on its own.
As such, the business benefits remain higher for SaaS solutions in perpetuity.

Section 5
SaaS Solutions
CDN Capabilities Seal the Deal for SaaS Providers
Many websites do not yet use content delivery network (CDN) services, which are often
viewed as prohibitively expensive.
Thus, for many small- and medium-sized businesses, SaaS-based bot protection can
bring the added benefit of CDN capabilities without the added high cost. Advanced SaaS
providers have established their own CDN capabilities, both for their bot intelligence
gathering and customer website performance enhancement, allowing them to provide
CDN capabilities at very reasonable rates.
Companies on the cusp of needing CDN capabilities, especially for serving international
markets, can solve both their website performance and bot protection challenges in a
single SaaS solution.

Section 5
SaaS Solutions
Conclusion: Core Competency or Product Add-On?
Considering the damage that malicious bots can inflict on businesses, website managers and owners face an
important decision when choosing a bot protection strategy to pursue. Their evaluation measurements should in-
clude many criteria, from the level of in-house bot protection expertise, to the cost of maintaining robust bot
defenses.
Below is a list of criteria, including their relative value, which website owners have indicated are most important
when comparing bot protection solutions. We provide them as questions website owners should ask potential
solution providers.
As they evaluate solutions against those criteria, companies must remain focused on the underlying driver for bot
detection and remediation - the protection of company data and other assets that form the core of their business.
As that protection rises in importance, so too does the need for a solution designed and built solely for bot
protection.

Section 6
The Distil Solution
About Distil Networks
The innovations at Distil Networks have made it possible for companies of all
sizes to comprehensively and cost-effectively protect their online content and
data from bots and bot networks anywhere around the globe.
Delivered as the first ever cloud-based bot detection and mitigation solution, the Distil system incorporates the
world’s largest bot-tracking database with technology to identify, track and mitigate bots in real time. Moreover,
Distil uniquely tracks each bot via a fingerprinting algorithm with over 40 variables, raising the bar significantly in bot
detection to 99.99% reliability.
For your online company, this means you gain the most thorough and proactive bot threat mitigation capabilities on
the market today. Deploying the Distil solution via SaaS model delivered in the cloud or installed behind your
firewall, you achieve the maximum bot protection with no need to change your underlying website infrastructure.
You can further boost your website performance with Distil by choosing to implement bot protection as part of
Distil’s content delivery network service.

Section 6
The Distil Solution
About Distil Networks continued
Distil has innovated the bot protection industry by uniquely combining two technologies that have simultaneously
matured: fingerprinting of website visitor connections; and machine learning algorithms used to detect online
behavioral anomalies. This combination enables Distil to identify, track and mitigate bot threats before they
damage your business. With each new bot fingerprinted, the Distil knowledge base grows and automatically
disseminates threat updates across our customer community in real time.
What are the benefits of real-time, proactive bot detection and mitigation? Distil helps you stop all types of bots
before they inflict real business damage. This includes content and data theft, bots inflating advertising costs due to
click fraud, competitors stealing inventory intelligence in real time, costs of serving false visitor traffic and more.

Section 6
The Distil Solution
Fingerprinting the Distil Way
Detecting malicious bots means little if you cannot do so in a timely manner with a high degree of accuracy. Detect a
malicious bot too slowly, and your business could lose its competitive edge. Inaccurately take action against
well-intended site visitors or customers, and you stand to alienate the lifeblood of your organization. This highlights
the need for a solution that provides bot detection and mitigation in real time - with the highest degree of detection
accuracy possible.
Only Distil leverages a bot detection methodology that incorporates over 40 bits of information when developing
a fingerprint for each user connecting to your website. Delving far deeper than user agents and IP addresses, Distil
looks into all connection properties from the first time a user engages your site. Our technology then inserts
JavaScript into the connection stream to capture even more detailed characteristics of the user. Once a complete
fingerprint is developed and a bot is detected, the bot has no way of escaping our detection ever again.

Section 6
The Distil Solution
Follow that Bot!
Nobody has to say it, because Distil automatically and proactively follows bots once the system has identified them
by their fingerprint. Even if a bot’s controller makes various attempts at obfuscation, such as accessing your site
using proxies or the TOR network, the fingerprint is inescapable.
Additionally, the fingerprint has so many unique properties that detection of a single bot enables Distil to render an
entire botnet harmless to our customers.

Section 6
The Distil Solution
Machine Learning Your Business
Taken alone, Distil’s fingerprinting technique far surpasses any other bot detection
method on the market today.
Yet, our detection methodology does not stop there. We leverage machine learning
algorithms to understand how the typical users on your website interact with both
the site and one another while there. By understanding the normal behavior patterns
of your various user types, we quickly recognize any abnormal patterns and can take
action against those users.
By focusing on your business and the way it works online, we maximize bot detection
and minimize the occurrence of false positives - the act of wrongly identifying a
legitimate site user as a bot.

Section 6
The Distil Solution
Machine Learning Your Business continued
Fingerprinting and machine learning are the cornerstones of our innovative bot detection and mitigation. But the
innovation does not stop there. We deliver bot detection and mitigation in multiple ways to suit how you have set up
your Web infrastructure.
Distil was designed with a laser focus - to solve the problem of malicious bots for every site on the web. To do so, we
created a system that any online company can use, because the more customers who join the community, the greater
the knowledge base of bots detected and fingerprinted.

Section 6
The Distil Solution
SaaS Model Allows for Flexible Deployments
1. Public cloud service with optional CDN
2. Private cloud deployment with a physical appliance
3. Private cloud deployment with a virtual appliance

Section 6
The Distil Solution
Distil CDN
Many online companies have sought the performance of a content delivery network (CDN) to serve website content
and applications to a dispersed user audience. However, they often find CDNs prohibitively expensive, particularly
for many small and medium-sized businesses.
Distil changes the equation by offering CDN capabilities as part of our bot mitigation solution.
To make the best bot detection and mitigation system possible, Distil established multiple points of presence (PoPs)
around the globe to collect bot data and disseminate it as fast as possible to our customers.
Through our PoPs, we detect and follow bots, provide real-time bot threat data to our massive community of
customers, and continuously update our service. Equally important, we act as a massively distributed Content
Delivery provider, serving your website and content from multiple points across the globe.
By establishing PoPs on major Internet routes across multiple continents, we have created a CDN capable of handling
huge traffic volume at higher performance levels than customers
could achieve on their own.

Section 6
The Distil Solution
Distil CDN continued
The higher performance of the Distil CDN arises from two areas of improvement. First, by minimizing bot traffic, you
relieve significant strain on your Web servers and application servers.
Simultaneously, our CDN routes your site content to users from distributed PoP locations, reducing the time it takes
for user requests to get answered.
For organizations that already have CDN or have built out their own global presence, Distil provides seamless bot
detection and mitigation within the parameters of your existing Web serving infrastructure

Section 6
The Distil Solution
Private Cloud Deployments
Distil can provide the same cloud-based service in private deployment either through a Distil physical appliance or
virtual appliance. The appliance sits behind your firewall with a heartbeat to the Distil bot database in the public
cloud, downloading the latest bad bot fingerprints in real time.
Whether your organization deploys Distil’s system in the public cloud or as a private installation, no changes to your
existing Web infrastructure are required. Switching over website content delivery and/or turning on bot detection
and mitigation can occur in just minutes.

Section 7
From Detection to Mitigation
Streamlining Actions from Detection to Mitigation
As noted earlier, the best bot detection solution in the world (which Distil have) helps little if an organization doesn’t
immediately follow detection with mitigation. However, the type of mitigation technique chosen varies based on a
number of factors. These include, but are not limited to, the following:
- Type of bot (fraud, theft, form spam)
- Value of the data
- Unique risk of false positives
- Risk tolerance of management
- Etc.
Distil has developed its system to accommodate the multitude of ways in which organizations may want to deal with
varying bot threats. Some may choose to immediately block all bots, while others may choose to vary responses.
These can include throttling down bot acceptance gradually, posting challenges to likely bots (e.g. Captcha forms or
other Turing tests.) monitoring a bot or even providing fake data to a bot.

Section 7
Streamlining Actions from Detection to Mitigation continued
Your organization cannot afford to delay a decision on how to handle a bot after detection. For that reason, Distil
includes scenario-based automated settings that you can specify prior to encountering a bot.
Your organization cannot afford to delay a decision on how to handle a bot after detection. For that reason, Distil
includes scenario-based automated settings that you can specify prior to encountering a bot.
Distil recognizes that some customers may want to add their own knowledge into the system based on past
experiences. As a result, Distil provides all customers access to the Distil Portal where they can set specific blocks
ahead of deployment time.

Section 7
Streamlining Actions from Detection to Mitigation continued
Customers frequently gain incremental protection by adding the following information into the portal:
- Blocking of specific IP addresses
- Filtering out requests, by country
- Blocking known malicious referrers and hot links
Once Distil is deployed, customers can modify their settings at any time via the portal.

Section 7
A Critical Mass of Bot Data in One Place: Ready for You
Each Distil customer benefits from a knowledge community that has been identifying and tracking bots for years.
Bots that Distil identifies for one customer instantly get shared across the Distil user community to your website. In
this way, as a new customer, you instantly benefit from the industry’s largest knowledge base of malicious bot
fingerprints. From day one, you gain protection equal to that of a company that has been tracking and fending off
bot networks for years.

Section 7
Test and Compare
Distil has brought innovation to a market badly in need of innovative solutions.
For too many years, bots went undetected, because no holistic solutions were ever designed for the sole purpose of
detecting and mitigating bots and bot networks. That is, until we at Distil Networks introduced our purpose-built
system of bot detection and mitigation.
Through our technology and fast-growing customer community, we have turned the tables in the turf war on bots
for those sites operating under our protection umbrella. Some had never fought bots before, while others had tried
to use add-on modules to their existing Web Application firewall or other site security products. But these systems
were never designed to address the bot threat and should be tested head-to-head against Distil to understand how
vastly superior

Section 7
Test and Compare continued
Distil’s bot-focused solution is to modular product add-ons.
Since Distil comes to you via a SaaS model, testing the system is painless and requires only minutes.
To see how Distil Networks can improve your bot protection while enhancing site performance, contact them for a
free performance test.

How To Protect Your Website From Bot Attacks

More Related Content

What's hot

Viewers also liked

Similar to How To Protect Your Website From Bot Attacks

More from London School of Cyber Security

Recently uploaded

How To Protect Your Website From Bot Attacks