Downloaded 33 times
![You notice that you're
sending mails
[..]
Received: from web01.yourcoolapp.com ([198.51.100.42])
Subject: WELCOME TO YOURCOOLAPP!
From: noreply@yourcoolapp.com
To: evilattacker@example.com
But then you notice that you're sending emails! And your attacker noticed
that the email address contain interesting information in their headers.
...and the DDoS attack on your server resumes.
...so you configure your mail gateway to not include servers IP addresses
...get new IP addresses
...hope for the best](https://image.slidesharecdn.com/ddosmitigation-140519135801-phpapp01/75/DDoS-mitigation-in-the-real-world-23-2048.jpg)
Uploaded byMichael Renner
DDoS mitigation in the real world
The document discusses DDoS attacks, their motivations, and their impact on services, emphasizing the challenges and costs associated with mitigation. Michael Renner shares personal experiences with the complexities of DDoS attacks and underscores the importance of proper planning and use of available technologies for defense. Key strategies for mitigating DDoS attacks include increasing attack surface, traffic scrubbing, and leveraging external DDoS protection services.
More Related Content
Similar to DDoS mitigation in the real world
DDoS mitigation in the real world
- 1. DDoS mitigation inthe real world 19th May 2014 Vienna DevOps/Security Meetup
- 2. Michael Renner @terrorobe My nameis Michael Renner In IT since 2001, focus on system administrations and web operations. Freelancing since 2010 for multiple companies doing a hodge-podge of tasks - from building data centers, DBA work, troubleshooting, performance optimization... If you've got questions, just ask!
- 3. Company I'm freelancingfor. Viennese company, founded around 2010 - our offices are on Mariahilfer Strasse Operations, Development, Customer Care, payment solutions, SMS delivery, you name it ;) And since a year ago also DDoS protection
- 4. DDoS? In computing, adenial-of-service (DoS) or distributed denial-of-service (DDoS) attack is an attempt to make a machine or network resource unavailable to its intended users. Source: http://en.wikipedia.org/wiki/Denial-of-service_attack
- 5. Why people DDoS ...whichleads us to the topic of our today's talk. DDoS is something that has gotten more and more news coverage in the last few years. Chances are high that you already tried using a service while it was being DDoSed or that you run a service on your own which was the target of an attack So why do people DDoS?
- 6. ...to extort money Thisone is easy. Was very prevalent in the gambling and sports betting area. Back at Inode in 2003 people where already DDoSing gambling companies! "Comes with the business"
- 7. ...to underline a politicalagenda Operation Payback in 2010 - done by Anonymous - targeted against recording industry prosecuting piracy as well as finance companies denying service to wikileaks
- 8. ...to escalate issuesin customer care Has happened in the past - if you annoy the wrong customer he might escalate things by his own means
- 9. ...because somebody pissed themoff on IRC User chats from your server/network, annoys the wrong people and they show them that they mean business
- 10. ...to prove apoint ...or just to prove a point, whatever that might be. The more high profile/recognized your site is the higher the chance that somebody will attack it just for the fun of it.
- 11. DDoS attacks have asymmetricalcosts The problem with DDoS attacks is that they have highly asymmetrical costs. The attack itself can be quite cheap whereas the lost revenue as well as time and money to bandaid-patch an attack can be quite expensive
- 12. Attacks are cheap "Pricesper hour may depend on your target. Price starts 10$ per hour to 100$ per hour depending on how Huge or Protected the target is." When we're talking money one must know that attacks are quite cheap. This is an example from end-customer site, completely with youtube video and everything! Depending on how smart the attackers are and how good you're connected attacks can be significantly cheaper.
- 13. Attacks happen when peoplehave free time • Weekends, holidays, bridging days • In their respective timezone! • More often than not: • the time when you'd like to have free time as well Another problem is that the attacks will happen when people have free time.
- 14. Attacks are hardto identify • "The DB server has too many connections open" • "I'm seeing packet loss and a large amount of softinterrupts on CPU0" • "The server is down" DDoS attacks are very disorienting Depending on how much instrumentation you have for your infrastructure you might not even be able to tell that you're getting DDoSed "Newrelic might not be enough to spot it" Depending on attack you might need OS, application level even network packet statistics.
- 15. Mitigating is hardwithout proper infrastructure • "Let's see if I can whip up an u32 classifier to filter out this pattern I'm seeing" And when the attack is already running you'll often have a hard time jury- rigging yourself out of the attack.
- 16. Hard to findthe sources of attacks • Most attacks use spoofed source addresses • If it's a reflection or amplification attack you won't even get those. • And even if you'v got real sources, it's a hacked server in nearly all of the cases More often than not management will be interested on who's doing the attack and why. This is nigh impossible if you're not the NSA, having a big picture of all the traffic going on. And even if you've got "real" source addresses nearly all of them will be hacked servers connected to an anonymous C&C servers
- 17. In the end... •...it's you and your customers who're unhappy so it's best to try not to get knocked off the net in the first place!
- 18. DDoS war stories Togive the users who haven't experienced a thorough attack yet a glimpse of what it may look like I've collected a few things we faced in the past.
- 19. So you runa website... Most of you operate a web/online service I guess, which generates you fame, fortune or both. And along the way you picked up a few people who don't have the best interests regarding your site, and so they decide to...
- 20. Somebody does aslow- read attack ...open up hundreds of connections, read a few bytes at a time... basically exhausting open workers, tcp connections, file descriptors, memory, whatever comes first. so you get a proxy setup like CloudFlare up and running, all HTTP requests proxied through They detect the botnet IPs either through pattern matching or existing botnet IP lists and start filtering those out or presenting them with captchas. Attack solved.
- 21. But they stillknow your server's IP address Then the attacker goes on netcraft.com, checks the former IP address of your site ...and decides to send a few hundred megabits worth of UDP packets your way. You ask your ISP for a new subnet, renumber your servers, add the new IP addresses to your proxy setup and hope for the best Things are quiet for now.
- 22. Suddenly you geta ERR_NAME_NOT_RESOLVED when visiting your site After a few hours the monitoring starts to page you. You check your site and you get a NAME NOT RESOLVED. ...turns out the cheap registrar where you bought your domain ...who still does your DNS ...isn't quite DDoS resilient. So you shop around, see companies like dyn.com or Amazon route 53, move domains over, wait for cached entries to expire and after a few hours you should be back online!
- 23. You notice thatyou're sending mails [..] Received: from web01.yourcoolapp.com ([198.51.100.42]) Subject: WELCOME TO YOURCOOLAPP! From: [email protected] To: [email protected] But then you notice that you're sending emails! And your attacker noticed that the email address contain interesting information in their headers. ...and the DDoS attack on your server resumes. ...so you configure your mail gateway to not include servers IP addresses ...get new IP addresses ...hope for the best
- 24. ...so you thinkyou're good for now. And then your ISP goes offline By then you think you've seen it all and you think you're good to go. ...most smaller ISPs are not equipped to sustain attacks even 10 to 50 gigabits in size. If regular traffic is in the single-gigabit per second range it's easy to knock them offline. Usually they then just filter the destination IPs at their carriers and be done with it.
- 25. Mitigating attacks... in Theory! ...so,this should give you a nice overview what can happen in the real world. So what can we do about these attacks? In the end all of the DDoS Mitigation services out there are doing the same thing.
- 26. Increase your attack surface •Geo-distribution is the way to go • either via DNS or IP anycast • -> more traffic sinks! First part is increasing the attack surface Most customers have just a single datacenter, amazon availability zone and so on. Very easy to concentrate your attack on. If you operate multiple Datacenters it's much harder to knock all of them offline - loadbalancing done either via DNS & geoip or IP anycast setups - same prefix announced on multiple locations This all boils down to have more traffic sinks to filter the traffic out on
- 27. 150 Gbit/sec sounds scary •Less scary when it's: • 40 Gbit Hongkong • 20 Gbit Frankfurt • 35 Gbit London • 25 Gbit US West • 30 Gbit US East I'm just pulling a few numbers out of my hat to give you an example. Suppose you've got a 150gbit attack running - which to this day is still a very large number when it comes to DDoS attacks - this gets far more manageable when you operate multiple sites where the traffic has to go to And even if they manage to knock one of the sites off the net all the others are still online and answering requests.
- 28. Scrubbing the traffic •In general: you want to filter out all "DDoSy" or anomalous requests from your traffic • Statistical models, signature-based filters, hand-written rules, etc. • Specialized hardware with in-ASIC rule processing • Beyond the scope of this talk The other big part is to scrub the traffic and remove all packets which belong to "probably unwanted" traffic. The further down the traffic goes through your stack the more expensive is the processing of a malicious request. Compare the time needed to throw away a request on the firewall vs. throwing it away in the web application. Big Companies use specialized hardware with network chips, being capable of doing complex rule processing at wirespeed of the interface. They often use statistical models to detect network anomalies, have rules for known attack patterns (e.g. NTP reflection attacks) and can also deploy custom rules specific to attacks currently going on.
- 29. Don't build yourown scrubbing setup unless you've got good reasons In most cases it's much cheaper to just lease scrubbing services from one of the established players instead of building your own infrastructure.
- 30. Mitigating attacks... ...in Practice! ...somuch for basics - what can we actually do to improve our security?
- 31. Don't do DDoSmitigation as an afterthought First and foremost... it's much nicer to talk and discuss about these things during office hours with sober colleagues than on a saturday night at 4 am after 8 beers
- 32. Decide on ascope • Threat modeling • Build playbooks • Prepare & implement low-hanging fruits right now You need to do threat modeling - which attacks do you expect, how you're going to manage them. Prepare contacts up front even if you don't want to buy something right now. Have a playbook describing how you're going to react Implement low hanging fruits right now!
- 33. Have good OpSec Oneexample for a low-hanging fruit: Operational Security Make sure you don't leak IP addresses you don't want to It makes sense to have two different network segments - one for public- facing IPs, another one for the actual servers. Actual server IPs don't even have to be public ones if you can avoid it. Things to check: Mail Headers Test systems like beta.yourcoolapp.com MX records!
- 34. Choose an approach Andas with every idea - there's more than one way to do it!
- 35. Proxies • The easy/cheapsolution • Protocol proxies (HTTP, SMTP, etc.) • Geo-distributed The cheap approach would be a proxy setup. You get a list of protocol- level proxies where you point your DNS records to - you configure the destination for the proxies and all the traffic hits these servers first. Well-known company is CloudFlare, offering this service even for free Good if you need a fast solution
- 36. Proxy Caveats • Fewcontrol over proxy servers • Can hurt troubleshooting times • Applications don't see end-user IPs anymore • Fix for HTTP: X-Forward-For • If your real IPs are leaked it's game over again • Not feasible for all protocols This comes not without draw-backs though. If you need to troubleshoot things you might not have the level of control you'd like, since somebody else is already terminating your HTTP requests for you. Your applications don't see IPs anymore If you should leak your IPs it's easy for an attacker to just go for your real servers. And it's not feasible for all protocols, VoIP, UDP-based gameserver protocols, etc.
- 37. Routed solution • Youneed a /24 network prefix • IP tunnel to scrubbing center • All scrubbing locations announce the prefix • Traffic gets sent to closest location, scrubbed and forwarded to you Those get easier to manage if you've got a routed setup Routed approaches are more expensive and complex to maintain Pro: Protocol- and application agnostic
- 38. Routing caveats • Takestime to implement & test properly • More expensive to setup and maintain • Routing & BGP can be scary It's not without it's drawbacks though:
- 39. Appliances There's still appliancevendors in the DDoS market, e.g. Arbor or Radware. Was a bit surprised by that, but it absolutely makes sense Appliances targeted at end users should always come with a traffic scrubbing offer, otherwise they're snakeoil - it's easy to DDoS a single device running behind even a 10Gbit link. They can make live easier though since they automate tasks like route changes and provide additional information for the scrubbing center
- 40. Ask your ISP somethingyou shouldn't neglect is that your ISP may already have DDoS infrastructure set up - it doesn't hurt to ask them what it is they're running and if they can make you an offer
- 41. Common caveats • Yourapplications latency is most likely going to increase • There can be false positives & overfiltering • Your system is becoming more complex Something to consider with any choice you're going to make is Latency is going to increase false positives and overfiltering - how are you going to spot this? your system is getting more complex
- 42. Interested? Talk to us! [email protected] [email protected] Ihope that gave you a bit of an overview of how the current situation looks like when dealing with DDoS If you're interested - please talk to us Ixolit is both a Prolexic and a Dyn Partner Prolexic is one of the major traffic scrubbing operators in the world. Got recently acquired by Akamai. Dyn.com offers anycasted DNS setups which are DDoS resilient Already have a few production setups running for _huge_ sites which solved all our DDoS issues to this day
- 43. Questions? Do you haveany questions? Thanks!