I'm ready to go. This dude's a pro. This guy, another cool guy. I mean I got some awesome guys here. I'm bragging on them, not myself.
Another awesome guy. He's going to do a talk on DDoS. And this should be very interesting because DDoS is the talk of the hour.
And sometimes, even on this subject, and I think that we're kind of like, NAMS will say communities like on the sidelines being defined by Sarah Palins and all that stuff.
So it's kind of interesting.
Thank you. Thank you, everybody. So first of all, thanks for staying. Okay. So I understand I'm between you guys and dinner, and it's late anyway.
So everybody has lots of questions. Okay. I'll fill them. I will stay as long as you guys want. Formal presentation, I'll try to do it in say 45, 50 minutes. Okay.
I'll set up try. Okay. So we're going to talk about denial of service. Denial of service has been in the news a lot lately.
Okay. Has anybody not heard of Operation Payback? All right. So I think we all know about Operation Payback.
It's when the Internet's decided that they hate each other. Okay. And then pretty much it's attacks, counterattacks, counter counterattacks, law enforcement takeouts, a whole bunch of really, really fun stuff.
That's pretty much what I've been living for the past week. Okay. So this is the commercial slide. So if anybody has any shmoo balls, you can come in with them.
But really, the keys out of this are we have a lot of servers. And at any given point, we're in between 15 and 25% of the Internet's web traffic.
So we see a lot of DDoS's. We also do a lot of DDoS mitigation. Okay. The other thing with this is we're a distributed model.
So if you think of us as a huge geographically distributed reverse web proxy and load balancer, that's roughly what we do.
We put servers out in all the ISPs. We deliver content through them. But because we're dispersed, we can counter distributed threat.
And we'll talk about that here shortly. Okay. So that's it for Akamai. So what are the threats? What are they doing? Right.
So extortion. Everybody remember this? So the first time I heard about extortion, I was like, okay, that's cool. You know, 1995 called it once.
It's a threat. Vector's back. Actually, this happens quite a bit today because when you're looking at it in your business and, you know,
your e-commerce site and somebody gives you an extortion letter for $30,000, you're going to pay it. Okay.
The reason that you're going to pay it is because two hours of downtime will cost you way more than $30,000.
The problem is that every time you pay, they keep coming back for more money. Okay. But this actually still happens today.
It happens a lot more frequently than you would know. Okay. Then state sponsors like Amazon, Amazon, Amazon, Amazon, Amazon,
then state sponsors, everybody knows Estonia, Georgia, cyber war, the, you know, the world is ending.
However, last year there was an attack against the United States. It was 200 gigs per second.
I'm going to talk about that as a case study here in a minute. But for comparison, operation payback at the most, they had 1.5 gigs per second.
So Anonymous isn't able to bring a sizable amount of traffic. I mean, it's still sizable when it hits your network circuits.
But overall, it's not that big. Okay.
After this protesters, I love this guy because he reminds me of everything that's America.
You know, he's got the go USA sign. He's got the, hey, get a brain war in. You know, his cardinal shirt is awesome.
Anyway, the lesson out of this guy is that if people hate you enough to show up outside your office with the sign to protest you,
they hate you enough to go rent a botnet or get an activist army to come attack you and take down your website.
Okay. Mostly just for PR reasons. We'll talk a little bit about that. Okay. So the VTARs. All right.
So operation payback. How long has operation payback been going on? Does anybody know? A week? No.
Operation payback has been going on for about three months. Okay.
It started when an Indian company named Iplex announced that they were conducting denial of service attacks against the Pirate Bay
to keep people from downloading content via torrents. Okay.
So then the collective wisdom, the internet said, well, you know what? You can't DDoS us. Stick it to the man.
We're going to DDoS you. And ever since then, it's been game on for Iplex, the MPAA, the RAA, the U.S. Copyright Office got whacked,
the UK Intellectual Property Office, the Australian RIA. Yeah. That was a APEC, APEC, ACTA. I forget which one.
Yeah. There's a whole bunch of Spain. Spain got whacked. But really they've had this army and they've had lots of campaigns.
And they've just been hitting everybody throughout the course of three months. Really awesome to watch.
It's kind of like watching the two school bullies fight it out. You're like, ooh, either way, no matter what happens,
somebody that I hate is going to lose. Okay. Which is, you know, it is what it is.
And then late last week, let's see, week before last, the, let's see, with the whole WikiLeaks thing.
So WikiLeaks released all the State Department things. They themselves were the target of a DDoS attack.
Okay. To try to keep the material that they had from coming out. And then the collective wisdom of the internet,
because you've got this big vigilante gang, they're like, oh, hey, let's take out these other guys that have cut off WikiLeaks' funding,
or, you know, our attorneys representing girls that are suing Sunch. And, you know, a whole bunch of different things happen.
And what happened is there's, you know, there's the original DDoS on WikiLeaks.
There's the counter DDoS against the people that are doing that, or the people that are expected to be doing that.
Then there's the DDoSs against all the companies that don't play along with WikiLeaks.
And then there's the counter DDoS now against all the command and control servers that the protesters, activists, criminals are using to launch their own DDoSs.
So it's been fine. Basically, the internet hates itself. Okay.
So here's another guy. This guy's a favorite of mine. He's a 17-year-old. He got caught cheating at SOCOM online.
So, and then they banned his account. So he's like, well, you know, screw you. I'm going to stick it to the man.
He steals his mom's credit card, rents a botnet. So the going price now for a botnet is $50, 10,000 notes, 24 hours.
Okay. That's cheap, right? The attacker's capability to conduct DDoS attack is minimal. Minimal at best.
Especially if you have stolen credit card numbers. Yes.
How much did you say?
$50.
Wow. Thank you.
Okay. So, and that's what this guy did. But, you know, the best is what this tape is saying down here at the bottom is,
if you're dumb enough to get caught cheating at SOCOM, you're dumb enough to get caught renting a botnet and doing DDoS.
Anyway, and in, when was it? May? So May of this year, they, oh, my daughter's a VR. Okay.
And anyway, in May, they sentenced him to, I don't know, thousands of dollars of fines and community service and all that stuff.
She just did it at 17. Okay. But the key out of this is that, yes, the barrier to entry to do a DDoS is very, very, very minimal.
And so just as bandwidth penetration to the home gets faster and faster and faster, the ability for people to conduct DDoS against servers gets faster and faster and faster.
Okay. So December has had more DDoS than just operation payback. Okay.
We actually announced this probably yesterday and this morning. We did a lot of press.
But there is a, quite a few DDoS attacks against e-commerce merchants that has been going on since Black Friday.
So, all right, the day after Thanksgiving when everybody goes and buys stuff online.
And Cyber Monday where everybody buys all their Christmas presents online.
Anyway, there's a whole bunch of DDoSs against e-commerce merchants, more likely just to kill their income stream, right, to deny them sales.
So, I don't know. There's a lot of questioning about the motivation behind it.
But I've seen in the past two weeks, I've seen maybe 30 individual DDoSs. Okay.
That's a lot. That is a lot. Okay.
Big attack traffic per year. This is staff from Arbor's Network. You guys can see there's a big growth curve.
Us last year on July 4th, we hit 124 gigs per second. That was against one customer. Okay.
So, against one target site, the total aggregate traffic was 200 gigs per second.
So, the ability to DDoS things is just growing. And it's growing better and better.
A lot of that's a botnet problem. Okay. Because activists can't. Today, activists cannot achieve this volume of traffic just by using the means that they have.
Okay. So, those are the bad guys. Who they are, what they're doing.
Okay. So, fail paddocks. Network and devices. Okay. So, there are two different things that happen here.
One is the actual circuit. Okay. So, it's basically, if you have a pipe, you stuff things into the pipe and then nothing else can fit through the pipe.
Okay. That's pretty easy. That's just a matter of metrics. So, if you're the attacker, you can grab, you know, HPing, SYN floods, ICMP floods, anything you want.
And attack another computer, attack another site. And it really is just a function of how much bandwidth do I have versus how much bandwidth do the defenders have.
And can I shove more at them than they're able to receive. Okay. There's also web objects. So, you know, flooding attacks are pretty, I don't know, they're the most brute force denial service that's out there just because it's bandwidth saturation.
There are web objects that are a little bit more leveraged than bandwidth saturation because if I can take any of these objects, and you guys can see them on there.
If I take any one of these objects, I send a small, relatively small HTTP request. Okay. It's, you know, it's measured in, what, bytes.
And I can request something back that's measured in megabytes or, you know, even in K. That's a leveraged attack. Right.
So I use a small amount of attack traffic and I request a large object that actually fills the pipes coming back from that, from that target site.
Okay. And then the wonderful thing is that if the network is attacked, everything else that resides on that network is also attacked.
Okay. So if I attack the web server and I saturate the bandwidth going to the web server, the mail server that's in the same subnet also can't talk.
Okay. Unless you do quality service. There's some mitigation factors for that. But pretty much everything dies. So does anybody know what tick is?
Okay. So the trusted internet connections where the government's taking all their internet access points and reducing them down to 50 was the last number I heard.
But more realistic, maybe 100. But in doing so, yes, it's easier to monitor. It's easier to keep track of the traffic going in and out.
It's, it forces people to ask for permission to add new traffic. But it also creates denial of service opportunities for people.
Choke points.
Choke points. Exactly. Which is by design. Okay. By design for, for the security side of it being able to monitor and restrict the traffic.
But you're also able to monitor and restrict the traffic. Okay.
The number is hardly like six.
Yeah. Yeah. It changes. It changes. There's, there's policy. There's reality. And then there's how the heck do we get there from here?
Yes. Okay.
All right. So DNS. Okay. DNS is great because DNS controls all the other services you have. So if you can do a denial of service against DNS, and there are a couple different attacks.
So there's a reflector attack where you send, you send a request for resolution out to a whole bunch of DNS servers, and then they all of a sudden come to the authoritative source and try to pull that information.
This is bad. Okay.
But if DNS dies, everything else dies. Okay. Something is, if anybody's looking for a research project, this bottom bullet here, DNS creates network application and OS load because you have more lookups.
You have more frequent lookups and it creates load on the operating system. Okay. On the DNS server operating system.
That's a very good area for research. Just saying. Okay.
Mail servers, mail servers have finite resources so you can flood them with, you can basically flood them with email.
You can also just, because, because they're email servers and they accept attachments, you can flood them with attachments.
It saturates the network. It fills up the storage space. It fills up the size limits on inboxes.
Okay. Oh yeah. Ham spam. All the, all the great spam filtering that we have. That creates a lot of OS load.
Okay. So, web and app servers. These, this is the fun part that I like. Okay. So load large objects. Talk about that, right?
So a large object, that increases the IO on the actual web server. So web server, the simplified version of a web server is it has a listener, listens for traffic, pulls objects out of, out of disk using IO and delivers that.
Okay. So the more you can make that server process grab stuff out of, out of disk, that increases the IO and it'll actually take down the operating system.
Okay. Exercise business logic, I'll show you about that. Siege. Siege is a, is a tool. It's actually pretty cool. I use it for web stress testing.
Okay. So really Siege, you give it a C file of URLs that you want it to act. You tell it how many processes you want it to spin up.
You tell it how long you want it to run. And basically it stands up a whole bunch of requesters and in the dates basically floods out a web server so that way you can see what the performance is.
Okay. It's actually really good if you use it to stress test your own servers to determine what your settings are.
So Apache, Apache and IAS have, they have got pools of spare servers and there's kind of a management of their servers. So they have a master process, stands up a server, server serves so many requests that it dies and it responds a new one to replace it.
Well, there are lots of sliders and widgets in there that you can set and Siege will help you stress test that. Okay. But you can also use Siege because it gives you feedback on how many query requests were sent, how many were dropped.
It's basically like, like some of the metrics you expect out of ping, okay, just regular ICMP ping. Only it's for web traffic. Okay. So it can tell you what was dropped, how many requests were dropped, how many were returned, what the average latency was.
So you can start to see, well, if I run an attack of a certain size, so many concurrent users for so long, at what point do I actually get a degradation in service.
But you can also turn that around and use that to measure target. So if you're, say you've got a bot or you've got some other way and you're attacking, you can actually use the metrics from Siege to find out if you're having actually any effects on target and if you're slowing down that web server.
Okay. Shell scripts work. So this is just a, you guys are pretty smart. That's just a shell script loop that calls WGET. Calls WGET and requests a page. So you just take this thing, write a script, say okay, execute it and then you know when you want to stop it you just give it a control save and stop it.
All right. So Loik, how many people have seen Loik this week? Okay, you're all criminals. Anyway, Loik is interesting to me and the reason is that it's a tool built by Anonymous for Anonymous.
It's pretty much their weapon of choice. It's just a flutter. It's a flutter. It does TCP flooding, it does VDP flooding. More recent versions of it send valid HTTP and actually get a valid response back, which is actually a step in the right direction for them.
But up here at the top where I have it circled, you can take this thing, you can give it an IRC channel for command and control. It will go sit in that IRC channel and then one person can come in, issue command to all the copies of this that are running that are sitting inside that channel.
So you think of it as just an opt-in botnet. That's pretty much what it is. Okay.
All right. This is another one that I found. This is MPAA leech. Basically, you see all these blocks in here. Each block is an image file of the MPAA website.
And then what happens is they've got a JavaScript control. You see down here in the lower right hand corner, so it's like right here.
You've got a little slider and you can basically throttle this thing and set the refresh rate. And what it will do is so this one is set up for every 15 seconds, it will go through and reload every image that's on that page.
So the whole idea being, hey, you know what, you hate the MPAA, wow, I hate the MPAA. Why don't you go to load this webpage, let it hide, let it, you know, just minimize it, let it run, and it floods out MPAA.
So you don't even have to have technical know-how like you would with Loic because you've got to install the.NET Framework, you have to install SP1, you have to grab Loic, you have to install it, you have to set it up.
This thing you just say here, click this link, join the protest. Okay.
Isn't that more of a gray area item because of the fact that they brought a web service, you have people going to the server, but they're not expecting so many refreshes and so much data.
That is correct. Okay. So yes, it is more of a gray area.
I still don't think you could explain it away.
But from a prosecution standpoint, it would be very difficult for you to, you know, this person is performing a criminal activity or civil unrest, cyber protest, whatever.
Yes. There's a fine line between loading a page numerous times and doing sort of a sit-in and loading a page a lot of times.
You've automated the sit-in.
Okay. All right. You would violate terms of service.
There's all kinds of things. Yep.
But they don't.
Talk to the lawyers afterwards. Talk to the lawyers afterwards. We must move on.
Okay. This is a more recent version. I grabbed this this week.
Basically, this is a Loic version, but it uses similar to the refresh of the MPAA, which does only they use the Loic brand.
Okay. Because the Loic, you know, fear the Loic.
And this one, you just go and you put in an URL and you hit the I'm a charge in my laser button.
You actually read that? Yeah.
You hit the I'm a charge in my laser button and it goes and loads whatever that target URL is and it just loads it repeatedly.
Once again, what they're trying to do is trying to make a tool that you don't need a lot of technical ability to go ahead and launch this DDoS.
So database servers. This is my favorite attack ever. Can anybody tell me what it does?
Here are my web penetration testers.
Anybody? Anybody? Doug? Doug, what does it do?
It's a big query, right? So you see here, A, they're probably susceptible to SQL injection. We won't even go into that.
You get the table equals search results. But basically it's, you know, all these things are blank.
And start year is zero and end year is 2009 and per page equals five billion.
So what it does is MPAA has this, excuse me, RIA, they have this page where you can go, you can search for anybody who's had a gold platinum record ever.
You're going to make me sad to tell you that parameter is really accepted?
Yes, that parameter is accepted. And then they allow the web browser, right here, yeah, this thing right here.
So they allow the web browser to specify how many records it wants returned.
Normally, yeah, that's okay. But you have to limit it inside the code. So what happened is they allowed that to be passed through.
So this query goes and the application server takes all this, makes this big, huge query that's basically select star from, you know, search results.
Where, yeah, where one equals one, okay. And it makes this big, huge database query, pulls a big, huge result set and gives it to the application server.
So the database goes, ah, this is a large query. I can't process it all at once.
The application server says, oh my God, this is a huge return result. I can't process it at once.
And it basically tanks down the database server, the application server, and nothing works. That's why it's my favorite.
Okay. So those are the basic fail patterns. So July 4th, what does this thing look like?
All right. So July 4th was a very large attack. You can see here peak traffic to government customer one for us was 124 gigs per second.
That was like five years of their traffic delivered in the process of two or three days. Whoops. Okay.
And then here at the bottom is one of my favorites. So OC-192 is 10 gigs per second at $2 million a month.
And that's off, that's off the network's contract. Anyway, so in order to protect against this, this top attack, you need maybe 13 OC-192s at $26 million a month.
Okay. That's just to have enough bandwidth to absorb this thing.
And the rest of the time when you're not under attack, you're using 0.1% of it.
Okay. So the timeline of what this thing looks like. This is actually a picture of a knock. So let's, you know, get another commercial.
But about, you know, two o'clock, we can alert firing in our operation center. Okay. Also notice that it's July 4th.
Okay. So the people that do DDoS attacks, they never attack, you know, Monday morning at 11 o'clock, you know, when everybody's just getting into work.
No, they attack Friday night at 6 o'clock. They attack during the holidays when you're sitting on the beach with a drink with an umbrella.
So that way it slows down your response to that attack. Okay. And that's exactly what happened here.
So anyway, we get alert fired. Boom. Look, we start looking at it. Yeah. You know what? They've got this traffic spike.
So this is actually out of our management system. But usually everything's full. Oh, now we got the spike over here.
Okay. So let's, let's start calling people what the heck is going on. Okay. By 8 o'clock attack increases rapidly.
So you take a look at this. These are the first probes that we detected here. And that's the spike that I just showed you.
And about 8 o'clock, everything just goes through the roof. Whoops. You know, yeah, somebody's under attack. Okay.
2100, we started pulling logs. We set logs up to come. When we started looking at logs, we found, you know what?
There's 100,000 unique IPs in 30 minutes. Let's start looking at these things. What do they look like? Let's start blocking.
You know, find out the attack pattern so that we can start filtering on it. In this case, what we found out, yeah, and then, you know, we started mitigation measures.
What we found out is, yeah, there were three different spikes. There was a small intersection in IPs between them all.
So it's not like we had, you know, 2000 IP addresses that we could block that, you know, block them by class C and say, hey, have a nice day.
We block your ESNs. In this case, we looked at it and we're like, wow, it's really, really spread out. The IP space is diverse.
However, they're all coming from Korea. So we said, okay. So we said, okay, cool. We have the signature for the attack.
Let's start blocking Korea. And that's what we did at 0300. So right here, boom, we started blocking Korea.
Traffic went to nothing. Like, okay, cool. That's great. But because Korea, you know, we blocked the whole peninsula, the whole country,
what we did was we took all of our traffic, we moved it out onto other servers, and then we took all the traffic for coming from South Korea and put it on one set of servers.
Okay. So what we do is we call it the quarantine zone, okay, or unofficially we call it the sacrificial lamb.
Okay. And that's what happened. You just say, here, we're going to throw it to the wolves. That's okay.
And you rely on the limitations of processing power and the speed of the network cards to actually throw out that attack.
So you could go to U.S. government websites if you were in South Korea, only it would be slow, okay, which is okay.
All right. And you can actually see when we, here at 0300, when we put them back on, then, yeah, we got the traffic going again.
But, you know, it's throttle that's manageable. It's not as much of a spike. Okay. And then after that, they changed targets.
It was basically, once you detect it, start mitigation, it's just a game of opt to the races because they'll say, oh, okay, I'm not getting an effect on target.
I'm going to switch to a different target, different URL, different website, different something.
I'm going to try a different tool. I'm going to try a different attack signature. And pretty much it's just a game.
Okay, they change it. We change to adapt to what they change to. They change again. We adapt to what they change to until they give up.
So you see over here on the seventh, so after, what, three and a half days, four days of attack, they basically said, okay, cool, you know, yeah, we're done.
On your previous slides, what was your response time?
I mean, you can take a look here. So a lot of the response time was in escalation.
In other words, trying to get a hold of people that are out on the beach to say, hey, what's, you know what, I want to do this.
I've got these things queued up, ready to go out. Do you allow me to do this?
Were they hinting? Were they...
Usually at that point it's, oh my God, our hair is on fire. Do it now. Click the switch.
Okay. All right. So observations out of this, attacks are sophisticated. I wouldn't say necessarily sophisticated, but they know how to target.
And they have really good command and control in that they can switch targets and they can switch targets rapidly.
Okay. The amount of volume that you have in an attack is almost directly proportional to what your command and control is.
That was one of the problems with Anonymous over the past week is that because they're, you know, it's 3,013 year olds sitting in an IRC channel,
all of them have ADHD and they're trying to decide what to do next, and it's about as effective as you might sound.
That's why they have tools like Javascript-Loy, so that way they can coordinate the efforts of all these people
without actually having to coordinate the efforts of all these people.
Okay. You know, it was a three day duration. You know, they're relatively large. And you know what? They're fast.
Because once they decide here, here this is our target, and this is specific to botnets,
once they decide here this is our target, that traffic ramps up really quickly because they issue the commands to their bots.
Okay. So building defenses, basic strategies, monitoring, tuning, caching, filtering, we'll go through all this.
So the first thing is instrumentation and monitoring. These are a lot of queries that we set up
so that we can actually detect a traffic spike for one of our customers.
So for some, you know what, your network, like the percentage of total network capacity and OS and DB load,
that's pretty familiar to people. You know what? Server is running at 85%. Server needs to be looked at.
Same thing with network. Network is saturated at 85%. Network needs to be looked at.
Percent increase of 50% traffic in over eight hours. That's a traffic spike. That's typical traffic management.
Increase of 50% of traffic over historical average. So that's to get the attacks that start relatively slow and ramp up slow.
Okay. So they don't just hit you all at once. They gradually build.
Percentage of bizarre user agent strings. And by bizarre user agent strings, I mean WGET, Python, Erl-lib, nothing,
or anything really strange. So some of the tools give you really strange user agent strings. Server response times. Yes.
Are you thinking about trying like a concept scripting or some kind of attack via user agent string?
Yes. Okay. So the database one, yeah, that was, I mean, that's for all purposes, that's a, it's input validation,
but it's similar to SQLI. But yes, okay, which brings up another interesting point is that a lot of times people will conduct a DDoS,
okay, and they'll whack your infrastructure. So you're running around with your hair on fire going, oh my God, you know,
the e-commerce server is down. How do I get this thing up? And yet at the same time, they're doing, you know, SQLI probing.
They're using the DDoS to keep you occupied so that you don't notice that they're doing data exfiltration. Okay.
Server response times. So we do this all the time. We have, we pull metrics off when we pull content off the servers,
and we measure the latency from the servers. That's just part of what we do. But when we notice that the latency from the server
has increased to a certain percentage, something's going on. So it might be a DDoS attack if it's in addition to all these other things,
or it just might be that they're having problems inside the data center. Okay. And then increases in errors because, you know,
web servers are failing, so they're sending lots of, like, 503, you know, server unavailable errors. Okay.
That's a sign that something bad is happening. Okay. And then the big question is, is it an actual attack or an awesome marketing campaign?
Okay. Because marketing departments can DDoS you. Okay. Because you go to, say, the Super Bowl, and you have a Super Bowl commercial
with your URL to your main webpage and big flashing lights, and everybody goes, oh, okay, I'll follow that URL.
And they type it in their computer and they go to your website. Okay. The difference here is that, well, the similarity is that
you have a large volume of traffic. You have to deal with it and you have to deliver that content.
The difference is that if it is legitimate traffic, you want to deliver it because those people are going to become customers.
If it's attack traffic, you can drop it. You can do a whole bunch of jerk things to it because that's attack traffic. Okay.
You're not as concerned about delivering to those guys. You can do it. It's a little bit different approach from mitigation.
Okay. So communication escalation, they pick in convenient times. We have this thing we just set up called the red button
where we send a little bit of your traffic through us on a daily basis. And then when you say, yes, I'm under attack,
we go in and flip the red button and all your traffic comes in through us. That little 5% is so that we can make sure that
the configuration doesn't change and that when we actually do failover or we actually do flip the red button,
that things don't bomb out on us. Okay. All right. So caching and dispersion, this is what we do.
Basically, the idea behind this is you've got all these attackers over here. They're these red guys. They attack one server.
That fails out. The end users actually go to an alternate server and come in and get that content.
Or we can take this one thing and say, here, this is the quarantine server.
Everybody's going to attack that. All the other users and the other customers are going out to the other servers.
So just because the attack is distributed, the defense needs to be distributed too.
Application, WebDB servers. Tuning means higher throughput. If you don't know how to tune Apache,
there are guides out there. I actually wrote a blog post on how to tune Apache.
Tuning means that you can handle a higher capacity and therefore you can survive a bigger traffic spike,
whether it's intentional or not. Input validation doesn't mean cross-sectional SQLI.
It's also allowing users to set limits on how much data is returned.
It's also allowing users to not create workloads on your servers. So it's what we call a highly leveraged attack.
Send a small packet, small request, makes a lot of work on the servers.
And if you send a whole bunch of those, the servers will fail over.
In application caches. So a lot of things. Databases have cache queries.
Oracle does, MySQL does, Postgres might in the more recent versions, but it historically hasn't.
But you can cache queries. So if a query comes a bazillion times, you can say, you know what, this one,
we're going to cache it and we're just going to serve it out of the database cache instead of actually doing the table lookup.
You can have caches for applications. So WordPress has a cache where it basically takes every page that it serves
and instead of making a database query and generating the page, it just pulls it out of cache and delivers it.
Same thing with load balancers, you're caching the page.
One of the tricks that we do is when you have a page that's supposedly dynamic, say a bulletin board,
the Twitter main page, not that we do this with the Twitter main page, but anything that's supposedly dynamic content,
but is anybody going to really notice if they reload the page 10 seconds later and the content hasn't changed?
So what we'll do is we'll go and set up a time to live on that particular page for 10 seconds.
Not a whole lot, right? But what it does is it allows us to cache that particular page so we can deliver it out of cache
a bazillion times instead of going back to the application server and pulling it out.
And you can do that on your load balancers, you can do that upstream, you can do that through us.
And then communication with upstream filtering and caching. So being able to tell people, hey, you know what,
I've got an increased server load, you need to set more liberal caching policy and continue to deliver out of cache
until the application server that has a dynamic content is ready to accept it again.
Another defense is a low bandwidth site. Low bandwidth site, it keeps the attackers from leveraging that small request,
large object delivered. Google works here, if you can use the first query is looking at target site for anything
that's a downloadable file. So any kind of Word documents, PDFs, usually these are larger files.
There are a couple more you could think about, you could add.zip in there,.gz, anything in there.
Finding what these are and then during a DDoS situation, blocking these.
You say, you know what, I'm not going to give you this large content, here's why, because I'm under a DDoS.
The other thing is Google Image Search. So if you go to Google Image Search, you can put in targetsite.com,
opt on the left hand side, there are widgets, one of those is here, show me where the large objects are.
Or where the large images are. And you can start looking at, okay, what things, you know what,
instead of this large image, I'm going to send something that's a smaller payload. What's that? Goatsy.
Don't send Goatsy, especially when it's a thrall net of activists, because it just makes them madder.
Then they just attack more. But conversely on the other side, if you're attacking something,
you can actually use Google Image Search and Google Regular Search to find large objects
that you then feed into the URLs that you continually reload.
And then you can use analytics, you can use Google Analytics, you can use Weissel.
You know, anybody, there are lots of web profiling tools that are out there.
The other thing here at the bottom is can you use mobile site? So mobile sites are usually optimized,
they don't have a lot of load on the server, and they are a low bandwidth site.
So a lot of times if you say, wow, we're under attack, instead of serving the main site, let's serve the mobile site to everybody.
And you serve that, it at least gets the content out to people.
It might look a little bit funny on a regular web browser, but it gets it out there, and you know,
your services still continue to stay up.
Okay, traffic redirect. This is a little trick that we have where somebody requests a page.
The first time you request the page, we give you a chunk of JavaScript that says, set a cookie, redirect yourself to the same page.
And then on our side we look and we say, here, if you have a cookie set, you have a refer that's the originating page,
here, we'll actually give you the page now.
Okay, the trick to it is the little JavaScript snippet, it's small, I can deliver it all day long, and that's what the attackers get.
Okay, but legitimate browsers or legitimate users with web browsers that have JavaScript interpreters,
they will actually run through and set the cookie and follow the redirect, and they can actually get the page.
Okay, that's a limitation in the tools that the attackers use where it does not, for the most part,
their tools do not understand JavaScript and they don't understand redirects.
All they understand is make a TCP connection, send get slash HTTP 1.1, and maybe they send a user agent string.
Okay, however, it's very easy for the attacker to manually retarget.
And they say, you know what, I request this page, something happened when I reloaded it manually to see how effective I was,
maybe I need to change my targeting and they'll change it up.
Okay, HTTP traffic filtering, filter on user agents, okay, null JavaScript,
there's a whole bunch of these things you don't usually see as user agent string.
Filter on it, a lot of times attack tools don't have accept languages, they don't understand cookies,
they don't understand JavaScript, and they don't send a refer.
You can start to filter requests on these things and start to say, look, if the traffic has these things,
then yeah, let it into the regular content, otherwise let it fail out.
Okay, DNS, make DNS resilient, put multiple servers behind a Vip, use DNS servers, or DNS services,
there are people that have DNS services that are built to withstand denial of service, we do it.
There are lots of people out there that do it.
And the other thing is you can use DNS to resolve things, so one of the ways that we black hole traffic is we say,
here, for this particular geographic area, when they request anything in that domain,
we're going to resolve it to 127.001.
So it's just like the old classic IRC trick, you know, hey, did you try attacking 127.001?
Or something better is like 127.5.8.25, because they don't understand it's a class A.
And then the quarantine service that I talked about.
Okay, so email, maximum attachment size, all right, so a lot of people set this up already,
just because they're worried about performance, and the Yahoo's for marketing,
who decide to send 35 main files to everybody in the company.
That works, that also helps you out in a DDoS attack.
GreetPos, GreetPos is cool, you just have to understand.
So GreetPos is basically when email servers talk to each other and they say,
hello, my name is Mike, I have an email, and the other guy says, oh, cool, Mike, I'm glad to meet you, I'm Joe.
Wait a second. Okay, send me your traffic.
Okay, that's all GreetPos is, but what it does is it slows down the,
it throttles the amount of attack traffic that the bad guys can send at you.
Okay, DNS tricks work here, spam filtering techniques, so you can say,
especially real-time black hole lists, you can start using those,
and black hole in specific areas.
And then network traffic scrubbers and filters.
So a lot of times they have geographic contents, geographic filters,
so that you can say, here, these particular subnets, these particular areas,
whatever the signature of the attack is, we're going to filter and not allow that traffic through.
And then they drop layer 3 through 5 attack traffic.
So anything that's not valid HTTP, drop it, drop it, don't care.
Okay, if it's a SYN flood, you know, it's just a SYN, SYN, SYN, drop it, we don't care.
That's not legitimate traffic, and that's what the scrubbers will do for you.
The way that this works in our world is because we're a proxy,
if the traffic doesn't actually, isn't on port 80 and isn't valid HTTP, we just drop it.
There's no server name in there to listen to it, we just drop it, we just absorb the bandwidth.
Okay, and then one cool trick is hiding the target servers, right?
So a lot of times you set up DNS that will force the traffic through your scrubber,
and people can go in and use FC host tricks to attack you directly.
Well, you can set up ACLs at your ISP to force the traffic to go through the scrubber and to you.
Okay, and we do that quite a bit.
Okay, one advanced technique that we have is called waiting room.
So it's a lot like the JavaScript redirect that I mentioned before,
where you get the JavaScript redirect, only we manage a queue in an alternate site.
So when you come in, we drop you into the alternate site, we load out of cloud storage,
say here, here's your alternate site, and then we manage the queue there,
and when the web server says, hey, you know what, I have space, I can take more requests,
then we start to migrate users back to that web server.
Okay, so the whole idea being they have a way to communicate with us what their current load is
and how many concurrent sessions they can tolerate,
and then we manage the queue and move users into it.
Okay, and that's it for defenses.
The trick out of all of this is that there are a ton of things that you can do to protect yourself against a DDoS.
It really comes down to, you know, the basic risk management stuff, putting on my long cap.
Basically, how much does it cost for an outage, what's your tolerance for it,
and how often are you the target of a DDoS, or have you said anything about WikiLeaks this month?
But yeah, it really is, you know what, how much tolerance do you have?
A lot of people have business continuity planning, where they can say,
here, an outage costs us this much, or an outage would bring this impact.
You can use those exact same figures and say, here, an outage is an outage,
whether it's a flash mob, whether it's a botnet DDoS, whether it's a bunch of yahoos and IRC,
and here's, you know, and because it's all the same, sites unavailable, here's what we need to do to protect it.
And that's all I have, folks.
Alright, questions? Questions from the crowd?
Anybody have a question? Yes, sir?
Oh, God. Well, this is the week for retaliation.
It's happening a lot. It depends on the nature of the attack.
Obviously, the biggest payoff for you as a defender is to take down the command and control.
Whether it's a botnet, whether it's activists and IRC, whether it's, you know, a kid running slow-lars,
anything like that, is to take down the command and control because it's a numbers game.
And if you can reduce the amount of attacking machines, you can increase your survivability.
So retaliation, yes, provided that you know what you're doing, and you're prepared for the legal consequences.
Okay.
That is correct. So if you take a look at over the past week, the Swedish lawyers that are responsible
on the whole legal suit against Assange, they're like, you know what, this is a brochure site.
It has the pictures of our attorneys and their phone numbers. You know what?
If it blows up, it doesn't matter. Okay?
If you look at some of the other companies that were attacked, they lose money on an hourly basis
for each hour of outage. And so, you know, obviously, they need to protect their stuff.
Okay? So, you know, it really depends on what you've got. Yes?
The attack on PayPal, I heard PayPal didn't even get kicked that much. Like, did you hurt them at all?
They were down for like how long?
I do not know. I'd have to go look at. So Panda Labs, if you're really interested in Operation Payback,
Panda Labs blog has a pretty good blow-by-blow action and counteraction and records of down times and things.
And you can go check it out. So it's just Panda Security and then go follow their blog link.
But their blogger is pretty good at keeping track of what the sustained outage was and sustained downtime. Yes?
I was going to think about the Xerxes tool.
So Xerxes. Okay. Yeah, yeah, yeah. I did the whole talk and I didn't talk about Xerxes.
So basically, there's a guy named Jester. He has this tool called Xerxes. Yes?
Oh, okay.
Yes. If you're in the room, you're pretty quick.
He was hacking the Islamic fundamentalist site last year.
Exactly. So really what it looks like his tool is, is it's a version of either Solaris or Rudy, which is, what do you do?
You send connections. It fills up the amount of servers that are available to listen and the connections never end.
So you basically reach the limit on how many servers you can have running and then it dies.
Yeah. So it's a pretty good tool. It looks like a set of scripts. He built the interface for it just for show.
It's, you know, it's really a bunch of little scripts running underneath of it. Yeah. Pretty decent. Pretty decent.
I think you could do, you could get similar results by using, by using Rudy and Tor. Okay. Anybody else? Yes?
You were saying to take down the CNC for the botnet, but that still seems a bunch of rental soldiers sitting on the battlefield.
So there's so many people aren't making new bots and just taking over existing architecture. Yes.
Specifically, when you research a lot of people who are powering the finance a little bit that already. Yeah.
It's like, oh, I'm in a hospital. I don't know, I'm in a mall. Yeah. Yeah. Taking down the command and control is a lot more critical for, you know, activists.
I understand, but the problem is that the command and control could be anything. It could be Twitter, it could be IRC. That is correct.
They've shown that they can use CNC from any platform. It's not the solution. Okay.
Especially considering that a lot of these things are short duration because they eventually get cleaned up. Right?
So you're here at ISP and you're looking and you're like, wow, we've got 3000 subscribers and, you know, 2000 of them are using 95% of our bandwidth.
What the heck? And you start to go, look, here's why. Okay. So, so eventually they will get cleaned up, especially if you continue to deliver that traffic.
So, you know, it's a combination of things. It's protecting yourself and then it's taking down the command and control that they're using. Okay.
Anybody else? All right. Thanks folks. Clean up.
Okay. Thanks to Marcus and Tiffany.
Thank you all for coming. And as we're wrapping up here, we want to give a couple of thanks to some sponsors, both exploit hub and recursion ventures have made donations to the Charitable Organizations that Marcus and I have chosen.
We're going to make some donations to Hackers for Charity and also the Community for Creative Nonviolence in Washington, DC homeless shelters in our nation's capital.
So we're going to be making some donations to them. But on the other hand, we, we've kind of broken even with the fluid here. However, the chairs are something that if you are happy about DogePong, really enjoyed it.
We're going to be putting it up on the site. If you can, if you're interested in paying for the chairs that you sat on during this presentation and this conference, put up some money.
Either put in the cauldron that's out near the front door or go online and do it if you're watching online and you're enjoying these presentations.
For those of us that are still here for logistical purposes, and Marcus is going to close out, I just want to say what we're going to do is I need a group to help me take out the trash.
Because like I said, we are the janitorial service for today. And for the chairs, those of you who are in the folding chair area, if you could take your chair you're sitting on, fold it and put it by the red door in the back of the room.
We'd be very appreciative. If you're sitting on a chair that has any kind of comfortable padding, that's ours. It belongs to the Hacker space. It's the uncomfortable ones we rented that need to be returned.
And let me pick that tomorrow. We also have some adult beverages in the back that we need to get rid of before school starts in this space tomorrow.
And I also have a huge pool of beer. Yes, a huge pool of beer. Now please take a bottle of beer on your way out, not with the group you're driving in Virginia.
But we need to get rid of some of this alcohol before tomorrow.
Okay everyone, thanks for coming. Just a couple of things, real quick.
Mic!
Is that?
Okay, she mentioned the PayPal link. I put a PayPal link on the homepage.
I really, ironically, I'm supposed to be somewhat of a business man, but I hate asking for money.
But if you could please stop the odd with that. I want to give a special thanks to Mike and Bob.
The last two speakers actually were in, what was it, Mountain View, California yesterday?
They were actually flew in from Mountain View to be here. They actually did base rest and dojo cutting in one weekend.
Woo!
That's awesome.
That's it.