Month: August 2016

by Roland Dobbins, Principal Engineer & Kleber Carriello, Senior Consulting Engineer

When organizing a huge, high-profile event like the Olympics, there are always chances for things to go wrong – and, given human nature, we tend to simply accept it as a given when things go as planned, and to notice and highlight difficulties in execution.

A great deal has been written and spoken about the challenges facing the organizers, sponsors, and contestants in the 2016 Rio Olympics. And if we think about it, we can extrapolate potentially thousands of potential pitfalls and difficulties which accompany any event of similar complexity.

Success is Blasé

We’ve come to view Internet applications and services in much the same way. When they’re working well, we don’t even notice how amazing it is that we’re able to instantly view live streaming video of the Olympic competitions, along with scores and expert commentary, pretty much anywhere on the globe, on our computers, smartphones, and tablets. But if we somehow can’t get access to the latest and greatest content and information instantly – and share it and discuss it online with our friends – then we become intensely frustrated and vocal with our displeasure. The uninterrupted availability and resiliency of online information services, apps, data, and content is now de rigeur for sporting events of any size, at scale. This is manifestly true for the Olympic Games.

Yes, the Rio Olympics experienced – and largely overcame – significant challenges which at times seemed almost insurmountable. Many problems, some of them factual, some of them less so, have been described and discussed and dissected in excruciating detail.

Even before the opening ceremonies began, public-facing web properties and organizations affiliated with the Olympics were targeted by sustained, sophisticated, large-scale DDoS attacks reaching up to 540gb/sec – directed towards public-facing properties and organizations affiliated with the Olympics. While many of these attacks were ongoing for months prior to the start of the Games, attackers increased their efforts significantly during the actual Olympics themselves, generating the longest-duration sustained 500gb/sec-plus DDoS attack campaign we’ve observed to date.

And nobody noticed.

This is the sine qua non of DDoS defense – maintaining availability at scale, even in the face of skilled, determined attack. And just like the countless other services we rely upon every day such as electricity, fresh water, transportation, and emergency services, the ultimate metric of success is that the general public can go about their business and pursue their interests without ever knowing or caring that titanic virtual struggles are taking place in the background.

By any metric, the Rio Olympics have set the bar for rapid, professional, effective DDoS attack mitigation under the most intense scrutiny of any major international event to date. And did we mention that the attacks ranged up to 540gb/sec in size?!

An Ongoing Attack Campaign, Expanded

Over the last several months, several organizations affiliated with the Olympics have come under large-scale volumetric DDoS attacks ranging from the tens of gigabits/sec up into the hundreds of gigabits/sec. A large proportion of the attack volume consisted of UDP reflection/amplification attack vectors such as DNS, chargen, ntp, and SSDP, along with direct UDP packet-flooding, SYN-flooding, and application-layer attacks targeting Web and DNS services. The IoT botnet utilized in most of these pre-Olympics attacks was described in detail in a recent weblog post by our Arbor ASERT colleague Matt Bing. This very same botnet, along with a few others, was also used to generate the extremely high-volume (but low-impact, thanks to the efforts of the defenders!) DDoS attacks against an expanded list of targets throughout the 2016 Rio Olympics.

One of the characteristics of information security in general, and DDoS defense in particular, is that we see new attack methodologies pioneered by more skilled attackers and used sporadically for years (and sometimes decades) before they’re ‘weaponized’ and made more broadly available to low-/no-skill attackers via automation. We’ve encountered various types of high-volume/high-impact reflection/amplification attacks since the late 1990s; and then, 3 1/2 years ago, they suddenly became wildly prevalent due to their inclusion in the arsenal of DDoS botnets-for-hire and so-called ‘booter/stresser’ services. This has led to a highly asymmetrical threat environment which favors even the most unskilled attacker due to the fact that these Internet ‘weapons of mass disruption’ are now available to the masses via a few mouse-clicks and a small amount of Bitcoin. We’ve seen this pattern repeat itself over and over again, with disparate groups of miscreants totally unaffiliated with one another independently rediscovering more sophisticated attack mechanisms, and then proceeding to weaponize them with nice GUIs and even 24/7 online ‘customer’ support!

Everything Old is New Again

For the relatively small number of people who have a reason to think about how the Internet actually works, the only protocols they tend to remember are TCP, UDP, and ICMP. Since those protocols represent by far the largest proportion of Internet traffic, little if any thought is given to other IP protocols.

In reality, there are 256 Internet protocols, numbered 0-255. TCP is protocol 6, UDP is protocol 17, and ICMP is protocol 1. On the IPv4 Internet, only 254 of those protocols should ever be observed – protocol 0 for IPv4 (but not for IPv6!) is reserved, and should never be utilized, even though routers and layer-3 switches will happily forward it along. Protocol 255 is also reserved; most routers and switches won’t forward it. Of the set of less-familiar IP protocols, Generic Routing Encapsulation (GRE), used for unencrypted ad-hoc VPN-type tunnels, is protocol 47.

Starting in late 2000, we began to observe more skilled attackers occasionally using these lesser-known protocols in DDoS attacks – almost certainly in an attempt to bypass router ACLs, firewall rules, and other forms of DDoS defense which were configured by operators who only took TCP, UDP, and ICMP into account. In many cases, these attacks initially succeeded until the defenders finally inferred what was going on, generally via analysis of NetFlow telemetry using collection/analysis and anomaly-detection systems such as Arbor SP.

Example crafted GRE DDoS attack packet.

And now we’ve seen those same attack techniques rediscovered, weaponized and utilized during the Rio Olympics. In particular, significant amounts of GRE DDoS traffic was generated by the attackers; this ‘new’ attack methodology has now been incorporated into the same IoT botnet referenced above. As with all ‘new’ types of DDoS attacks the miscreants stumble upon, we expect to see other botnets-for-hire and ‘booter/stresser’ services adding GRE to their repertoires in short order.

We also observed uncomplicated, high-volume packet-floods destined for UDP/179. As most (not all) UDP reflection/amplification attacks tend to target UDP/80 or UDP/443 in order to confuse defenders who might not notice that the attackers are using UDP instead of TCP (TCP/80 is typically used for non-encrypted Web servers, and TCP/443 for SSL-/TLS-encrypted Web servers), we believe the attackers were attempting to masquerade an attack on the BGP routing protocol used to weave Internet-connected networks together. BGP runs on TCP/179; the irony is that one of the few best current practices (BCPs) actually implemented on a significant proportion (not all!) Internet-connected networks is to use infrastructure ACLs (iACLs) to keep unsolicited network traffic from interfering with BGP peering sessions.

DDoS Defense Gold – It’s All About Teamwork, Especially at the Olympics

The defenders of the Rio Olympics’ online presence knew they’d have their work cut out for them, and prepared accordingly. A massive amount of work was performed prior to the start of the games; understanding all the various servers, services, applications, their network access policies, tuning anomaly-detection metrics in Arbor SP, selecting and configuring situationally-appropriate Arbor TMS DDoS countermeasures, coordinating with the Arbor Cloud team for overlay ‘cloud’ DDoS mitigation services, setting up virtual teams with the appropriate operational personnel from the relevant organizations, ensuring network infrastructure and DNS BCPs were properly implemented, defining communications channels and operational procedures, et. al.

And that’s why the 2016 DDoS Olympics were an unqualified success for the defenders! Most DDoS attacks succeed simply due to the unpreparedness of the defenders – and this most definitely wasn’t the case in Rio.

The stunning victory of the extended DDoS defense team for the 2016 Rio Olympics demonstrates that maintaining availability in the face of large-scale, sophisticated and persistent DDoS attacks is well within the capabilities of organizations which prepare in advance to defend their online properties, even in the glare of the international spotlight and an online audience of billions of people around the world. The combination of skilled defenders, best-in-class DDoS defense solutions, and dedicated inter-organizational teamwork has been proven over and over again to be the key to successful DDoS defense – and nowhere has this been more apparent than during the 2016 Rio Olympics.

Related