Avi Freedman discusses the use of big data to cope with the increasingly large scale DDoS attacks. If you weren’t aware of just how “big” DDoS has gotten, the recent attack on Dyn (hopefully) serves as a wake-up call. Within the last month we’ve seen multiple 500 Gbps+ attacks launched by competing IoT botnets. DDoS is now hyperscale! So if DDoS is so big, why are defensive solutions so small? By small, I mean based on relatively limited, single server architectures, rather than on cloud-scale technology. After all, if you search today for any sort of DDoS defence solution, you’re going to be looking nearly exclusively at a set of physical appliances. Even cloud-based DDoS services are based on stacks of appliances, just operated at service provider PoPs. One reason is there’s no practical way around using ASICs and network processors to perform the variety of packet and traffic flow inspections needed to “scrub” IP traffic clean of DDoS packets at high bit rates. However, scrubbing internet traffic of the bad stuff is just one half of the DDoS defence story.  Before you scrub, first you have to find the bad stuff . And the detection layer is where the “smallness” of traditional DDoS protection approaches has reached the end of the road. Appliance-based DDoS detection has hit its ceiling In the out of band DDoS protection architectures which are most common today, a detection appliance receives traffic summaries (NetFlow, sFlow, IPFIX) and BGP routing data detects attacks based on that inbound data, then signals to mitigation layers to scrub the traffic in question. The problem with this isn’t necessarily the overall architecture, but the detection appliance’s compute and storage limitations. A multi-core CPU with NxGB of RAM and some TB of storage is a lot of power for a laptop, but not so much when dealing with huge volumes of traffic flow data. It takes most of the compute power just converting binary wire to text/numeric data. So a ton of compromises must be made in analysing the data to detect attacks, leading to fairly substantial inaccuracies. Big data helps DDoS detection sccuracy The application of big data to DDoS detection is transformative for accuracy, based on two factors. The first factor is how comprehensively the data is examined. For example, to perform any kind of baselining, it’s common for appliances to have to segment traffic flow data based on which router exported the flow records. So let’s say a host IP is being hit by a DDoS attack, but it’s coming in via multiple routers. Instead of seeing a large bump of network-wide traffic going to that host, the detection appliance will see a small bump of traffic across several routers — none of which will trigger any alert or mitigation. A big data approach doesn’t have the computing constraints, so it can always look at network-wide traffic, and so it will naturally notice attacks that would otherwise get missed. The second factor has to do with automation. With compute-constrained appliances, administrators either have to manually configure and maintain many individual IP addresses to baseline, or worse, configure cumulative baselining against a CIDR block, which severely dilutes accuracy. With big data scale, it’s possible to have an adaptive approach to baselining, where the system continuously figures out the set of IPs that are “interesting” based on how much total traffic they’re receiving within a given segment of time, then baselines and evaluates them for anomalies. Overall, big data capabilities have proven to increase DDoS detection and mitigation accuracy by 30 percent or more. Of course, just knowing that big data helps doesn’t mean it’s necessarily easy to achieve. Not all of the many big data platforms and technologies are suitable for DDoS detection, and not all IT or network teams have time and expertise to build a system. Some keys to building big data-powered DDoS detection are to ensure that the system can ingest streaming flow data at high rates; plan sufficient storage to retain data for a relatively long period of time to allow for network-wide anomaly detection; and allow for ad-hoc queries so that there is flexibility both in detection policies as well as forensic analyses to cope with both known and zero-day exploits.  Despite these challenges, the good news is that big data technology, platforms and expertise are proliferating. DDoS is hyperscale, but big data can help defensive strategies scale to meet the challenge. Source: http://www.scmagazineuk.com/the-big-data-era-for-ddos-protection-has-arrived/article/569500/

See the article here:
The big data era for DDoS protection has arrived