This past week Github suffered the biggest DDOS attack of all time (up until now)

github Incident Report
https://githubengineering.com/ddos-incident-report/

Prolexic report (Division of Akamai which mitigated the attack)
https://blogs.akamai.com/2018/03/memcached-fueled-13-tbps-attacks.html

Wired.com report
https://www.wired.com/story/github-ddos-memcached/

Technical description and mitigation methods for the “UDP amplification attack” aka “The memcached attack” on Github

https://blog.cloudflare.com/memcrashed-major-amplification-attacks-from-port-11211/

The size of the DDOS, approx. 1.35 Tbits/sec was approx. twice the
size of the largest DDOS in 2016, the infamous Mirai botnet attack.

For those who have never heard of memcache, it’s one of a family of automatic caching spps (others include RabbitMQ and Redis) which are commonly deployed between nodes in a distributed application. Because due to various possible reasons, a calculation or result can be returned earlier than the recipient node is ready, a caching node running one of these apps can free up the worker nodes for additional work.

I’ve seen this deployed sometimes between a webserver and its database server on different machines, and in the Technical Forums I remember it was deployed in an Owncloud/Nextcloud setup. This is commonly seen in Big Data compute networks like Elastic and Hadoop. Anyone running one of these kinds of networks should re-inspect their caching nodes to make sure they won’t be made a part of one of these attacks.

TSU

Hello

Have never heard of memcache. What are the reasons such things can happen?
Thanks.


James McCoy
Equipment Operator at Francis Drilling Fluids

And these attacks have showed up before. It’s a kind of an amplification attack, right? That majorly causes overwhelming of a target server?

@jamesmccoys
Apps like memcache are used mainly for

  • When you have distributed apps deployed on different network devices and particularly in mesh networks and data farms where data might be retrieved from multiple sources or processing a request might be done across multiple nodes, some requested data might arrive sooner than other data to complete the request. To maximize efficiency, reduce waiting and idle nodes, applications like memcache can automatically store early data so worker nodes can accept new tasks.
  • As long as data is passing through the node running memcache, it’s possible to apply transformations (alter the data, typically adding metadata) to your advantage, for instance applying labels that enhance the ability to later find, sort, categorize and otherwise use the data.

In plain words, apps like memcache can make your distributed application (application whose parts are deployed on different networked machines) run faster and better.

Yes, it’s a particular amplification attack, but not highly leveraged like DNS DDOS amplification attacks.
But, the possibility of sheer numbers against an exposed memcache port can also have major consequences.

This following is my general description of why this attack makes memcache and similar apps most vulnerable…

As an extreme performance enhancing app, little attention is paid for implementing better security like SSL because the overhead for encryption/decryption/authentication is extremely significant when you’re chasing incremental performance on a very large scale. That is why at best IP filtering is the only way to block intrusions, and even that might be compromised under the right circumstances. Apps like memcache really should not be exposed to the Internet, period… and the entire network isolated from the Internet (ie no public IP addresses and no forwarding to this network). But, for various reasons better network design is not always followed and so apps with very weak security can be exposed either through ignorance or because the network wasn’t designed properly (we’re not always able to control every aspect of our jobs).

TSU