On October 21, 2019, AWS suffered a DDoS attack and several of their services experienced a partial outage that lasted just over 8 hours and brought down a large portion of the internet. Amazon confirmed that a DDoS (Distributed Denial of Service) attack caused the interruption in its services, leaving a large number of websites and applications inaccessible or partially accessible while they attempted to mitigate the issues.
“It is quite possible that yesterday’s attack on Amazon was much larger than that of GitHub’s; we will have to wait for the technical details.” -Rakesh Wagh via LinkedIn
To give further context, the attack functioned by bombarding Amazon’s DNS service, Route 53, with queries causing congestion as Route 53 attempted to resolve DNS requests to the correct IP addresses. This prevented many from reaching the EC2 instances and S3 buckets where applications, websites, and many backend services are hosted. The attack also affected several of their other services including Elastic Load Balancing (ELB), Relational Database Service (RDS), Elastic Cloud Compute (EC2).
In addition to the internal problems, the DDoS attacks had a major effect on Digital Ocean, their status page reading, “Our engineering team is continuing to monitor the issue impacting accessibility to S3/RDS/ELB/EC2 resources across all regions. We continue to monitor the situation closely, and we will post an update as soon as the issue is fully resolved” during the attack, causing even more trouble for Developers.
Due to the number of different sources used in DDoS attacks, it’s incredibly difficult for services like Route 53 to determine legitimate traffic. In this particular situation, Amazon’s own DDoS-mitigation platform, Shield Advanced, contributed to the problem when it began flagging legitimate queries, adding to the outages experienced by businesses.
According to many IT professionals who took to Twitter and LinkedIn during the attack, it had to have been fairly large, such as Rakesh Wagh who compared it to the DDoS attack on GitHub in February of 2018.
“Last year, a 1.3Tbps DDoS attack pummeled GitHub for 15-20 minutes. Think about it – a server was bombarded with 1.3Tb of data each second!” Rakesh said in a LinkedIn post he made about the attack, “It is quite possible that yesterday’s attack on Amazon was much larger than that of GitHub’s; we will have to wait for the technical details.”
Amazon still has not provided any technical details about this attack, but hopefully, we will receive some soon. Considering the massive inconvenience this created for many systems admins, including one we spoke to who asked to remain anonymous, but she rather appropriately summed up the frustration by saying:
“Well, it sucked because I couldn’t deploy code (that I had written hours before) to our test environment. It doubly sucked because our users were waiting on this change! And it triply sucked, because it affected other services that we use like Digital Ocean.”
Why is this important?
By resolving domain names to the correct IP address, your DNS is the key to your customers accessing your products, services, and applications. If you’re looking for best practices, we’ve written a guide on how to set up your DNS so that it will be protected in these situations.
While this alone should be a significant wake-up call to anyone whose site or application went down that it might be time to rearchitect your DNS to protect against a DDoS attack, the sheer scale and the number of AWS customers affected immediately reminded us of the infamous DDoS attack on Dyn back in October of 2016. The 2016 attack spawned the DNS series we wrote in 2017, and updated in Spring this year. You can read our DNS eBook below, and we would also suggest checking out the infographic which outlines data on just how many sites don’t use multiple DNS providers:
What you might not know right now, is that these attacks are on the rise, and they’re getting better. For a list of the DDoS attacks in 2018 alone, SecureList provides quarterly overviews of DDoS activity and CloudFlare has an article about some of the more famous DDoS attacks. But even before this attack, plenty of cybersecurity groups agree that DDoS attacks are more common and getting better.
Now more than ever, it’s important to make sure you have your DNS with multiple providers because as Morgan Lucas, an IT blogger rather aptly states in his article about the Amazon DDoS attack: “It’s Always DNS.”