matthew

April 11, 2011 @ 11:44 AM via web

Last night around 8:00pm PDT a large denial of service attack was launched against a CloudFlare customer. Unlike most DDoS attacks we see, this one targeted the DNS infrastructure and used recursive DNS providers to amplify the attack. The attack appears to be largely emanating from China and the site being attacked is Chinese in origin. We isolated the attack to one data center (San Jose). While the attack affected the western United States and parts of Asia, the rest of the world's traffic has been unaffected.

In San Jose, we have rate limited the number of DNS requests based on the requesting IP address. This is a somewhat crude solution which, unfortunately, blocks some legitimate DNS requests. To counteract this, we have increased the TTLs on domains that are not under attack so legitimate requests will get cached by upstream providers. This means that as soon as one DNS request is resolved for a legitimate domain, it should remain in the upstream provider's cache until after the attack is under control.

We are also working with the registrar of the domain under attack to fail it off of CloudFlare. Unfortunately, unlike most attacks, since traffic is pointed at CloudFlare via the upstream registrar, we cannot simply fail this type of an attack off our network without the cooperation of the registrar or the client. We have also arranged a conference call with the owner of the domain and a Chinese translator in order to get the domain moved off our system. We have contacted many of the largest recursive DNS providers to have them block requests to this domain and then whitelist their IPs so their legitimate requests are allowed through. Our team is also driving additional hardware we had been testing in our office to San Jose to help offset the load caused by the attack. We are also redistributing the DNS clusters to a broader range of IP addresses so we can rate limit the affected name server domains without affecting as many users. These changes will come online within the next hour.

The whole team has been working non-stop since the attack began last night to mitigate it. While we have been able to isolate the attack to only a subset of users, the fact that it is causing any spillover effects is unacceptable. We will get this under control and then make changes to our DNS deployment in order to ensure this kind of attack does not cause harm like this in the future.

In the meantime, if you are experiencing DNS problems and you need to switch your name servers away from CloudFlare you can do sure assured that we will keep your settings. Through your DNS Settings page you can export your Zone file in a format that can be uploaded to any other DNS provider. Your account at CloudFlare will remain intact and we can help you reenable it when this problem has subsided.


Two updates: one short term, one long term.

Short term: we have gotten in touch with the correct person at the domain's registrar and convinced them to remove CloudFlare as the domain's authoritative DNS provider while the attack is ongoing. This allows us to get all our regular users back to a normal state. The attack traffic is still coming in as the DNS TTLs for the attacked domain expire, but it looks like things are headed in the right direction and should improve quickly for those users still experiencing issues.

Long term: beginning this week a massive new deployment of servers begins to ship out to our 5 existing data centers and 9 new data centers. We've been planning this for the last three months but the timing is apt. We're adding a bunch of new servers in our existing data centers (Tokyo, San Jose, Chicago, Ashburn, Amsterdam) and adding new presence over the next 30 days in Hong Kong, Los Angeles, Dallas, Miami, New York, Frankfurt, and Paris. Shortly after that, we plan to bring Singapore and London online. Had this new infrastructure been in place, the attack from the last 12 hours (which was big) would likely have been isolated to Hong Kong and, given the amount of new hardware we're deploying, I'm confident no one would have noticed it other than our support team who would have quietly dealt with it.

Thank you for your patience. While this was a painful attack, we are improving the process so these issues are stopped in the future, and every zombie machine that was used in the attack has been logged and will be stopped from harming other CloudFlare members going forward.