Update on Tucows Services Outage - FINAL UPDATE
by
Chris Scott
on Thu 04 May 2006 08:14 AM EDT
This article has been rewritten now that we have a better understanding of this outage.
Overview:
At approximately 12:00 PM ET yesterday (5/3/06), a domain hosted on the Tucows Managed DNS service which we resell was the target of a distributed denial of service (DDoS) attack. This caused a DDoS on the Managed DNS service as well as the rest of the Tucows network in this datacenter and some upstream providers. Due to this, other Tucows services we resell were also affected: Blogzerk, Domain Management, and Email Defense.
Tucows and their upstream network providers took steps yesterday to help mitigate the network DDoS which resulted in Domain Management and Blogzerk being back on line late yesterday afternoon with some intermittent slowness. The domain that was the target of the attack was removed from the Managed DNS service and the Managed DNS servers were renumbered. These changes began to take effect last night and the Managed DNS service was back to normal this morning at approximately 9:45 AM ET.
What Happened:
For Email Defense customers, any incoming mail would not have been received during this outage. No email was lost. Most email would be deferred and retried at a later time by the sending server. Some email may have been bounced back to the sender--this depends on the retry policy of the sending server.
Note that most mail servers use an exponential backoff when retrying queued mail. This means that the delay between retries increased with each attempt. Due to the long nature of this outage, mail sent to your domain yesterday may still be queued for delivery on the sending server. If you have not received mail sent yesterday, this is likely why. You can either have the sender, resend the email or, if possible, have their administrator force sending the mail in the queue without waiting for the next retry time. This may also result in message sent earlier actually arriving later than messages sent at a later time during the outage.
For Blogzerk customers, the blogs and the publisher's control panel were down during this outage. The blogs were back up yesterday evening but were still seeing some intermittent slowdowns until this was resolved.
For Managed DNS domains using this service were not resolving until approximately 9:45 AM this morning. Please note that normal hosting customers are not using Managed DNS. The only customers using this are those that have explicitly requested it, typically to do URL forwarding.
Next Steps:
Tucows was already in the process of upgrading their hardware and bandwidth to better handle future growth and attacks like this. This work will continue and accelerated where possible.
