Yesterday’s Hotmail, Outlook.com, SkyDrive outage? Hotmail got too hot

Mar 13, 2013 11:47 pm by Kip Kniskern | 5 comments

Some Hotmail.com, Outlook.com, and SkyDrive services were down for as much as 16 hours yesterday, with an outage that started at 1:35pm PDT and wasn’t fully restored until this morning at 5:43 am PDT, according to a blog post by Microsoft Corporate Vice President Arthur de Haan on the Outlook blog.  The outage, according to a Microsoft spokesperson (as quoted by GigaOm), “affect(ed) a small number of users’ access to Hotmail and Outlook.com, but was serious enough to warrant the blog post today.

de Haan explained the outage in a “root cause analysis”:

On the afternoon of the 12^th, in one physical region of one of our datacenters, we performed our regular process of updating the firmware on a core part of our physical plant. This is an update that had been done successfully previously, but failed in this specific instance in an unexpected way. This failure resulted in a rapid and substantial temperature spike in the datacenter. This spike was significant enough before it was mitigated that it caused our safeguards to come in to place for a large number of servers in this part of the datacenter.

These safeguards prevented access to mailboxes housed on these servers and also prevented any other pieces of our infrastructure to automatically failover and allow continued access. This area of the datacenter houses parts of the Hotmail.com, Outlook.com, and SkyDrive infrastructure, and so some people trying to access those services were impacted.

Now we’re not datacenter experts (we have enough of a time keeping our one server running), but a firmware update that causes “a rapid and substantial temperature spike”, and one that needed “a mix of infrastructure software and human intervention” to bring the datacenter back online sounds a bit ominous.  Of course, for those affected (our numerous Hotmail.com accounts didn’t seem to be impacted), just being down for up to 12 hours was probably ominous enough.  Still, service was restored, everyone has all their data back online, and nothing (supposedly) caught on fire.  We can be grateful for that, at least.

This entry was posted in News and tagged Data Center, Hotmail, Outlook.com. Bookmark the permalink.