Redundancy is Key to Successful Blizzard Preparedness
Does your CEO shovel snow off the roof in the middle of a blizzard? That’s just what Datto CEO, Austin McChord, did Saturday morning as the Blizzard of 2013 barreled down on Datto’s HQ in Norwalk, Connecticut. Snow accumulated on the roof of HQ and shut down the air conditioners located on the building’s roof. The air conditioners are needed to cool Datto’s server room, so it’s imperative that they continue to operate. While that’s not the type of disaster recovery we’re used to at Datto, Austin’s hands on approach didn’t surprise anyone.
Winter Storm Nemo, so named by the Weather Channel, dumped a total of 40 inches of snow in parts of Datto’s home state of Connecticut. The entire Northeast experienced wild weather, including hurricane-force winds; the strongest winds were recorded in Cuttyhunk, Mass. at 83mph, followed closely by Westport, Conn. with 82 mph. In total, the storm left 650,000 homes and businesses without power.
Over 700 Datto devices were affected by the power outages. The good news is, most customers who experienced an outage got their power back before Monday morning. Only a handful of customers required emergency assistance from Datto to get back up and running, and this happened quickly. The advantage of a weather disaster is that there is time to prepare. Most Datto Partners were able to take actions prior to Nemo, so their clients would not be drastically affected. Many of our partners took their Datto device to another location they knew would not lose power. If needed, they could then virtualize the server(s) locally on the device to continue the operations of the business.
For Datto’s Director of Technical Support Victor Mathieu, “the scary disasters are those you can’t prepare for, like a tornado or earthquake.” At this point, Victor and his team are more concerned about disasters such as the tornadoes in Mississippi and Alabama. These instances come without warning and can have serious devastating effects, which can make them more harmful to businesses in their wake, since there is no chance to prepare.
We learned a few things from Nemo. One is that you shouldn’t take weather predictions from a groundhog—as Punxsutawney Phil recently predicted an early spring. Another is that as prepared as you think you may be, things can still go wrong. Given the nature of our business, Datto has multiple layers of redundancy to be as prepared as possible.
In anticipation of Nemo, part of Datto’s preparedness planning included:
- Sending a satellite team of Datto Support to Datto’s east coast data center, in case of power outages, loss of internet connectivity or generator failure at HQ
- Key members of Datto Technical and Customer Support spent Friday night at HQ so that Datto would have reps on-site Saturday morning to assist Partners
- Datto does not close operations for natural disasters
- All Partners located in the vicinity of the storm received priority Datto Technical Support
- Communication with all Partners prior to the storm, recommending ideas to mitigate data loss and downtime
Granted, our original storm preparedness did not account for our CEO and another Datto employee shoveling snow from the roof of our office building to get the AC unit working. It proves that while you may have a complete disaster plan laid out, you always have to allow for the unknown. And yes, if the shoveling didn’t work, we had another plan lined up (a portable AC unit).
If you have a good story of how you or your client tackled Nemo, please share it: firstname.lastname@example.org.