Buckets of inspiration from a flooded data center

With only the very recent passage of aid for Hurricane Sandy recovery, it’s a fierce storm that lingers on my mind. When Sandy came up the east coast last October, I followed the news from two perspectives.

As a DC resident, I felt very fortunate to have escaped damage. The overall destruction and heartache this storm caused is still hard to believe. Storms like this create heroes – the first responders and others who risk their lives to help others.

As a technology professional, I was also sympathetic to those trying to keep services running. I found one inspiring technology story about a bunch of people who don’t qualify for hero status, but they went above and beyond in their own way.

The servers at Peer 1 Hosting, a data center in lower Manhattan, keep about 100 high-tech companies and their websites up and running, but when Sandy hit shore, everything that could go wrong did. First, power was cut off to their building. Then, the storm flooded the basement and their generators. Their last hope was the backup generator ... on the roof.

Gerry Smith at The Huffington Post wrote: “As Peer 1 began notifying customers that its facility may lose power, employees from Squarespace and Fog Creek [two Peer 1 clients] arrived to see if they could help.” Some of Peer 1’s independent contractors and friends of employees also came to help.

buckets.jpg

30 people formed an old-fashioned bucket brigade. Each person carried two 5-gallon buckets filled with diesel fuel up several dark flights of stairs before handing them off to the next person, who would continue upward. The buckets eventually made their way up 17 slippery flights to the roof, where the fuel was poured into the back-up generator.

“The diesel-filled jugs were heavy, and hauling them up the stairs was physically draining,” said a software developer at Fog Creek. “Especially because we’re not all that fit.”

They did this from Tuesday afternoon until Wednesday morning. Each time they filled the generator, they could rest until they heard the low-fuel alarm, and then the bucket brigade went back into action.

The amazing efforts of the Peer 1 employees and their clients staved off disaster and kept servers and services running with little or no downtime. All of us here at DelCor salute these technology heroes.

This story also makes me think about geographic redundancy, which I’ll address in another post later this week. Stay tuned!

photo credit: theenmoy