I’m curiously puzzled, but not entirely surprised, how a company such as Amazon (AWS) allowed its servers to be interrupted for any length of time due to severe storm damage in northern Virginia this past weekend. Companies using cloud servers are both expectant and dependent on being able to pull information from cloud sources to operate their businesses without interruption. After all, IT professionals have been preaching the security and reliability of the cloud for quite some time to manage large data off-site. Steps for Amazon to repair credibility should be transparent and swift.
Redundancy Issues 101
Failing to address possible redundancy issues early on in cloud infrastructure is a basic design-maintenance (101) issue, which can become extremely expensive.
- Did backup generator design properly address power load requirements in a long-term outage?
- Was (regularly planned generator testing) implemented for inclement weather situations, or other emergencies?
- An adequate and properly maintained generator is the first line of defense in outages and should be highest on the maintenance list.
- What surge-protection plans were implemented for grid-spikes which can disrupt and even destroy electronic equipment, or interfere with telecommunication operations
This issue, reported by news outlets as a downtime factor, should be the addressed openly and honestly with constituents early on. Credibility and believability in cloud support 24/7, is at stake.
Utility Service Provider Design
Obviously the external electric grid design plays a role in any outage, no matter their cause. Early planning stages in design and control are key factors in redundancy, efficiency and reliability.
- Did through collaboration exist between utility and customer in facility design process?
- What redundancy features did utility provide in design phase of cloud site?
- What site factors led Amazon to believe this area utility was capable of handling unforeseen outages through prevention techniques?
- How much utility infrastructure is above-ground vs. underground therefore susceptible to damage by weather or other contingency factors?
- Are back-up substations available to redirect power if local grid goes down?
- What is the utilities track-record on outages, repairs to infrastructure, and down-time?
- Where does the cloud site stand in the hierarchy of restoring service, high, medium, low?
Off-Site Redundancy-Backup Facility
Inherently, these type utility outages will occur due to a national infrastructure grid that is aging and vulnerable to costly disruptions. Above ground utility pole grid is notoriously aged and lacks design upgrades to protect critical areas from massive outages. This is a known fact which businesses must work around, building on site and off-site redundancy.
Amazon’s cloud services for the Eastern U.S. should have been automatically switched to a redundant system, such as its West Coast Operation. Why this did not happen is a mystery, but Amazon should own up to its design miscalculations and move to inform customers on future plans regarding eliminating down-time.
Image Repair – Epitaph
Move candidly and quickly to report steps to correct outage issues. Hire a PR Firm to manage media campaign designed to restore credibility. Hopefully this is not an epitaph for Amazon Cloud Services. The problem remains that many companies, not just Amazon, are risking their business operations on poorly designed redundancy. Since proper design and maintenance is not a revenue-driven expenditure, sadly it does not get the attention needed. It is a striking example of how an ambivalent policy can get organizations into embarrassing situations which, expectedly, get highly covered by media outlets.
This evaluation of what happened, and its causes, are introspections on Amazon Cloud Services site in Northern Virginia and does not reflect actual events at the time. It is an educated guess as to what could have happened based on public knowledge.