皆様
US-EAST続きですが、一部のEC2に影響のあった障害が、AZの一つで発生しました。
先週のAZとは別のゾーンとのことで、すでに復旧済みです。
影響範囲はAZ内でも更に小規模で、30分程度。EBS等他のサービスへの影響はなかった模様ですが、日本の皆様で本障害に関するトラブル等があった方は、状況について投稿いただけますと助かります。
以下、Service Health Dashboard：
http://status.aws.amazon.com/に掲載されている経過報告の写しです。
======
May 12, 7:15 PM PDT We wanted to give some additional information about Tuesday's event. At 12:05pm PDT, power was lost to a small number of instances in a single Availability Zone in the US East Region. Power was fully restored by 12:35pm PDT, and by 1:08pm PDT the majority of affected instances had fully recovered. This was a different Availability Zone than the ones that experienced problems last week.
Tuesday's event was triggered when a vehicle crashed into a high voltage utility pole on a road near one of our datacenters, creating a large external electrical ground fault and cutting utility power to this datacenter. When the utility power failed, most of the facility seamlessly switched to redundant generator power. One of the switches used to initiate the cutover from utility to generator power misinterpreted the power signature to be from a ground fault that happened inside the building rather than outside, and immediately halted the cutover to protect both internal equipment and personnel. This meant that the small set of instances associated with this switch didnt immediately get back-up power. After validating there was no power failure inside our facility, we were able to manually engage the secondary power source for those instances and get them up and running quickly.
The switch that failed arrived from the manufacturer with a different default configuration than the rest of our switches, which caused it to misinterpret this power event. We have already made configuration changes to the switch which will prevent it from misinterpreting any similar event in the future and have done a full audit to ensure no other switches in any of our datacenters have this incorrect setting.
Because the majority of the equipment failed over correctly, this event affected a small subset of the instances in the Availability Zone, and no other Availability Zones were affected. Amazon EBS volumes were also not affected, and many impacted customers were able to launch replacement instances to quickly recover. We understand disruptions to instances can impact customers' applications, and we continue to take the learnings from unexpected events such as these and feed those back into our standard procedures so that we can mitigate the impact if something like this transpires again.
====
小島
Locked
