Addition of new servers led to massive outage last week: AWS

Nov 29, 2020
New Delhi: The addition of new servers to the network triggered the massive outage of Amazon Web Services (AWS) last week, affecting several websites and apps, the company has acknowledged.

AWS, which is the Cloud arm of e-commerce behemoth Amazon, said they were able to confirm a root cause which was not driven by memory pressure as earlier reported.

“Rather, the new capacity had caused all of the servers in the fleet to exceed the maximum number of threads allowed by an operating system configuration,” the company said in a technical blog post.

“As this limit was being exceeded, cache construction was failing to complete and front-end servers were ending up with useless shard-maps that left them unable to route requests to back-end clusters,” it added.

The capacity addition was being made to the front-end fleet of AWS.

Each server in the front-end fleet maintains a cache of information, including membership details and shard ownership for the back-end clusters, called a shard-map.

On Thursday morning, the multi-hour outage affected a large portion of the Internet.

Several apps and services posted on Twitter that they were experiencing problems with the AWS services.

Some of those were Acorns, Adobe Spark, Autodesk, Coinbase, Glassdoor, Flickr, iRobot and The Washington Post, among others.

Amazon said it mitigated the impact to the subsystem within its Kinesis Data Streams APIs and other dependable services — responsible for the processing of incoming requests.

“We continued to slowly add traffic to the front-end fleet with the Kinesis error rate steadily dropping from noon onward. Kinesis fully returned to normal”.

Amazon Kinesis enables real-time processing of streaming data. In addition to its direct use by customers, Kinesis is used by several other AWS services. These services also saw impact during the event.

Outside of the service issues, AWS experienced some delays in communicating service status to customers during the early part of the outage.

“We want to apologise for the impact this event caused for our customers,” AWS said.IANS