The Six-Month Time Bomb: When EFS Ran Out of Burst Credits

For almost six months, everything appeared to be working perfectly.

A web application was using Amazon Elastic File System (EFS) as shared storage. Multiple application instances could access the same files, deployment was straightforward, and there was no obvious reason to question the architecture.

Then, seemingly without warning, the website became extremely slow and eventually appeared to stop working altogether. The server load was unusually high, requests timed out, and application processes started piling up.

At first glance, it looked like a compute problem. It was not. The real cause was an exhausted EFS burst-credit balance.

Why the failure looked like a CPU problem

One of the misleading symptoms was the high Linux load average. A high load average does not necessarily mean that the CPUs are fully utilized. It also includes processes waiting in an uninterruptible state, which commonly happens while processes are blocked on storage I/O.

The application workers were waiting for EFS operations to complete. New requests continued to arrive, more workers became blocked, and the request queue grew:

EFS throughput became constrained.
File operations took longer.
Application workers remained busy waiting for I/O.
Incoming requests accumulated.
The system load increased.
The website eventually became practically unavailable.

Adding CPU capacity would not have solved the underlying problem.

Why it took six months to fail

The EFS file system was configured to use Bursting Throughput mode. In this mode, sustainable baseline throughput is related to the amount of data stored in the EFS Standard storage class. When the workload remains below that baseline, the file system accumulates burst credits. When throughput exceeds the baseline, those credits are consumed.

This creates a potentially deceptive situation for relatively small file systems:

The stored-data volume results in a low sustainable baseline throughput.
An existing credit balance temporarily permits much higher throughput.
The application performs normally even though its long-term workload exceeds the sustainable baseline.
The imbalance remains hidden while credits are available.
When the balance reaches zero, performance drops dramatically.

In this case, the accumulated credits lasted for approximately six months. The architecture was already undersized from a throughput perspective, but the credit balance concealed the problem. It was a slow-moving capacity problem—a six-month ticking time bomb.

Temporary burst performance must never be mistaken for sustainable throughput.

Choosing the appropriate EFS throughput mode

Bursting Throughput

Bursting Throughput can be appropriate when throughput naturally grows with the amount of stored data and periods of high activity are followed by quieter periods. It requires careful monitoring. A continuously declining credit balance shows that the workload is not sustainable in this mode.

Elastic Throughput

Elastic Throughput is generally better suited to unpredictable or strongly fluctuating workloads. Throughput automatically scales with demand, so the application does not depend on a finite burst-credit balance. This improves operational resilience, although its usage-based cost should be evaluated for workloads with consistently high throughput.

Provisioned Throughput

Provisioned Throughput is useful when the required throughput is known and higher than the stored-data volume would otherwise support. It provides predictable capacity independently of file-system size, but the configured level must be reviewed as demand changes.

It is also important not to confuse the EFS throughput mode with its performance mode. Throughput modes determine how throughput capacity is supplied; performance modes concern characteristics such as latency and parallelism.

Monitoring that should have been in place

The most important CloudWatch metric for an EFS file system using Bursting Throughput is BurstCreditBalance. A useful monitoring baseline should also include:

BurstCreditBalance to detect sustained credit depletion
PercentIOLimit to identify when the file system approaches its I/O limit
PermittedThroughput to understand currently available throughput
MeteredIOBytes and TotalIOBytes to observe I/O consumption
application latency, request duration, timeouts and error rates
processes waiting for I/O, correlated with load and actual CPU utilization

Monitoring only CPU, memory and HTTP status codes is insufficient. Shared storage is a critical application dependency and must be monitored as part of the service, not merely as an infrastructure component.

An alarm should trigger well before the credit balance reaches zero. More importantly, the trend must be evaluated. A balance that falls consistently over several weeks already demonstrates a structural throughput deficit, even if the absolute balance still looks comfortable.

What I would do differently

Estimate sustainable throughput during the design phase.
Choose the throughput mode based on the actual access pattern.
Treat burst credits as temporary headroom, not normal capacity.
Create CloudWatch alarms for credit balance and I/O limits.
Monitor the rate at which credits are being consumed.
Include EFS throughput and latency in load testing.
Correlate Linux load with CPU utilization and I/O wait.
Review whether frequently accessed files genuinely require shared network storage.
Test application behaviour under storage throttling and elevated latency.
Document the operational procedure for changing throughput modes.

Final takeaway

EFS can be an excellent shared-storage solution, but its behaviour depends heavily on the selected throughput mode and workload profile.

The dangerous part of this incident was not simply that performance degraded. The dangerous part was that the system worked well for six months before degrading. The burst-credit balance delayed the visible failure long enough to create confidence in an architecture that was not sustainable.

The best defence is not a larger credit balance. It is selecting the correct throughput mode, understanding sustainable throughput, and monitoring the metrics that reveal a growing imbalance before users notice it.