BK BERND KUHLENAll notes ↗
← Back to field notes

Why EFS Performance Can Degrade After Months

An Amazon EFS problem can be present long before it becomes visible. A small file system may serve a workload successfully for weeks or months while consuming a finite reserve of burst credits. The application looks healthy, dashboards stay green, and the architecture earns trust. Then the reserve reaches zero and the same workload encounters a much lower sustainable throughput ceiling.

This delay makes the failure feel sudden. Operationally, it is anything but sudden.

The hidden mismatch

With Bursting Throughput, baseline throughput grows with the amount of data stored in the EFS Standard storage class. AWS documents a baseline rate of 50 KiB/s per GiB, with every file system receiving at least 1 MiB/s of metered throughput. Work performed below the baseline earns credits; work above it consumes them.

The dangerous pattern is a workload whose long-term average is slightly above that baseline. It may not create an immediate incident because previously accumulated credits pay the difference. The balance, however, trends downward every day.

Why the symptoms point in the wrong direction

When storage calls take longer, application workers wait. On Linux, processes blocked in uninterruptible I/O wait contribute to load average, so a storage bottleneck can resemble a compute-capacity problem. Requests queue, worker pools fill and timeouts spread through the service even when CPU utilization is not the root cause.

Adding instances can make the situation worse if every new worker generates more requests against the same constrained shared file system. The bottleneck is shared, while the additional concurrency increases pressure on it.

Why six healthy months prove very little

A long period without failure validates only the combination of demand, baseline throughput and remaining credits during that period. It does not prove that the workload is sustainable after the credits are gone. A declining BurstCreditBalance is therefore not just a capacity detail; it is an early reliability signal.

Load testing must cover the steady state, not only a short peak test. A ten-minute test may demonstrate that the burst ceiling is high enough. It says little about a deficit that accumulates over thousands of hours.

A better investigation sequence

  1. Compare CPU utilization with load average and I/O wait.
  2. Inspect EFS BurstCreditBalance over weeks or months.
  3. Compare MeteredIOBytes with PermittedThroughput.
  4. Correlate storage constraints with application latency and worker saturation.
  5. Check how much data is actually in the Standard storage class, because that drives the Bursting baseline.
  6. Decide whether the workload belongs on Bursting, Elastic or Provisioned throughput.

The engineering lesson

Burst capacity is a buffer, not a capacity plan. Any credit-based service should be evaluated by the direction and rate of its balance, not merely by whether the current value is above zero. The earlier article in this series describes the real incident pattern in detail; the next article turns that lesson into a concrete CloudWatch monitoring design.

Official references

BERND KUHLEN · CLOUD ENGINEERING FIELD NOTES