Spot trouble before your customers do

← All projects
CASE 03

OBSERVABILITY · OPERATIONS · AUTOMATION

24/7on-call responder team

When I joined in 2000, a handful of servers and log-analysis scripts were enough to describe the starting point—not a monitoring system that could keep up with growing complexity.

Challenge

Replace reactive log analysis with monitoring that could grow with the infrastructure and flag trouble before users reported it.

Approach

Start with Nagios and Zabbix, introduce service checks and notifications, and retain performance metrics for trend analysis.

Outcome

An evolving monitoring practice, later complemented by Grafana and Checkmk, combining early warning with longer-term operational visibility.

Read the case study

A small setup with a growing blind spot. At the beginning there were only a handful of servers and scripts analyzing log files. As the platform grew, that approach could not keep up with either scale or complexity. More importantly, it was reactive: it told us what had already happened.

The boss and the customer are not monitoring tools. Why wait until a problem has happened to notice it? If a manager or a customer is the first to report an outage, operations is already behind. Beyond the incident itself, that puts confidence in the team at risk—inside and outside the organization.

The classic example: an expiring TLS certificate. In 2000, the certificate-renewal automation familiar today was not available in the same form. Let’s Encrypt arrived much later, in 2015. We needed to monitor approaching expiry rather than discover it through a broken connection. Certificate requirements and issuers can vary; even an automated renewal process needs to be monitored.

Detect, notify and keep the evidence. Our first choices were Nagios and Zabbix. We needed checks that detected problems and notifications that reached someone able to act. Notifications went especially to the 24/7 on-call responder team, so incidents could be acted on around the clock. At that time, cost constraints meant the website did not yet have a high-availability architecture. Reliable detection and rapid response were therefore central to maintaining the best possible service quality; monitoring did not itself remove the risk of downtime. Over the years, delivery channels included SMS, WhatsApp and PagerDuty; these were later developments, not a claim that all of them were used in 2000.

From incidents to trends. Recording performance metrics made it possible to look beyond a single alarm: how were load, latency and available capacity developing over time? Grafana and Checkmk later joined the toolset. The goal remained the same: identify deterioration earlier and use the data to understand where the system was heading.

The principle that stayed. Monitoring is not just a collection of dashboards. It should give the team a chance to act before someone else has to tell them there is a problem.

Historical reference: Let’s Encrypt’s first public certificate announcement (2015).