DevOps Monitoring & Alerting — Real-World Lab (Prometheus + Grafana)

DevOps uses email notifications to actively alert on-call staff and for unattended system monitoring, ensuring prompt incident response and satisfying service level agreements (SLAs). For email notifications to work, the system requires exporters, Prometheus scraping, Grafana alert rules, and a functioning notification delivery pipeline, with failures often occurring in the delivery layer. Configuring SMTP on the Grafana server involves setting up the email host, user credentials (requiring an app password from Gmail), and the sender information. Notifications are then configured in the Grafana UI by establishing contact points and notification policies that route alerts to specific recipients. Alerts are triggered by creating rules, like a high CPU threshold, and simulating the problem to confirm the email is delivered. DevOps reads alert emails to quickly understand the problem, identify the affected system using labels, assess the severity, and determine the necessary initial actions. Best practices involve monitoring for monitoring failures, reducing noisy alerts with time windows and context, including labels, and regular testing of the notification system. Prometheus requires node exporters to gather system metrics, providing data on CPU, memory, disk, and network usage, which are crucial for effective monitoring. A lab demonstrated the necessity of node exporter by illustrating how Prometheus alone cannot provide this key system data. The final lab involved setting up Prometheus on an EC2 instance to monitor a KIND Kubernetes cluster, using a DaemonSet for the node exporter.