Observability Platform

Without central metrics or logs, diagnosing an incident meant SSH-ing into individual machines and reading log files by hand — usually after users had already noticed.

1. L1Collection
   • Node Exporter
   • cAdvisor
   • Grafana Alloy
2. L2Storage
   • Prometheus (metrics)
   • Loki (logs)
3. L3Visualization
   • Grafana dashboards
4. L4Alerting
   • Prometheus alert rules
   • Grafana notifications

Metrics and logs in one place

Grafana queries both Prometheus and Loki, so a spike on a dashboard can be followed straight into the related logs.

Grafana Alloy as the log and telemetry agent

A single agent ships container and system logs to Loki with consistent labels, simplifying the collection layer.

Alert on symptoms, not noise

Alerts focus on user-impacting conditions (targets down, sustained errors, disk exhaustion) rather than every metric fluctuation.

• Deployed the stack with Docker Compose and persistent volumes for metrics and log storage.
• Configured Prometheus scrape targets for hosts and containers.
• Set up Alloy pipelines to label and ship logs to Loki.
• Built Grafana dashboards for host resources, container health, and application logs.
• Defined alert rules and notification routing.

• Monitoring endpoints kept on an internal network, not exposed publicly.
• Grafana protected by authentication with the default admin credentials changed.
• Retention configured so logs are not kept longer than needed.

• One place to see host, container, and application health.
• Faster root-cause analysis by correlating metrics with logs.
• Problems surface through alerts instead of user reports.

• Label cardinality matters — unbounded labels quietly degrade Prometheus and Loki.
• Dashboards are most useful when designed around specific questions an on-call engineer asks.