Guide: Observability Stack — Prometheus, Loki, Grafana

Guide: MinIO as an S3-Compatible Backup Target

Self-hosted S3-compatible object storage for backups: MinIO behind an Nginx proxy, with an rclone-based backup agent and versioned buckets. Your offsite backups without a cloud bill.

Guide: MinIO as an S3-Compatible Backup Target

Self-hosted S3-compatible object storage for backups: MinIO behind an Nginx proxy, with an rclone-based backup agent and versioned buckets. Your offsite backups without a cloud bill.

Metrics, logs, and dashboards in one compose file: Prometheus scraping node-exporter and cAdvisor, Loki with Alloy for log aggregation, and a pre-provisioned Grafana. The stack I use to watch everything else on this site.

What this is

A deployment guide with working examples — not a downloadable product. Everything below is the actual configuration I use, abridged to the parts that matter.

Deployment example

services:
  prometheus:
    image: prom/prometheus:${PROMETHEUS_VERSION:-v2.55.0}
    container_name: prometheus
    restart: unless-stopped
    networks:
      - observability
    volumes:
      - ./config/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus_data:/prometheus
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
      - --storage.tsdb.retention.time=${PROMETHEUS_RETENTION:-30d}
      - --web.console.libraries=/etc/prometheus/console_libraries
      - --web.console.templates=/etc/prometheus/consoles
      - --web.enable-lifecycle
    logging:
      driver: json-file
      options:
        max-size: 10m
        max-file: "3"
    labels:
      - observability.service=prometheus
  grafana:
    image: grafana/grafana:${GRAFANA_VERSION:-11.4.0}
    container_name: grafana
    restart: unless-stopped
    networks:
      - observability
    volumes:
      - grafana_data:/var/lib/grafana
      - ./config/grafana/datasources:/etc/grafana/provisioning/datasources:ro
      - ./config/grafana/dashboards:/etc/grafana/provisioning/dashboards:ro
      - ./config/grafana/dashboards-definitions:/var/lib/grafana/dashboards:ro
    environment:
      - GF_SECURITY_ADMIN_USER=${GRAFANA_ADMIN_USER:-admin}
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_ADMIN_PASSWORD:-admin}
      - GF_INSTALL_PLUGINS=${GRAFANA_PLUGINS:-grafana-piechart-panel}
      - GF_SERVER_ROOT_URL=${GRAFANA_ROOT_URL:-http://localhost:3000}
      - GF_AUTH_ANONYMOUS_ENABLED=${GRAFANA_ANON_AUTH:-false}
    logging:
      driver: json-file
      options:
        max-size: 10m
        max-file: "3"
    labels:
      - observability.service=grafana

Excerpt — prometheus, grafana from the compose file. The remaining services (proxies, init jobs, exporters) follow the same pattern and mount their configuration from a config/ directory.

Essential environment variables

Copy .env.example to .env and at minimum set:

NETWORK_NAME=observability
PROMETHEUS_VERSION=v2.55.0
PROMETHEUS_RETENTION=30d
NODE_EXPORTER_VERSION=v1.8.2
CADVISOR_VERSION=latest
LOKI_VERSION=3.2.0

Production checklist

• Change GRAFANA_ADMIN_PASSWORD in .env
• Set GRAFANA_ANON_AUTH=false
• Configure Prometheus remote_write if using long-term storage
• Set Loki retention in config/loki.yml
• Add SSL reverse proxy (Nginx + Let's Encrypt)
• Restart: docker compose restart after config changes
• Backup: docker compose down + tar volumes

A workflow for reading the stack

The stack earns its keep when something breaks, and the reading order matters: an alert fires on a metric, the Grafana panel shows when the value moved and what changed alongside it, and Loki answers why - switch to the logs view, keep the same time range and the same label set (host, container, service), and the error lines sit right at the moment the metric turned. Fix, then confirm recovery on the same panel instead of declaring victory from memory. That loop - metric, panel, logs, fix, panel - is the daily driver; everything else in the stack exists to make it fast.

Alert on these first, dashboards later

Dashboards are for humans who are already looking; alerts are for everyone else. Before building any pretty panels, define the boring five: host down (node-exporter up == 0), disk filling (node_filesystem_avail_bytes above 85 percent), any scrape target down (up == 0 per service), certificate expiry inside 14 days, and a container restarting in a loop (restart count rising over 15 minutes). Each one maps to a concrete 3 a.m. action, which is the real test of an alert. Pretty dashboards can wait until the pager is quiet.

Cardinality and retention pitfalls

Two silent killers live in every metrics stack. Label cardinality: a label whose values are unbounded - user IDs, request paths, container IDs - multiplies time series until Prometheus eats RAM and queries crawl. Keep labels to bounded sets (host, service, severity) and put detail in Loki, where it belongs. Retention defaults: the 30-day Prometheus default is fine for operations, but someone will eventually ask for "last year's numbers" - decide now whether that is remote_write to object storage or a polite no. For Loki, set retention explicitly in config/loki.yml; unbounded log retention is the quietest disk-filler in this stack.

FAQ

Why Prometheus and Loki instead of one tool for both? Metrics and logs have different shapes: metrics are pre-aggregated numbers you query and alert on cheaply, logs are raw events you grep when something breaks. Prometheus stores the numbers, Loki stores the events, and Grafana correlates both by time and labels. One tool doing both ends up mediocre at each.

How much disk does this stack need? With the defaults - 30 days of Prometheus retention and json-file log caps of 10 MB times three per container - a small single-node host runs comfortably in a few gigabytes. Raise retention only together with disk planning or move long-term storage to remote_write.

Why are Grafana datasources and dashboards pre-provisioned? Everything in the provisioning directories is applied at container start, declaratively. That means a rebuilt Grafana comes back with its datasources and dashboards intact - no click-ops, and the configuration lives in git next to the compose file.

How do I add my own applications to monitoring? Export metrics on /metrics and add a scrape target to config/prometheus.yml, or run Alloy on the app host to forward structured logs into Loki. Services already exposing Prometheus metrics - node-exporter, cAdvisor, most infrastructure images - only need a scrape entry.

A guide, not a product. This page is deployment documentation with working examples — there is no zip, no download, no support contract. You adapt the patterns to your own environment, and you own the result.

Want this running production-grade in your infrastructure instead? I set this up for teams as the one-day Observability workshop — details on the consulting page.