Area 07 · Deployment · Concept 19 of 20

Alerts and Dashboards

Get alerted on errors, latency or cost spikes, and see the trends at a glance.

11–20 · Production levelPhase 5 · Weeks 18–20≈ 1 hour

The idea

Dashboards show trends; alerts wake you when something needs action. For AI services, dashboard the usual signals (request rate, error rate, p50/p95 latency) plus AI-specific ones: TTFT, tokens per request, cost per hour and per feature, fallback rate, cache hit rate, guardrail blocks, thumbs-down rate.

Alert on symptoms users feel and on money: error rate above a threshold, p95 above your SLO, cost per hour far above normal, provider 429 rate rising. Each alert needs an owner and a runbook link.

Avoid noisy alerts; people learn to ignore them.

Why it matters

Without alerts you find out about outages and runaway costs from users and invoices.

Words to know

  • Golden signalsLatency, traffic, errors and saturation.
  • RunbookSteps to diagnose and fix a specific alert.

Try it without code

Choose five alerts

  1. Write five alerts for your app with threshold, window and who gets paged.
  2. Write a one-line runbook for each.

Use it in code

Expose Prometheus metrics for requests, latency, tokens and cost.

python
# pip install prometheus-client
from prometheus_client import Counter, Histogram, start_http_server

REQS = Counter("llm_requests_total", "LLM requests", ["feature", "model", "status"])
LAT = Histogram("llm_latency_seconds", "End-to-end latency", ["feature"],
                buckets=(0.5, 1, 2, 4, 8, 16, 32))
TOKENS = Counter("llm_tokens_total", "Tokens", ["feature", "direction"])
COST = Counter("llm_cost_usd_total", "Cost in USD", ["feature"])

start_http_server(9100)
# after each call:
# REQS.labels("chat", model, "ok").inc(); LAT.labels("chat").observe(secs)
# TOKENS.labels("chat", "in").inc(u.input_tokens); COST.labels("chat").inc(usd)
PRACTICE · TICK IT AFTER THIS

Build a dashboard with error rate, p95 latency, TTFT and cost per hour, and one alert that fires on a cost spike.

Common mistakes

  • Alerts with no owner or runbook.
  • Only alerting on errors, not on cost.

Go deeper