The idea
Dashboards show trends; alerts wake you when something needs action. For AI services, dashboard the usual signals (request rate, error rate, p50/p95 latency) plus AI-specific ones: TTFT, tokens per request, cost per hour and per feature, fallback rate, cache hit rate, guardrail blocks, thumbs-down rate.
Alert on symptoms users feel and on money: error rate above a threshold, p95 above your SLO, cost per hour far above normal, provider 429 rate rising. Each alert needs an owner and a runbook link.
Avoid noisy alerts; people learn to ignore them.
Without alerts you find out about outages and runaway costs from users and invoices.
Words to know
- Golden signalsLatency, traffic, errors and saturation.
- RunbookSteps to diagnose and fix a specific alert.
Try it without code
Choose five alerts
- Write five alerts for your app with threshold, window and who gets paged.
- Write a one-line runbook for each.
Use it in code
Expose Prometheus metrics for requests, latency, tokens and cost.
# pip install prometheus-client
from prometheus_client import Counter, Histogram, start_http_server
REQS = Counter("llm_requests_total", "LLM requests", ["feature", "model", "status"])
LAT = Histogram("llm_latency_seconds", "End-to-end latency", ["feature"],
buckets=(0.5, 1, 2, 4, 8, 16, 32))
TOKENS = Counter("llm_tokens_total", "Tokens", ["feature", "direction"])
COST = Counter("llm_cost_usd_total", "Cost in USD", ["feature"])
start_http_server(9100)
# after each call:
# REQS.labels("chat", model, "ok").inc(); LAT.labels("chat").observe(secs)
# TOKENS.labels("chat", "in").inc(u.input_tokens); COST.labels("chat").inc(usd)Build a dashboard with error rate, p95 latency, TTFT and cost per hour, and one alert that fires on a cost spike.
Common mistakes
- Alerts with no owner or runbook.
- Only alerting on errors, not on cost.