The idea
Users tell you when the system fails: thumbs down, rephrasing the same question, copying an answer (good sign), escalating to a human (bad sign), editing a draft before sending. Capture explicit feedback (buttons, comments) and implicit signals, linked to the trace ID.
Then close the loop: review negative feedback weekly, do error analysis on it, add representative failures to the golden set with corrected expected answers, fix, and confirm the fix with the eval suite.
Feedback is biased (unhappy users click more), so use it to find failures, not as your only quality metric.
It is how your eval set keeps up with what real users actually do.
Picture it
Words to know
- Explicit feedbackRatings or comments users give on purpose.
- Implicit feedbackBehaviour that signals quality, like retries or copies.
Try it without code
Design the buttons
- Sketch your app’s answer card with thumbs up/down and an optional comment box.
- List the implicit signals you could log.
Use it in code
A FastAPI endpoint that records feedback against a trace and queues it for review.
from fastapi import FastAPI
from pydantic import BaseModel
app = FastAPI()
REVIEW_QUEUE = []
class Feedback(BaseModel):
trace_id: str
rating: int # 1 = up, 0 = down
comment: str | None = None
@app.post("/feedback")
def feedback(fb: Feedback):
# also: langfuse.create_score(trace_id=fb.trace_id, name="user_rating", value=fb.rating)
if fb.rating == 0:
REVIEW_QUEUE.append(fb.model_dump())
return {"ok": True}Add thumbs feedback to your RAG app, collect 20 ratings from friends, and turn every thumbs-down into a golden case.
Common mistakes
- Feedback not linked to traces, so you cannot see what happened.
- Collecting feedback nobody reviews.