The idea
Reflection adds a review step: after producing a draft, the model (or a second model call with a critic prompt) checks it against the task and criteria, lists problems, and revises. It catches missing requirements, wrong formats and obvious errors.
It works best with concrete checks: run the code and read the error, validate JSON against the schema, compare the answer with the sources. A critic with no new information often just rubber-stamps the draft.
Each reflection round costs a full call. Limit rounds (one or two) and stop early when checks pass.
It is a simple way to raise quality on tasks with checkable criteria, like code that must pass tests.
Picture it
Words to know
- Evaluator-optimiserA generator and a critic in a loop.
Try it without code
Critique your own draft
- Ask a chat app to write a function.
- Then ask it to list three bugs or edge cases in its own code and fix them.
- Run both versions.
Use it in code
Generate code, run tests, feed failures back, up to two rounds.
import subprocess, anthropic
client = anthropic.Anthropic()
def generate(prompt):
m = client.messages.create(model="claude-sonnet-5-5", max_tokens=1200,
messages=[{"role": "user", "content": prompt + "\nReturn only Python code."}])
return m.content[0].text.strip().removeprefix("```python").removesuffix("```")
task = "Write slugify(text) that lowercases, replaces spaces with '-', and strips punctuation."
code = generate(task)
for round_ in range(2):
open("slug.py", "w").write(code)
res = subprocess.run(["python", "-m", "pytest", "-q", "test_slug.py"], capture_output=True, text=True)
if res.returncode == 0:
print("tests pass"); break
code = generate(f"{task}\nYour code:\n{code}\nTest failures:\n{res.stdout[-2000:]}\nFix it.")Write 5 pytest cases for a small function and measure how often one reflection round turns a failing draft into a passing one.
Common mistakes
- Reflection with no external signal.
- Unlimited revise loops.