The idea
Prompts change behaviour as much as code does, so they deserve the same discipline. Store prompts as files in Git (or in a prompt management tool), give each one an ID and version, review changes in pull requests, and run evals on every change.
At runtime, log which prompt version produced each response. When quality drops you can see exactly which change caused it and roll back to the previous version.
Prompt management tools (Langfuse, PromptLayer and others) let non-engineers edit prompts with versioning and labels like production and staging; make sure changes still go through evals.
Unversioned prompts edited in a dashboard are the most common untraceable production change in AI apps.
Words to know
- Prompt registryA store of prompts with versions and labels.
Try it without code
Find your prompts
- Search your code for every prompt string.
- Count how many are inline vs in files.
- Pick one to move into a versioned file.
Use it in code
File-based prompts with version and hash logged per request.
import hashlib, pathlib, yaml # pip install pyyaml
def load_prompt(name, version):
data = yaml.safe_load(pathlib.Path(f"prompts/{name}/{version}.yaml").read_text())
text = data["system"]
return {"name": name, "version": version, "system": text,
"sha": hashlib.sha256(text.encode()).hexdigest()[:10]}
p = load_prompt("support", "v3")
# call the model with p["system"], and log p["name"], p["version"], p["sha"] with the responseMove all prompts in your app into prompts/<name>/<version>.yaml and log the version with every trace.
Common mistakes
- Editing the production prompt without bumping the version.
- Prompt changes that skip evals because "it’s just text".