What you will have at the end
- A GitHub repo with a README, a demo GIF and a cost table
- A chat loop that streams tokens as they arrive
- A /json command that returns validated data
- Token and cost totals per message and per session
Step by step
- Set up the repoCreate a virtual environment, add a .env for the API key (and .env to .gitignore), and a prompts/system_v1.txt file.
- Basic chat loopRead input, append to a messages list, call the API, print the reply, repeat. Exit on /quit.
- Stream repliesSwitch to the streaming API and print text deltas as they arrive. Measure time to first token.
- Count tokens and costRead the usage object after each reply, compute cost from a price table in config, and print "in / out / $" under every message.
- Trim historyKeep the conversation under a token budget by dropping the oldest turns, never the system prompt.
- Add /json modeDefine a Pydantic model (e.g. a task with title, due_date, priority), force the model to fill it via a tool schema, validate, and pretty-print.
- Add prompt cachingMark the system prompt (and an optional reference document) as cacheable and show cache reads in the cost line.
- Write the READMEExplain the design choices, show a sample session, and include a table of cost per 10 messages with and without caching.
DONE WHEN
- Replies stream and never crash on long sessions
- /json always returns valid data or a clear error after one retry
- Every message shows tokens and cost
- The README has numbers, not just features
Stretch goals
- Add a /model command to switch models live and compare cost
- Accept an image path and send it with the next message
- Save sessions to SQLite