Area 01 · LLM · Build this

CLI chatbot with streaming, JSON mode and a cost meter

A terminal chat app that shows you everything from this area working together: a versioned system prompt, streamed replies, a structured JSON mode, and a live count of tokens and money for every message.

Phase 1 · Weeks 3–6

What you will have at the end

  • A GitHub repo with a README, a demo GIF and a cost table
  • A chat loop that streams tokens as they arrive
  • A /json command that returns validated data
  • Token and cost totals per message and per session

Step by step

  1. Set up the repoCreate a virtual environment, add a .env for the API key (and .env to .gitignore), and a prompts/system_v1.txt file.
  2. Basic chat loopRead input, append to a messages list, call the API, print the reply, repeat. Exit on /quit.
  3. Stream repliesSwitch to the streaming API and print text deltas as they arrive. Measure time to first token.
  4. Count tokens and costRead the usage object after each reply, compute cost from a price table in config, and print "in / out / $" under every message.
  5. Trim historyKeep the conversation under a token budget by dropping the oldest turns, never the system prompt.
  6. Add /json modeDefine a Pydantic model (e.g. a task with title, due_date, priority), force the model to fill it via a tool schema, validate, and pretty-print.
  7. Add prompt cachingMark the system prompt (and an optional reference document) as cacheable and show cache reads in the cost line.
  8. Write the READMEExplain the design choices, show a sample session, and include a table of cost per 10 messages with and without caching.
DONE WHEN
  • Replies stream and never crash on long sessions
  • /json always returns valid data or a clear error after one retry
  • Every message shows tokens and cost
  • The README has numbers, not just features

Stretch goals

  • Add a /model command to switch models live and compare cost
  • Accept an image path and send it with the next message
  • Save sessions to SQLite