AI Careers

The LLM Practical Round: Concurrency, Retries, Token Budgets and a Quick Eval

Some AI-lab loops now include a round where you work with a model through an API in a notebook. It is not a prompt-writing contest. It tests whether you can call a model a thousand times safely, cheaply and measurably. The patterns that pass, with code.

What the round is actually testing

Candidates report rounds - Anthropic's onsite is the best-known example - where you are given API access to a model and a notebook, and asked to build something: classify a dataset, extract structured fields from documents, chain several calls into a small pipeline. The task is usually simple. The evaluation is about engineering.

Interviewers are watching for the things that separate a demo from a system. Do you run calls concurrently, or sit through a thousand sequential round trips? Do you handle rate limits and transient errors, or does the first 429 kill the run? Do you know roughly what the job will cost before you start it? Do you check the outputs against something, or just eyeball three of them? And do you understand the ways models fail - confident wrong answers, format drift, sensitivity to prompt wording - and design around them?

The code below is provider-neutral. Wrap whichever SDK the round gives you in a single async function and the rest does not change.

Concurrency with a ceiling

The first win is concurrency. Model calls are network-bound, so a thousand of them run sequentially at two seconds each take over half an hour; with twenty in flight they take under two minutes. The second is putting a ceiling on it, because unbounded concurrency simply converts your speed-up into rate-limit errors. A semaphore is the idiomatic way to cap in-flight requests in asyncio.

import asyncio, random

async def call_model(prompt: str) -> str:
    """Wrap the provider SDK here. Raise RateLimited / Transient on those errors."""
    ...

class RateLimited(Exception): ...
class Transient(Exception): ...

async def with_retries(prompt, attempts=5, base=1.0, cap=30.0):
    for i in range(attempts):
        try:
            return await call_model(prompt)
        except (RateLimited, Transient):
            if i == attempts - 1:
                raise
            # exponential backoff with full jitter - avoids synchronized retries
            await asyncio.sleep(random.uniform(0, min(cap, base * 2 ** i)))

async def run_all(prompts, max_in_flight=20):
    sem = asyncio.Semaphore(max_in_flight)
    async def one(p):
        async with sem:
            try:
                return await with_retries(p)
            except Exception as e:          # one bad item must not sink the batch
                return e
    return await asyncio.gather(*(one(p) for p in prompts))

Retries that do not make things worse

Retry only what is retryable. Rate limits and server-side errors are transient; a request rejected for being malformed or too long will fail identically every time, and retrying it just burns quota. If the API returns a retry-after header, honour it instead of your own backoff.

Add jitter. If twenty workers hit a rate limit at the same moment and all wait exactly two seconds, they all retry at the same moment and hit it again. Randomising the wait spreads them out. Mentioning this unprompted is a small signal that you have operated something at scale.

Return errors rather than raising them out of the batch. A run over a thousand inputs that dies at item 612 has thrown away 611 paid-for results. Collect failures, report them, and rerun only those.

Know the cost before you press run

Interviewers like candidates who estimate before executing. Tokenise a sample of inputs, multiply by the dataset size, add the expected output length, and price both sides - output tokens typically cost several times more than input tokens. The numbers will rarely be exact, but the habit is what matters, and it often changes the design: a long shared instruction block is a candidate for prompt caching, a short classification can use a smaller model, and capping max_tokens on a yes-or-no question prevents a model from writing an essay you pay for.

def estimate_cost(n_items, avg_in_tokens, avg_out_tokens,
                  usd_per_m_in, usd_per_m_out):
    cost_in  = n_items * avg_in_tokens  / 1e6 * usd_per_m_in
    cost_out = n_items * avg_out_tokens / 1e6 * usd_per_m_out
    return round(cost_in + cost_out, 2)

# Prices are placeholders - read them from the provider's page on the day.
print(estimate_cost(10_000, avg_in_tokens=900, avg_out_tokens=60,
                    usd_per_m_in=3.0, usd_per_m_out=15.0))   # 36.0

Structured output and a twenty-item eval

When the task needs structured output, ask for it explicitly - with the provider's structured-output or tool-calling feature if available, otherwise with a JSON schema in the prompt - and validate every response. Treat a parse failure as a retryable error with one corrective retry that includes the validation message. Count how often it happens; if it is frequent, the prompt or the schema is wrong.

Then measure. Hand-label twenty examples before you write the prompt, run the pipeline against them, and report accuracy and the failure cases. Twenty is enough to catch a broken prompt and small enough to label in the time available. When you change the prompt, rerun the twenty and say whether the number moved. That loop - label, run, measure, change one thing - is the most important thing this round can show, because it is how every reliable LLM system gets built.

Finish by saying where the model is likely to fail on this data and what you would do next with more time: more labelled examples, a check for the hardest category, or a cheaper model for the easy majority with the expensive one reserved for low-confidence cases.

Keep reading

Related articles