Rishabh Singh

Every major model API caches by prefix: it reuses the start of a prompt only if that start matches a recent one exactly, token for token. Put a timestamp, the visitor's name or a shuffled tool list near the top, and each request begins differently, so nothing is ever read back. Prompts under the model's minimum length (1,024 tokens on OpenAI; 512 on current Claude models, up to 4,096 on some older ones) are never cached at all.

Neither case returns an error. A short prompt pays full price, and a shifting prefix pays more, since every request writes a fresh cache entry at 1.25× and reads none. The only trace is a cache-read count of zero in the usage, which is why the table above marks such requests cold or uncached. My prompt keeps the date after the cached part and builds the tool list in a fixed order. A zero there is one of the first things I look for on an agent's bill.

Public benchmarks test on someone else's data, and models get tuned toward them. Teams that ship pick models by running their own tasks. The Market fit tab does the same: it scores me against 110 job postings and shows the gaps.

Simon Willison calls this the lethal trifecta. An agent that can read private data, read content an attacker controls and send data somewhere can be talked into leaking it, whatever the prompt says. The fix has to be structural: take one of the three away. With no tools, the model's words are still a way out, so the design takes away the private data instead. The model never sees it, and code checks every draft.

Ask someone to approve two hundred actions a day and they approve nearly all of them. Human review works when it's rare enough to be read. Let people pre-approve routine work for a limited time, and keep the few actions that are hard to undo out of reach of any pre-approval.

pass@k asks whether any of k tries succeeds. For an agent acting for a user, the question is whether all of them do, which τ-bench calls pass^k. With independent tries the gap grows fast:Success per try 70%Tries 3pass@k pass^k

Model graders can favour one position (often the first), the longer answer and their own model family; that last effect is often small and varies by model and task. Score pairs in both orders, keep length in check, and drop the judge where code can decide: Garm scores its agent with deterministic checks against the database. Where a judge is needed, use more than one model family. This site's judge is from its agent's own family, so the eval set and my blind labels of its verdicts are published for anyone to check.

Structured outputs guarantee the shape of the JSON. They say nothing about whether the values are right, and a field can be well formed and invented. Check the values against the record they came from.

Embeddings capture meaning and blur exact strings: product names, error codes, SKUs. Search for “SYSPRO” and BM25 finds it, where an embedding may return near misses about ERPs. Production retrieval is often hybrid for that reason. This resume uses BM25 because job postings are full of exact terms.

When a call times out, you can't tell whether it ran. Retry without an idempotency key and the renter gets two messages, or a payment goes through twice. The fix is a transactional outbox: write every outbound action in the same transaction as the change that caused it, under an idempotency key the receiver checks, so a replay does nothing.