DistilCodeAll techniques

Technique 10 / 14

Extend cachePro

Pay cached prices after a pause instead of rewriting the whole prompt prefix at full price.

Interactive model

See the context change.

You step away for 12 minutes on a session with a 120k-token prefix.

Extend cachecached read at 0.1×

Next message after 12 min

  1. 01

    0 min: prefix cached

  2. 02

    4 and 8 min: refreshes at 0.1×

  3. 03

    12 min: refresh, then your message

  4. 04

    Prefix read from cache

  5. 05

    $0.144 for refreshes + next turn

Conceptual demonstration. Counts illustrate the flow, not measured production savings.

How it works

Less context. Same thread of work.

Provider prompt caches typically expire after five idle minutes. Step away to review a diff, or wait on a long command, and the next message rewrites the whole prefix at the cache-write price instead of reading it at roughly a tenth of the input price.

The cache heater sends a small refresh just inside that window while you are typing a reply, answering a question or permission prompt, or waiting on a running tool. Refreshes are capped per session and skipped when nothing would benefit, so the upkeep costs a fraction of the rewrite it avoids.

In practice

The operating loop

  1. 01

    Watch the window

    The heater tracks when the cache was last written and refreshes just inside the five-minute TTL.

  2. 02

    Refresh only when useful

    Refreshes run while you type, answer a prompt, or wait on a tool, never on an abandoned session.

  3. 03

    Cap the upkeep

    Each session has a refresh limit, so a long absence still ends in one rewrite.