Prompt caching: 90% off your stable prefix¶

Both Anthropic and OpenAI offer prompt caching: if the leading slice of your request is identical between calls, the provider serves it from cache at ~10% of the normal input cost.

convopack knows which messages are pinned and stable, so it's the natural place to decide where the cache breakpoints go.

• Anthropic: explicit cache_control markers, max 4 per request. convopack emits them automatically.
• OpenAI: automatic prefix caching for 1024+ token identical prefixes. convopack exposes a signature you can use to detect drift.

In [ ]:

from convopack import Message, Packer, TextBlock

from convopack import Message, Packer, TextBlock

1. Enable caching with one flag¶

Pass cache=True and convopack populates PackResult.cache_markers with the indices of kept messages that should receive a cache breakpoint.

In [ ]:

history = [
    Message(role="system", content=[TextBlock(text="You are an expert tutor on Nordic history.")]),
    Message(role="user", content=[TextBlock(text="When was Norway founded?")]),
    Message(role="assistant", content=[TextBlock(text="The Kingdom of Norway was formed in 872.")]),
    Message(role="user", content=[TextBlock(text="And the constitution?")]),
]

packer = Packer(
    budget=10_000,
    tokenizer="approx",
    pin=("system", "first_user", "last_user"),
    cache=True,
)
result = packer.pack(history)
print("cache_markers:", result.cache_markers)

history = [ Message(role="system", content=[TextBlock(text="You are an expert tutor on Nordic history.")]), Message(role="user", content=[TextBlock(text="When was Norway founded?")]), Message(role="assistant", content=[TextBlock(text="The Kingdom of Norway was formed in 872.")]), Message(role="user", content=[TextBlock(text="And the constitution?")]), ] packer = Packer( budget=10_000, tokenizer="approx", pin=("system", "first_user", "last_user"), cache=True, ) result = packer.pack(history) print("cache_markers:", result.cache_markers)

Two markers: index 0 (system) and index 1 (first_user). The last_user index is intentionally not marked because it changes every call, which would invalidate the cache.

2. Anthropic emits cache_control automatically¶

pack_anthropic() threads cache_markers through, and the resulting payload has cache_control: {"type": "ephemeral"} on the right blocks:

In [ ]:

import json

raw = [
    {"role": "user", "content": "When was Norway founded?"},
    {"role": "assistant", "content": "The Kingdom of Norway was formed in 872."},
    {"role": "user", "content": "And the constitution?"},
]
payload = packer.pack_anthropic(raw, system="You are an expert tutor on Nordic history.")
print("--- system ---")
print(json.dumps(payload.system, indent=2))
print("--- messages ---")
print(json.dumps(payload.messages, indent=2))

import json raw = [ {"role": "user", "content": "When was Norway founded?"}, {"role": "assistant", "content": "The Kingdom of Norway was formed in 872."}, {"role": "user", "content": "And the constitution?"}, ] payload = packer.pack_anthropic(raw, system="You are an expert tutor on Nordic history.") print("--- system ---") print(json.dumps(payload.system, indent=2)) print("--- messages ---") print(json.dumps(payload.messages, indent=2))

The system field is now a list (the form Anthropic requires when any marker is applied), the cache_control on system caches the entire system prompt, and the first user turn also carries a marker for an even longer cached prefix.

3. Detect OpenAI prefix drift¶

OpenAI's automatic prefix caching is great when your prefix stays byte-identical between calls. cache_prefix_signature returns a sha256 you can persist to detect drift:

In [ ]:

sig_v1 = packer.cache_prefix_signature(history)

# Simulate appending a new turn -- prefix unchanged.
history_v2 = history + [Message(role="assistant", content=[TextBlock(text="It was signed in 1814.")])]
sig_v2 = packer.cache_prefix_signature(history_v2)

# Now simulate changing the system prompt -- prefix changes.
history_v3 = list(history_v2)
history_v3[0] = Message(role="system", content=[TextBlock(text="You are now a chef.")])
sig_v3 = packer.cache_prefix_signature(history_v3)

print(f"v1 prefix sig: {sig_v1[:12]}...")
print(f"v2 prefix sig: {sig_v2[:12]}...  (same? {sig_v1 == sig_v2})")
print(f"v3 prefix sig: {sig_v3[:12]}...  (same as v1? {sig_v1 == sig_v3})")

sig_v1 = packer.cache_prefix_signature(history) # Simulate appending a new turn -- prefix unchanged. history_v2 = history + [Message(role="assistant", content=[TextBlock(text="It was signed in 1814.")])] sig_v2 = packer.cache_prefix_signature(history_v2) # Now simulate changing the system prompt -- prefix changes. history_v3 = list(history_v2) history_v3[0] = Message(role="system", content=[TextBlock(text="You are now a chef.")]) sig_v3 = packer.cache_prefix_signature(history_v3) print(f"v1 prefix sig: {sig_v1[:12]}...") print(f"v2 prefix sig: {sig_v2[:12]}... (same? {sig_v1 == sig_v2})") print(f"v3 prefix sig: {sig_v3[:12]}... (same as v1? {sig_v1 == sig_v3})")

Appending a new assistant turn doesn't change the prefix signature, so OpenAI's cache hits. Changing the system prompt does change the signature, so the cache is cold for the next call.

4. cache_info: how much would be cached?¶

cache_info(history) reports the markers, marked token count, and a rough hit-ratio estimate:

In [ ]:

info = packer.cache_info(history)
for k, v in info.items():
    print(f"{k:>20}: {v}")

info = packer.cache_info(history) for k, v in info.items(): print(f"{k:>20}: {v}")

Use this to tune what gets pinned. If hit_ratio is tiny, you're cache-marking a trivial slice — consider pinning a heavy few-shot example by index too:

Packer(pin=("system", 1, 2, "first_user"), cache=True)
# Indices 1 and 2 also get markers (up to the 4-marker cap).

Cost impact (back-of-envelope)¶

A typical agent with a 2K-token system prompt + first-user question, called 50 times per session:

• Without caching: 50 × 2K = 100K input tokens at full price.
• With caching: 1 × 2K full + 49 × 2K × 0.10 = 11.8K equivalent input tokens.

88% reduction on the prefix portion, just for setting cache=True.