OpenAI 上下文缓存

重复的长前缀不再重复计费
缓存命中部分的输入 token 按折扣价结算
对长系统提示、固定文档、多轮同一背景的对话收益最大

原理

大模型对输入的处理是「逐 token 读一遍」。如果连续多次请求的前缀完全一致，网关可以把这段前缀的处理结果复用，省下重复计算。

命中条件有三条：

前缀内容完全相同，一个字符都不能变
前缀长度达到该模型的最小缓存门槛
两次请求间隔在缓存有效期之内

缓存以「前缀」为单位，不是任意位置。如果只有中间一段相同、开头不同，是不会命中的。所以要把固定内容放在最前面，变化的内容放在最后面。

用法

不需要额外参数。只要请求结构满足前缀一致，网关会自动命中并在响应里报告。

usage.prompt_tokens_details 的元素字段

integer

本次请求命中缓存的输入 token 数。没有命中时为 0 或省略。

把命中率做高的写法

玩具示例（几乎不命中）：

{
  "messages": [
    { "role": "user", "content": "文档内容……\n\n问题是 A" }
  ]
}

正确写法（固定内容在前，变量在后）：

{
  "messages": [
    { "role": "system", "content": "【这一段是固定的产品手册，约 8000 token】" },
    { "role": "user", "content": "问题是 A" }
  ]
}

第二次请求只改最后一句，前面那一大段前缀不变，就能命中。

A toy example (almost never hits):

{
  "messages": [
    { "role": "user", "content": "Document content...\n\nQuestion is A" }
  ]
}

The right way (fixed content first, variable content last):

{
  "messages": [
    { "role": "system", "content": "[This segment is a fixed product manual, about 8000 tokens]" },
    { "role": "user", "content": "Question is A" }
  ]
}

The second request changes only the last sentence; the long prefix ahead of it stays the same, so it hits.

成本对比

项目	未命中	命中
输入计费	全价	缓存部分按折扣价
输出计费	全价	全价（输出不缓存）
延迟	基准	通常更低

具体折扣比例取决于上游模型，不同模型不一样。

Item	Miss	Hit
Input billing	Full price	Cached portion at a discount
Output billing	Full price	Full price (output isn't cached)
Latency	Baseline	Usually lower

The exact discount depends on the upstream model; it differs by model.

有效期

缓存不是永久的。长时间不请求会自动失效，届时同样的前缀会重新按全价计算一次。做定时批处理时，把同类请求集中在一个时间窗内跑完，命中率最高。

使用示例

让长前缀命中缓存

把稳定不变的长文本放在最前面，变化的部分放后面。以下例子里，产品手册全文固定，用户问题每次不同，缓存的前缀就能复用。

核对是否命中

响应里的 usage.prompt_tokens_details.cached_tokens 是本次命中缓存的 token 数。大于 0 表示命中，为 0 说明前缀变了或缓存已过期。

curl --request POST https://api.oairoute.com/v1/chat/completions \
  -H "Authorization: Bearer YOUR_OAIRoute_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [
      { "role": "system", "content": "以下是产品手册全文（约 8000 字）：……" },
      { "role": "user", "content": "退款流程需要几步" }
    ]
  }' | jq '.usage.prompt_tokens_details'

批量任务集中跑

同一批请求放在连续的时间窗内执行，缓存不容易过期。分散到不同时段，每批都要重新写入一次。

for q in "退款流程需要几步" "如何修改收货地址" "发票在哪里申请"; do
  curl --request POST https://api.oairoute.com/v1/chat/completions \
    -H "Authorization: Bearer $YOUR_OAIRoute_KEY" \
    -H "Content-Type: application/json" \
    -d "{\"model\":\"gpt-4o-mini\",\"messages\":[{\"role\":\"system\",\"content\":\"以下是产品手册全文：……\"},{\"role\":\"user\",\"content\":\"$q\"}]}"
done

Hit the cache on a long prefix

Put the stable, unchanging text first and the variable part after. In the example below the product manual is fixed in full and only the user question changes, so the prefix can be reused.

Confirm whether you hit

usage.prompt_tokens_details.cached_tokens in the response is the number of tokens that hit the cache. Above zero means a hit; zero means the prefix changed or the cache had expired.

curl --request POST https://api.oairoute.com/v1/chat/completions \
  -H "Authorization: Bearer YOUR_OAIRoute_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [
      { "role": "system", "content": "Below is the full product manual (about 8000 words): ..." },
      { "role": "user", "content": "How many steps does a refund take?" }
    ]
  }' | jq '.usage.prompt_tokens_details'

Run batch jobs together

Execute one batch of requests within a continuous time window so the cache doesn't expire. Spread across different periods and each batch has to write the cache again.

for q in "How many steps does a refund take?" "How do I change my shipping address?" "Where do I request an invoice?"; do
  curl --request POST https://api.oairoute.com/v1/chat/completions \
    -H "Authorization: Bearer $YOUR_OAIRoute_KEY" \
    -H "Content-Type: application/json" \
    -d "{\"model\":\"gpt-4o-mini\",\"messages\":[{\"role\":\"system\",\"content\":\"Below is the full product manual: ...\"},{\"role\":\"user\",\"content\":\"$q\"}]}"
done