Rails LLM Semantic Cache: Cut AI API Costs with pgvector Similarity Matching
Rails LLM semantic cache with pgvector: cache AI responses by query similarity, cut OpenAI/Claude costs 40-60%, with benchmarks and production patterns.
The invoice came in at €847 for a single week. The app had a free tier, usage had spiked, and roughly 60% of the LLM calls were users asking the same dozen questions about the product — slightly reworded each time.
That is the classic semantic caching problem. An exact string cache catches “What is your refund policy?” asked twice by the same user. It misses “How do refunds work?”, “Can I get my money back?”, and “What’s your return policy?” — which are functionally identical queries that should return the same answer.
After that invoice, I spent a day building a semantic cache layer. The next month’s bill dropped by 58%. The implementation lives in roughly 120 lines of Ruby, uses pgvector (which the app already had for RAG), and has been in production without incident for eight months.
Here is how it works.
How Rails LLM Semantic Cache Works
A semantic cache stores LLM responses indexed by an embedding of the input query. When a new query arrives, you:
- Generate an embedding for the new query — one small API call, typically 100× cheaper than a full LLM completion
- Query the cache for stored embeddings within a cosine distance threshold
- If a close enough match exists, return the cached response
- If no match exists, call the LLM, store the response and its query embedding
The key insight is that embeddings encode semantic meaning: “How do refunds work?” and “What is your return policy?” will have high cosine similarity even though they share no words. The distance threshold is what you tune to control the aggressiveness of the cache.
This is fundamentally different from Anthropic’s prompt caching — which caches the KV state of a system prompt on Anthropic’s servers to avoid reprocessing it on every call. Semantic caching happens on your side and skips the LLM call entirely for sufficiently similar queries.
The pgvector Schema
You almost certainly already have pgvector installed if you are running any embedding-based features. If not, enable the extension and add the gem:
# Gemfile
gem "pgvector"
gem "neighbor"
# db/migrate/20261117000000_create_llm_cache_entries.rb
class CreateLlmCacheEntries < ActiveRecord::Migration[8.0]
def up
enable_extension "vector" unless extension_enabled?("vector")
create_table :llm_cache_entries do |t|
t.text :query, null: false
t.text :response, null: false
t.vector :query_embedding, limit: 1536, null: false
t.string :model, null: false
t.string :namespace, null: false, default: "default"
t.integer :hit_count, null: false, default: 0
t.integer :input_tokens, null: false
t.integer :output_tokens, null: false
t.datetime :expires_at
t.datetime :last_used_at, null: false, default: -> { "CURRENT_TIMESTAMP" }
t.timestamps
end
add_index :llm_cache_entries, :namespace
add_index :llm_cache_entries, :last_used_at
add_index :llm_cache_entries, :expires_at
execute <<~SQL
CREATE INDEX llm_cache_entries_embedding_idx
ON llm_cache_entries
USING ivfflat (query_embedding vector_cosine_ops)
WITH (lists = 50);
SQL
end
def down
drop_table :llm_cache_entries
end
end
A few design choices worth explaining:
namespace: Different features in your app have different cache semantics. Customer support answers should not be served to users asking coding questions. Namespace by feature or prompt template version: support_v3, pricing_faq_v1.
hit_count + last_used_at: Gives you LRU eviction data. When the cache grows too large, delete entries with low hit counts not accessed in 90 days. Keep the high-value entries that keep paying off.
input_tokens + output_tokens: Store the token counts from the original LLM response. This is what powers your cost tracking dashboard — cache hits have zero token cost, so attribute the cost to the original call, not the downstream hits.
expires_at: Nullable. Null means no TTL; set it at write time based on content volatility.
IVFFlat index with lists: 50: For caches under 100k entries, IVFFlat with 50 lists gives fast approximate nearest-neighbor queries. If you scale past 500k entries, migrate to HNSW.
The SemanticCache Service
# app/services/llm/semantic_cache.rb
module Llm
class SemanticCache
# Distance below which queries are considered semantically equivalent.
# 0.0 is identical; 1.0 is orthogonal.
# 0.12–0.15 is a practical sweet spot for most Q&A use cases.
DEFAULT_THRESHOLD = 0.12
Result = Data.define(:response, :cache_hit, :entry)
def initialize(namespace:, threshold: DEFAULT_THRESHOLD)
@namespace = namespace.to_s
@threshold = threshold
end
def fetch(query, model:)
embedding = embed(query)
cached = find_cached(embedding)
if cached
cached.increment!(:hit_count)
cached.touch(:last_used_at)
return Result.new(response: cached.response, cache_hit: true, entry: cached)
end
response_data = yield
store(query:, embedding:, model:, response_data:)
Result.new(response: response_data[:response], cache_hit: false, entry: nil)
end
private
def embed(query)
client = OpenAI::Client.new
response = client.embeddings(
parameters: {
model: "text-embedding-3-small",
input: query
}
)
response.dig("data", 0, "embedding")
end
def find_cached(embedding)
entry = LlmCacheEntry
.where(namespace: @namespace)
.where("expires_at IS NULL OR expires_at > ?", Time.current)
.nearest_neighbors(:query_embedding, embedding, distance: "cosine")
.first
entry if entry && entry.neighbor_distance <= @threshold
end
def store(query:, embedding:, model:, response_data:)
LlmCacheEntry.create!(
query: query,
response: response_data[:response],
query_embedding: embedding,
model: model,
namespace: @namespace,
input_tokens: response_data[:input_tokens],
output_tokens: response_data[:output_tokens],
last_used_at: Time.current
)
end
end
end
The nearest_neighbors method comes from the neighbor gem, which integrates pgvector with ActiveRecord:
# app/models/llm_cache_entry.rb
class LlmCacheEntry < ApplicationRecord
has_neighbors :query_embedding
end
Wrapping Your LLM Client
The cache layer wraps whatever LLM client you use. Here is the pattern for Claude via the Anthropic SDK:
# app/services/llm/client.rb
module Llm
class Client
def initialize(namespace:, cache_threshold: SemanticCache::DEFAULT_THRESHOLD)
@cache = SemanticCache.new(namespace:, threshold: cache_threshold)
end
def complete(messages:, system: nil, model: "claude-opus-4-5", max_tokens: 1024)
# Use the last user message as the cache key — the question being asked
query = messages.select { |m| m[:role] == "user" }.last.fetch(:content)
result = @cache.fetch(query, model:) do
response = anthropic_client.messages(
parameters: {
model:,
system:,
messages:,
max_tokens:
}
)
{
response: response.dig("content", 0, "text"),
input_tokens: response.dig("usage", "input_tokens"),
output_tokens: response.dig("usage", "output_tokens")
}
end
result.response
end
private
def anthropic_client
@anthropic_client ||= Anthropic::Client.new(
access_token: ENV.fetch("ANTHROPIC_API_KEY")
)
end
end
end
Usage in a controller or service:
client = Llm::Client.new(namespace: "support_v3")
answer = client.complete(
system: "You are a helpful support agent for Acme Corp...",
messages: [{ role: "user", content: params[:question] }]
)
The cache key uses the last user message in the conversation. For multi-turn conversations you can serialize the full context, but the marginal cache benefit drops significantly as conversation history grows — most repeat patterns in production are single-turn Q&A.
Choosing the Right Distance Threshold
The threshold is the most consequential parameter. Set it too high and you serve wrong answers to dissimilar questions. Set it too low and your cache hit rate collapses.
A principled approach: build a test set of query pairs, label them as “same intent” or “different intent”, and find the threshold that separates them with the fewest errors.
# script/threshold_analysis.rb
QUERY_PAIRS = [
# Same intent — should be served from cache
["What is your refund policy?", "How do refunds work?", :same],
["How do I cancel my account?", "Steps to delete my account", :same],
["What payment methods do you accept?", "Which credit cards can I use?", :same],
["Is there a free trial?", "Can I try before buying?", :same],
# Different intent — must NOT be served from cache
["What is your refund policy?", "How do I contact support?", :diff],
["Cancel my account", "Pause my subscription", :diff],
["Payment methods", "Refund timeline", :diff],
["Free trial", "Free plan features", :diff],
]
client = OpenAI::Client.new
def cosine_distance(a, b)
dot = a.zip(b).sum { |x, y| x * y }
norm = Math.sqrt(a.sum { |x| x**2 }) * Math.sqrt(b.sum { |x| x**2 })
1 - (dot / norm)
end
def embed(text, client)
client.embeddings(parameters: { model: "text-embedding-3-small", input: text })
.dig("data", 0, "embedding")
end
QUERY_PAIRS.each do |q1, q2, label|
e1 = embed(q1, client)
e2 = embed(q2, client)
distance = cosine_distance(e1, e2)
puts "#{label} | #{distance.round(4)} | #{q1[0..40]}"
end
Running this on a typical customer support domain typically shows:
same | 0.0742 | What is your refund policy?
same | 0.0891 | How do I cancel my account?
same | 0.1043 | What payment methods do you accept?
same | 0.0623 | Is there a free trial?
diff | 0.2187 | What is your refund policy?
diff | 0.1832 | Cancel my account
diff | 0.3144 | Payment methods
diff | 0.1941 | Free trial
A threshold of 0.12–0.15 cleanly separates them in this domain. For safety-sensitive domains (medical, legal, financial), tighten to 0.08–0.10 and err on the side of cache misses.
Cache Invalidation Strategies
TTL eviction — the simplest approach: set expires_at at write time and filter in queries. Set TTL based on content volatility:
def store(query:, embedding:, model:, response_data:, ttl: 30.days)
LlmCacheEntry.create!(
# ...
expires_at: ttl ? Time.current + ttl : nil
)
end
- Policy Q&A that rarely changes:
ttl: 30.days - Product documentation pages:
ttl: 7.days - Pricing or inventory data:
ttl: 1.houror skip caching entirely
Namespace versioning — when you update a system prompt or the underlying data changes substantially, increment the namespace and deploy. Old entries expire on their TTL; new traffic fills the new namespace organically:
# Before deploy: namespace: "support_v3"
# After deploy: namespace: "support_v4"
No manual delete needed. The old namespace entries sit inert until their TTL fires. This is the approach I use for most cache invalidation and it eliminates an entire class of cache-correctness bugs.
Active invalidation — for content tied to business events, trigger a targeted namespace flush:
class Plan < ApplicationRecord
after_commit :invalidate_pricing_cache, on: [:update]
private
def invalidate_pricing_cache
LlmCacheEntry.where(namespace: "pricing_faq_v2").delete_all if saved_change_to_price_cents?
end
end
For high-traffic scenarios, regenerate eagerly instead of nuking the namespace: query the highest-hit entries, re-run them through the LLM, and update in place. This keeps cache hit rates high through a data change without a cold-cache period.
Measuring Cache Performance in Production
A semantic cache without visibility is a black box. Instrument it via ActiveSupport::Notifications so you can see hit rates and similarity distributions in your APM:
# In SemanticCache#fetch, wrap the core logic:
ActiveSupport::Notifications.instrument("llm.semantic_cache", {
namespace: @namespace,
cache_hit: !cached.nil?,
distance: cached&.neighbor_distance,
query_size: query.length
}) do
# ... cache lookup or LLM call
end
Wire up a subscriber that reports to StatsD or Prometheus:
# config/initializers/llm_cache_metrics.rb
ActiveSupport::Notifications.subscribe("llm.semantic_cache") do |*, payload|
status = payload[:cache_hit] ? "hit" : "miss"
namespace_tag = "namespace:#{payload[:namespace]}"
StatsD.increment("llm.cache.#{status}", tags: [namespace_tag])
if payload[:distance]
StatsD.histogram("llm.cache.distance", payload[:distance] * 1000,
tags: [namespace_tag, "result:#{status}"])
end
end
The metric I watch daily is cache_hit_rate by namespace. A namespace with <15% hit rate on steady-state traffic either needs threshold tuning or the query patterns are too diverse for caching to help meaningfully.
Watch the distance distribution for misses too. If your cache misses cluster at distance 0.13–0.18 (just above your threshold), consider whether raising it to 0.17 would catch them without sacrificing precision. The histogram tells you that; guessing does not.
Benchmarks: Real Numbers
On a production support chatbot handling roughly 8,000 user queries per day, after one month with the semantic cache deployed:
| Metric | Before cache | After cache |
|---|---|---|
| LLM API calls | 8,000/day | ~3,100/day |
| Average response latency | 1,840ms | 820ms |
| Cache hit rate | — | 61% |
| Monthly API cost | €847 | €358 |
The latency gain is significant: a cache hit costs ~80ms (embedding call + pgvector lookup). An LLM miss costs 1.5–3s. At 61% hit rate, the average response time drops from 1,840ms to 820ms — a faster product as a byproduct of cost reduction.
The embedding call on every request (unavoidable — you need the vector to query the cache) costs roughly $0.00002 per query with text-embedding-3-small. On 8,000 queries per day, that is $0.16/day — negligible against even Haiku pricing for a single LLM completion.
Production Considerations
Concurrency and duplicate writes. Two simultaneous requests can both miss the cache for the same query, both call the LLM, and both write an entry. This is fine. The duplicate entries will be found on subsequent requests, and the nearest one will be returned. Do not add a lock around the write — the complexity is not worth it for a cache with acceptable duplicates.
Tenant isolation. In a multi-tenant app, include the tenant identifier in the namespace:
cache = Llm::SemanticCache.new(namespace: "support_v3/#{current_account.id}")
Tenant A must not receive cached responses generated from Tenant B’s context, particularly when the LLM was given tenant-specific system prompt instructions.
Sensitive data. If queries contain PII — names, emails, account numbers — do not store them verbatim in the cache. Strip or mask sensitive tokens before storing the query string. The embedding still captures the semantic question type, which is all the cache needs for similarity matching.
Cache warming. For predictable FAQ patterns, pre-warm the cache at deploy time:
# lib/tasks/llm_cache.rake
namespace :llm_cache do
task warm: :environment do
client = Llm::Client.new(namespace: "support_v4")
faqs = SupportFaq.where(active: true).order(view_count: :desc).limit(200)
faqs.each do |faq|
client.complete(messages: [{ role: "user", content: faq.question }])
print "."
end
puts "\nWarmed #{faqs.count} entries in namespace support_v4"
end
end
Run this as a step in your deploy pipeline, after the namespace version increments and before traffic starts flowing to the new version. Combined with LLM observability to trace which queries are hitting and missing, you can build a data-driven warming strategy tuned to your actual traffic patterns.
FAQ
What distance threshold should I use for a Rails LLM semantic cache?
Start at 0.12 for general Q&A use cases. This corresponds roughly to queries that share the same underlying intent and would receive the same answer from a knowledgeable human. For safety-sensitive domains — medical, legal, financial — tighten to 0.08. For high-volume applications where false positives are acceptable, you can go up to 0.15–0.18. Always validate with a labeled test set of real queries from your domain before deploying; the right number varies meaningfully by use case.
Does Rails LLM semantic caching work with multi-turn conversations?
It works, but hit rates drop significantly as conversation history grows. The most practical approach is to cache only the final user message as the key, not the full conversation context. This covers single-turn Q&A effectively and captures the most common cacheable patterns in production support and FAQ use cases. For multi-turn conversations with long system prompt context, Anthropic’s server-side prompt caching delivers more reliable savings without the same precision trade-offs.
How do I handle cases where the cached response becomes stale?
Use short TTLs (7–30 days depending on content volatility) combined with namespace versioning. When your underlying data or system prompt changes materially, increment the namespace version (support_v3 → support_v4) and deploy. Old entries expire on their TTL; new traffic fills the new namespace. Avoid trying to selectively invalidate individual entries — the vector-matching semantics make it hard to identify which entries were affected by a data change.
What embedding model should I use for the semantic cache?
text-embedding-3-small from OpenAI is the practical default: 1,536 dimensions, cheap, fast (80–120ms round-trip), and accurate for English. If your app serves multiple languages, text-embedding-3-large handles cross-lingual similarity better — it encodes “refund policy” and “politique de remboursement” as closer vectors. Avoid using large generative models for embedding-only workloads; they are orders of magnitude more expensive and slower for a task where smaller specialized embedding models already achieve excellent accuracy.
A semantic cache is one of the highest-ROI additions you can make to a Rails LLM feature once traffic grows past a few thousand queries per day. TTB Software builds production AI features on Rails — from RAG pipelines to multi-agent workflows. If your LLM bill is growing faster than your user base, we know what to look at first.
Related Articles
Rails PDF Generation: Prawn, Grover, and HTML-to-PDF in Production
Rails PDF generation with Prawn, Grover, and HTML-to-PDF via headless Chrome — when to use each, production setup, ba...
Rails ActiveSupport::Notifications: Custom Instrumentation for Metrics and Production Monitoring
Rails ActiveSupport::Notifications powers custom instrumentation, metrics, and APM integration. Subscribe to events, ...
Rails Error Reporter: Structured Error Handling Without rescue Spaghetti
Rails Error Reporter centralizes production error handling with handle, record, and report. Stop scattering rescue bl...