Agent-Cache: Multi-Tier LLM Ca... Note

Agent-Cache: Multi-Tier LLM Caching for Valkey and Redis

Agent-cache is a three-tier caching solution designed to optimize LLM operations by reducing token usage and execution time. It uses Valkey or Redis to cache LLM responses, tool outputs, and session states. The architecture includes an exact-match LLM response cache, a tool output cache for function call results, and a session state cache for agent checkpoints. Each tier employs distinct TTL strategies, with LLM responses cached for hours, tool outputs for shorter durations, and session state for active user sessions. Cache keys for LLM responses incorporate prompt hash and model parameters, while tool output keys use tool name and argument hash.Manual invalidation is required, as the system does not track dependencies, allowing targeted cache clearing using Redis glob patterns. In case of Valkey/Redis unavailability, agent-cache defaults to graceful degradation, skipping the cache while allowing per-tier configuration for fail-fast or local fallback options. Observability tools like OpenTelemetry and Prometheus are integrated to monitor cache performance and health. The library provides adapters for LangChain, LangGraph, and Vercel AI SDK, handling serialization for each framework.It is designed for environments already running Valkey or Redis, supporting standalone, sentinel, and cluster deployments with hash tags for efficient key distribution. The primary risks involve stale tool outputs, potential cache key collisions, and memory pressure, mitigated through short TTLs, comprehensive cache keys, and memory policies. Session state loss during restarts is addressed by RDB snapshots or AOF persistence.Agent-cache is beneficial for agents with repetitive prompts or tool invocations, aimed at controlling token costs and leveraging existing Redis infrastructure. However, it is not suitable for tools that mutate external state, highly dynamic prompts resulting in low cache hit rates, or when semantic similarity matching is needed over exact-match. The library effectively bridges the gap between framework-specific caching and general-purpose Redis, best utilized when agent loops and tool behaviors are well understood.