Spring AI Prompt Caching and Chat Memory: Where the Tokens Go — LLM Cost Control 2/4
This section focuses on controlling costs associated with output, conversation history, and repeated static content in AI models. Output and reasoning tokens are significantly more expensive than input tokens, with some models costing up to eight times more for output. Reasoning processes can generate hidden "thinking" tokens that also incur higher output rates. Spring AI offers controls like maxTokens for provider-independent length limits and provider-specific settings to manage reasoning effort. Conversation history, which involves resending the entire chat log with each request, drives up input token costs rapidly. Storing and resending history means even small conversations can lead to substantial token usage over time. Spring AI provides MessageWindowChatMemory to manage conversation history by using a sliding window of a specified number of messages. For very long sessions, VectorStoreChatMemoryAdvisor offers an alternative by storing history in a vector store and only retrieving relevant messages. Repeated static content, such as system prompts or tool definitions, is charged on every request without caching. Prompt caching reduces these costs by storing processed prompt prefixes for reuse. Anthropic and AWS Bedrock allow users to specify caching strategies, while OpenAI caches prompts automatically for requests over a certain token count, though cache writes now incur a fee. Local models like Ollama use caching to improve speed by saving GPU processing time, but there are no per-token charges to reduce. Explicit planning for caching and managing cache keys is crucial for cost optimization with these models.
maxTokensfor provider-independent length limits and provider-specific settings to manage reasoning effort. Conversation history, which involves resending the entire chat log with each request, drives up input token costs rapidly. Storing and resending history means even small conversations can lead to substantial token usage over time. Spring AI providesMessageWindowChatMemoryto manage conversation history by using a sliding window of a specified number of messages. For very long sessions,VectorStoreChatMemoryAdvisoroffers an alternative by storing history in a vector store and only retrieving relevant messages. Repeated static content, such as system prompts or tool definitions, is charged on every request without caching. Prompt caching reduces these costs by storing processed prompt prefixes for reuse. Anthropic and AWS Bedrock allow users to specify caching strategies, while OpenAI caches prompts automatically for requests over a certain token count, though cache writes now incur a fee. Local models like Ollama use caching to improve speed by saving GPU processing time, but there are no per-token charges to reduce. Explicit planning for caching and managing cache keys is crucial for cost optimization with these models.