Predictable AI spend in BigQue... Note

Predictable AI spend in BigQuery: Introducing cost management for generative AI functions

BigQuery generative AI functions allow users to integrate large language models directly into SQL queries for tasks like classification and sentiment analysis. These functions are billed per token, making proactive budget management crucial for large-scale use. Google Cloud has introduced token-based cost management for these functions through daily quotas managed via IAM.Four daily quotas exist: AI input tokens per day, AI input tokens per day per user, AI output tokens per day, and AI output tokens per day per user. These quotas apply globally and cover specific generative AI functions in BigQuery, excluding embedding functions. Administrators can view and adjust quotas directly in the Google Cloud console or programmatically through APIs.To manage costs effectively, users can lower quotas immediately or request increases, which are often auto-approved. Monitoring usage on the Quotas & System Limits page helps detect spikes and right-size budgets. Token quotas act as a preventive, AI-specific guardrail, distinct from broader Cloud Billing spend caps.Enforcement involves stopping work when quotas are reached, either rejecting queries upfront or halting LLM calls mid-query. The AI.COUNT_TOKENS function aids in estimating input token usage without incurring extra costs. By understanding token pricing and estimating usage, administrators can set appropriate quota values to align with budget goals.Best practices include starting with per-user quotas, measuring input data with AI.COUNT_TOKENS, being mindful of the higher cost of output tokens, and iteratively adjusting limits based on observed usage. Implementing these token-based guardrails provides cost predictability for generative AI in BigQuery.
CdXz5zHNQW_TKvYyrBXNC.png