VentureBeat
Follow
Enterprises are overpaying for simple AI queries — Snowflake's gateway now auto-routes to cut costs up to 3x
Enterprise teams struggle with using single AI models for all tasks, as they are either too expensive for simple queries or insufficient for complex ones. Model routing, which automatically selects the best model for each task, is emerging as the solution. Snowflake's Cortex AI Gateway now features dynamic model routing, allowing users to opt for "auto" selection, which intelligently assigns tasks to models balancing quality and cost. This capability can reportedly reduce token costs by up to threefold, as it prevents overspending on simple questions by using less capable, cheaper models. Other major players like Databricks, AWS, Google Cloud, and Nvidia are also developing similar model routing technologies. Snowflake emphasizes that model routing involves more than just price and performance, but importantly incorporates governance and context. The dynamic routing mechanism employs an "advisor pattern" where a smaller model attempts a task first, escalating to a larger model if needed. Additionally, a classifier, trained on past queries, routes straightforward questions to more suitable, simpler models. This auto-routing is optional, allowing customers to manually specify models if desired. Snowflake integrates routing with its existing data governance, extending role-based access controls from data to models and agents. Open models can run within a customer's region to meet data residency requirements, with all inference staying within Snowflake's secure boundary. Enhanced context, provided by tools like Horizon Context and Cortex Sense, allows cheaper models to handle tasks effectively by removing the need for extensive exploratory work. Agent memory is also incorporated into the context, preventing redundant problem-solving and reducing costs. The competitive landscape for model routing includes platforms like OpenRouter, Nvidia's Switchyard, and Databricks' Smart Routing. Differentiation is shifting towards integrated governance, data locality, access controls, and cost attribution within a preferred platform. When choosing a model router, organizations should prioritize the one that best aligns with their existing data governance and team structures, rather than solely focusing on features or price. The decision on which router to adopt depends heavily on an enterprise's existing data infrastructure and governance model.