From Chat to Decisions: Bringi... Note

From Chat to Decisions: Bringing SLMs into Application Logic with Microsoft Olive and ONNX

This document explores Jev-style typed-decision interfaces for accelerating application workflows, focusing on semantic understanding leading to structured actions. It highlights how models can make fast, bounded decisions in applications like order processing, contrasting with traditional rules for explicit constraints and large models for open-ended problems. The core concept is using Small Language Models (SLMs) for frequent, localized decisions, keeping the decision close to the action. This approach is beneficial for vertical applications and embodied systems, where low-latency, task-level intent decisions are crucial.The architecture emphasizes a division of labor: large models for open-ended tasks, small models for bounded semantic decisions, and deterministic code for constraints. The document details a process of fine-tuning a general decision component using Qwen3.5-0.8B, employing LoRA fine-tuning and exporting to ONNX. This workflow serves as a concrete path from data to a deployable decision capability, focusing on learning bounded choices rather than long answers.Preprocessing expands each question into training examples, ensuring decision types like choice, noul, and score align with specific output formats. The Olive workflow, using Microsoft Olive, standardizes fine-tuning settings and ensures a reproducible process. A key distinction is the use of standard full-sequence language-model SFT, not the Hmm project's candidate-letter mixture. The export process uses ModelBuilder to package the base model, adapter, tokenizer, and configuration into a deployable ONNX unit.Navigating version boundaries and compatibility issues during the export process is highlighted as a critical learning point, emphasizing the need for robust version management. Testing involves two stages: ensuring the model runs and verifying that its decisions are usable by the application. This includes extracting probabilities from option-letter logits to interpret the model's output as distributions for choice, noul, and score types, providing a measure of concentration rather than calibrated correctness.