DEV Community
Follow
From the Fantasy of Local Models to a Truly Controllable Sports Agent
The project began as an experiment with locally deployed large language models, with the goal of achieving data privacy and independence from cloud-based model providers. The author purchased a LazyCat AI Pod and deployed a distilled version of the Qwen 32B model, but soon realized that the model's capabilities were limited and its responses were slow and outdated. The author attempted to build a general-purpose chatbot, but it was frustrating to use and could not compete with cloud-based models. Narrowing the scope to a sports scores chatbot seemed more feasible, but the model still lacked knowledge of current scores and sports rules. The author collected rules and documentation and integrated them into the question-answering workflow using vector retrieval, but this approach had limitations, such as context size constraints and information overload. The author also experimented with multi-model routing, but this did not solve the underlying architectural problems. A decision flow node was added to select the appropriate scores APIs based on user intent, but the implementation relied on keyword matching, which proved to be inadequate for handling diverse question types and conversational contexts. The project highlighted the challenges of building a practical and effective question-answering system, including the need for robust intent recognition, API planning, and context engineering. The author's experiences demonstrated that simply deploying a model locally does not guarantee practical value, and that a deeper understanding of the system's architecture and limitations is necessary to create a useful and reliable application. The project ultimately led to a greater appreciation for the complexities of building a successful question-answering system and the importance of careful design and implementation.