Bringing Muse and SAM 3.1 to C... Note

Bringing Muse and SAM 3.1 to ChatbotX: An Omnichannel Chatbot That Handles Voice and Images

We integrated Muse Spark, Muse Image, Muse Voice Transcribe, and SAM 3.1 into ChatbotX, an open-source omnichannel chatbot platform. This allows a single API key to enable these AI capabilities across various chat channels without new service deployments. ChatbotX supports Messenger, Instagram, WhatsApp, Telegram, Zalo, TikTok, Threads, and Webchat, offering a flow builder, AI Agents, a shared inbox, and APIs. Our hackathon focus was the AI layer, integrating Muse as a new model provider for AI Agents and AI flow steps. AI Agents powered by Muse can automatically answer customers, process images and documents, and perform actions like tagging or handing off conversations. Within flows, Muse transcribes audio, extracts structured data, generates images, and crops specific objects.The integration involved several new capabilities, including automatic replies with action-taking for AI Agents using Muse Spark and Wonder models. Flow steps now include Extract Data, Generate Text, Analyze Image, Generate Image, Edit Image, Speech to Text, and Segment Image, each leveraging specific Muse or SAM models. For example, Speech to Text uses Muse Voice Transcribe to convert WAV recordings to text. The Segment Image step, powered by SAM 3.1, can identify and crop objects from images.An example flow demonstrates handling customer complaints involving audio and product photos. It transcribes audio, extracts order details, segments the product image, analyzes visible damage, and generates a personalized reply. The architecture routes customer messages through integration workers for lightweight AI tasks and heavy queues for compute-intensive tasks like image generation or speech-to-text.Muse API keys are securely stored per workspace, with all model calls handled by workers for reliability. AI Agents running on Muse gain enhanced tools, including an image reader, document reader, and improved AI Agent Actions. Flow steps are designed to read input from and write results to custom fields, enabling chaining of operations. Image-related steps handle various formats and ensure cross-channel compatibility. Speech to Text rigorously validates WAV file formats before sending requests to Muse.Text and data steps, such as Generate Text and Extract Data, leverage Muse Spark and Wonder models for text generation and structured data extraction. Extract Data uses Zod schemas to ensure validated, structured output from text, images, or files.