DEV Community
Follow
I shipped an LLM efficiency + security kernel — and deleted my own best idea
The author attempted to improve Large Language Models by orchestrating multiple LLMs together, a concept termed "mitosis." This approach involved splitting tasks, having LLMs compete, and then synthesizing the best answer. However, rigorous testing revealed that this method worsened correctness, decreasing passing tests from 95% to 83% while increasing costs significantly. After confirming these negative results across three independent experiments, the author deleted the failing feature. The core lesson learned is that an idea that sounds good in a pitch may not survive actual measurement. Instead, the author developed and shipped BIOMA, a lightweight, provider-agnostic kernel that preprocesses LLM requests. BIOMA employs three key mechanisms: efficiency through context "apoptosis" to reduce token usage, security via a "cognitive firewall" for secret redaction and flood detection, and speed through an efficient signaling system. The efficiency mechanism typically reduces input tokens by 80% and can achieve up to 97% reduction. The security features successfully prevented any secrets from being leaked during red-teaming exercises. BIOMA is designed to work with any LLM provider without vendor lock-in. The code is source-available under a license that allows free use for non-competing purposes and converts to MIT after two years. The author emphasizes the importance of measuring everything and retaining only what is validated by data, even if it means discarding the initial project goal.