Trail of Bits Blog
Follow
Can chatbots craft correct code?
The author, reflecting on an AI Engineer Code Summit, challenges the optimistic vision of AI replacing developers. They argue LLMs, unlike compilers, lack determinism, a critical property for software correctness. Compilers provide semantic preservation, ensuring code's meaning remains consistent during transformation, while LLMs, by design, are nondeterministic. This lack of guaranteed output poses risks to security and correctness. The author highlights the ambiguity of natural language prompts. An example reveals how an LLM "fixed" code but introduced errors by misunderstanding context. LLMs struggle with legacy codebases due to contextual complexities, undocumented APIs, and historical constraints beyond their comprehension. They suggest that LLMs excel in basic tasks, but struggle with complex, established systems, potentially leading to incorrect and harmful outputs. The author emphasizes that understanding the deterministic nature of compilers and the probabilistic outputs of LLMs is very important. This is critical for assessing the validity and safety of generated code. The author's argument is that LLMs confidently provide incorrect solutions when they lack adequate historical and contextual information.
