Eli Bendersky: Rewriting pycparser with the help of an LLM

The author, a maintainer of the pycparser project, shares their experience rewriting it to remove a PLY dependency. They initially used a YACC-based parser, but its growing complexity and maintainability issues prompted a switch to a recursive-descent parser. The author leveraged a coding LLM, Codex, to perform the initial rewrite, which surprisingly succeeded. The process involved extensive testing and iterative prompting to refine the code generated by the LLM. While the LLM generated code was functional, it required significant refactoring for readability, clarity, and efficiency. The author ultimately achieved a successful transition, releasing a new version of pycparser. This post details the challenges and benefits of this LLM-assisted code refactoring project.