Planet Python
Follow
Go Deh: Finding Flaws in Technical Documentation
The author developed a novel approach to documentation testing using AI. After updating project tutorials and documentation in Markdown files, they considered using AI to test their effectiveness. They instructed ChatGPT to create an isolated chat environment where it would read the documentation and attempt to use the described tool. The AI agreed, generating a detailed prompt for the new chat and monitoring its progress.The AI then reported back on documentation deficiencies it found. For instance, it noted the documentation lacked an explicit statement about the tool's intended Linux use, despite providing setup details for Linux and WSL. This led the AI to try running the tool in a Windows environment, highlighting an implicit assumption by the author that needed clarification.The author appreciates this method of having AI act as a new user, actively trying to complete tasks and reporting encountered issues. This goes beyond mere wording review, providing a practical test of the documentation's followability. While the AI may present invalid points, human judgment remains crucial in deciding what changes to implement.The author emphasizes that documentation must serve both human readers and AI assistants. With a growing number of users likely to employ AI for project interaction, documentation needs to be clear enough for direct human use and for AI to effectively assist users. This means considering how well documentation supports AI interaction, not just writing exclusively for AI.Ultimately, the author proposes testing documentation similarly to code, by assessing whether a user, human or AI, can successfully follow it to achieve a task. This innovative testing method offers a new perspective for improving documentation effectiveness on future projects.