RSS Google AI Blog
Follow
InstructPipe: Generating Visual Blocks pipelines with human instructions and LLMs
Visual Blocks is a visual programming framework that allows users to program by connecting blocks into a flow of blocks, creating an AI pipeline. However, novice users may struggle with setting up and linking nodes from a blank workspace. To address this, the authors introduced InstructPipe, an AI assistant that generates machine learning pipelines with text instructions. InstructPipe consists of three modules: two large language model (LLM) modules and a code interpreter. The LLM modules generate pseudocode for a target pipeline, and the interpreter renders the pipeline in the visual editor for human-AI collaboration. The authors implemented InstructPipe with a two-stage LLM refinement prompting strategy, followed by a pseudocode interpretation step to render a pipeline. Users can describe a desired pipeline in natural language, and InstructPipe automatically generates a corresponding, editable pipeline. The authors evaluated InstructPipe through a two-day hybrid workshop and a user study, demonstrating that InstructPipe allows users to create AI pipelines with significantly lower workload. InstructPipe empowers users to build sophisticated workflows with lower workload, enabling rapid idea prototyping and significantly reducing user interactions. The authors hope InstructPipe serves as a catalyst for future research, fostering innovation in human-AI collaboration and unlocking new levels of expressivity and creativity in machine learning and beyond.