Google Cloud Blog
Follow
Introducing Physical-Engineering’s Last Exam for agents
The Google AI & Infrastructure team, in collaboration with Mercor, has launched Physical-Engineering's Last Exam (PLE), a new benchmark and open competition. PLE assesses a frontier model's ability to complete mechanical, electrical, civil, and structural engineering tasks. This initiative addresses a gap in existing benchmarks, which primarily focus on textbook questions or isolated design problems. Unlike previous exams like Humanity's Last Exam or Agents' Last Exam, PLE specifically measures AI's capability for physical world-building tasks.PLE comprises 85 long-horizon tasks, mirroring real-world engineering projects with an average of 61 files per task. These files include various engineering documents, requiring agents to reason through codes, limits, and calculations to produce production-ready deliverables. The tasks were developed through collaboration with practicing engineers, either authored by experts or co-authored with agent pipelines.Initial testing on 14 frontier models using seven agent harnesses revealed that all models scored below 50% on the private test set. The highest-performing model achieved a mean score of 48.3%. A significant finding was that only 41.3% of top-scoring agent deliverables passed all hard safety and feasibility checks, despite satisfying 91% of individual constraints on co-authored tasks.The choice of model significantly impacts performance, with up to a 50-point spread, while harnesses have a smaller effect (median 3-point spread) but affect run cost. This benchmark offers a practical step toward developing AI tools that can support physical engineers. Researchers can access 29 public tasks, grading tools, and join the open competition at ple-bench.com. Google is sponsoring 13 models for free evaluation on its servers, with prize details to be announced soon.