Android Bench 2.0: Pushing the... Note

Android Bench 2.0: Pushing the frontier with challenging long-horizon tasks

Android Bench has been updated to version 2.0, introducing long-horizon tasks (LHTs) that mirror complex, multi-day engineering challenges. This upgrade also includes agentic evaluation, starting with agents from corresponding model providers. The previous benchmark focused on simpler, incremental code changes, but the new LHTs include tasks like app dependency upgrades, feature additions, and building apps from scratch. Traditional binary scoring is being replaced with continuous scoring to better reflect the nuanced progress made on these complex tasks. This new scoring method considers functionality, visual fidelity, and avoids regressions, with penalties for instruction deviations. Currently, the highest pass rate for LHTs is around 28%, a significant drop from the prior benchmark's success rate. AI models demonstrate strength in writing new code over refactoring existing code, excelling at established deterministic transformations. However, they struggle with tasks requiring runtime validation, breaking framework changes, or knowledge gaps in unreleased libraries. Porting cross-platform apps to Android remains a particularly challenging area for AI. The evaluation now incorporates commonly used agents to assess performance in agentic workflows. New models like Gemini 3.8 Flash and OpenAI’s GPT-6 have been added to the leaderboard. Android Bench 2.0 aims to provide transparency and empower developers by offering a more robust environment for evaluating AI for Android development. Feedback on the updated methodology and leaderboard is encouraged for future improvements.