AI agents are typically evaluated using fixed benchmarks, resulting in a single score, which provides limited insight into their capabilities. This approach is insufficient, as it fails to reveal where an agent's performance degrades or how effortless its successes were. For data agents, understanding these nuances is critical for effective data discovery, a "needle in a haystack" problem where agents must identify relevant datasets from vague human queries.To address this, an information theory-based approach called Discovery Bench is proposed, which aims to add fidelity to benchmarks. This framework modulates the difficulty of evaluation cases by generating easy and hard variations of each query, allowing for a more detailed understanding of agent performance. The core concept behind modulating difficulty is "surprisal," which measures the uncertainty remaining about the correct dataset given a query. Informative terms in a query have high surprisal, sharply distinguishing the target from others.By adding or removing terms with varying informative power, the difficulty of evaluation cases can be adjusted. This method, called iterative surprisal-based query refinement (iSQR), generates queries with calibrated ambiguity levels (high, medium, low), grounded in bits rather than subjective opinion. This allows for a more objective assessment of difficulty, explaining why specific words were added or removed.Discovery Bench has revealed "cliffs" in agent performance, where a slight increase in ambiguity causes a complete failure, a phenomenon missed by traditional pass/fail evaluations. It also identifies a "sweet spot" for ambiguity, showing that more specificity is not always better for agent performance. This provides actionable signals for improving agents, such as addressing specific failure modes like time-sharded tables or context blow-up.The field is moving towards meta-benchmarking, with other research exploring item response theory and auditing ground truth directly. A critical finding from implementing Discovery Bench was the discovery of significant errors in an established benchmark, KramaBench-astronomy. This highlights the crucial need to "evaluate your evals," as flawed benchmarks lead to incorrect conclusions about agent performance.Furthermore, the method of generating ambiguity sweeps itself needs evaluation. When comparing LLM-generated steering terms with TF-IDF surprisal-grounded terms, the results differed drastically, emphasizing the need for robust evaluation of the evaluation method itself. The conclusion is that evaluations should produce signals, not just verdicts, providing a continuous axis of assessment and scrutinizing the evaluators as much as the agents.