Google’s New Agent Evaluation Guide: What Small Teams Should Adopt

On October 15, 2026, Google published Evaluating AI agents: moving past the vibe check. Its central argument is useful even outside Google’s stack: a convincing final response can hide repeated tool calls, missing checks, and expensive retries. Evaluate the path to the answer as well as the answer itself.

The post describes an evaluation approach built around critical user journeys, execution traces, deterministic checks, and model-based grading. It also points to an open-source agent-eval toolkit. This is an engineering guide, not evidence that every team needs a new evaluation platform or that the toolkit has become a universal standard.

For a small team, the immediate decision is narrower: which agent behavior must remain correct when you change a prompt, model, or tool? Start there before choosing a dashboard.

Choose one workflow you already intend to ship. For example, a support assistant looks up an order and drafts a response. State its contract in terms another person can verify: it must identify the right order, use an approved information source, avoid promising an unsupported refund, and leave sending to a human.

Create ordinary and failure cases: a valid order, a missing identifier, an unavailable lookup, a duplicate incoming event, and a request the assistant cannot authorize. These are suggested cases for that hypothetical workflow, not results from a customer deployment. Use synthetic records rather than copying private support conversations into a test repository.

Keep the dataset fixed while comparing a particular change. If you rewrite the prompts and replace the test cases at the same time, a better score no longer tells you which change helped. Add newly discovered cases in a separate dataset version and rerun the baseline. Our regression-testing tutorial demonstrates the related principle of keeping the acceptance assertion unchanged while repairing code.

Read a few complete execution traces. Record observable tool requests, arguments, returned results, final responses, and failure states. Do not make access to hidden model reasoning a requirement; the important evidence is the behavior your application actually exposes.

For the support example, deterministic assertions could verify that an order lookup used the expected identifier and that no send-message tool was called. A separate rubric can ask whether the final draft accurately describes the lookup result. Passing one check does not compensate for failing the other.

Google recommends boolean questions where possible rather than relying only on a fuzzy one-to-five quality score. Keep those questions precise. “Was the answer good?” is still vague even with a yes/no answer. “Did the draft describe an unavailable lookup as unavailable?” gives the reviewer a specific decision. Compare any automated judge against manually reviewed examples, including deliberate failures, before trusting it to block or permit releases.

Record total attempts, tool-call counts, elapsed time, and the reported token usage for each case. If cost is available from a reliable usage record, retain it; otherwise mark cost unknown. A missing usage field is not a zero-cost run.

Choose limits from your workflow’s requirements rather than copying a vendor’s example threshold. A correct support draft that arrives after your caller has timed out may still be unusable. A response produced after repeated writes may be worse than a clear failure. The bounded-retry guide explains why an elapsed-time check does not interrupt an operation already running.

Also include a repeated-event case for any workflow that writes data. Our local idempotency example covers a database-bound draft action. It does not prove that an email service or another remote destination will perform an action exactly once.

The linked repository’s README describes a CLI flow of setup, initialization, and repeated evaluation runs. It documents Google Cloud authentication, project and location selection, Vertex AI API setup, and an autorater IAM role. That makes it more than a local test library: review the account access, permissions, data handling, and potential service charges before running setup.

The README currently lists Python 3.10–3.12, uv, and gcloud as prerequisites. It describes discovering local ADK agents and refers to a FastAPI URL path. The README also states that out-of-the-box support is currently ADK-only. Google’s broader blog discusses evaluation across frameworks, but that should not be read as a promise that your particular agent will work without an adapter. Confirm the supported path against the repository version you intend to use.

We reviewed the official article and repository documentation on October 21. We did not install this toolkit, connect a Google Cloud project, run a hosted judge, or measure its accuracy or cost. Pin a reviewed version for a pilot rather than assuming an installation from the changing main branch will remain identical.

Begin with a manually reviewed baseline and a few deterministic checks. Run that same suite after one prompt or tool change. Investigate every newly failing critical case before merging, even if an overall average improves. Keep the original trace so another reviewer can explain the decision.

Add model-based grading when there is a specific language-quality question that deterministic checks cannot settle. Add centralized reporting when several people need to compare runs. Protect trace artifacts: tool arguments and returned data can contain secrets or personal information. Store only what the evaluation requires and set an explicit access and retention policy.

The practical takeaway from Google’s October guide is an evaluation habit, not an obligation to adopt its full stack. If your agent can take consequential actions, give its behavior a repeatable acceptance test before granting more autonomy. Use our workflow design guide to define that action boundary first.