Council Post: Why Enterprise AI Needs A New Approach To Evals
Chhaya Methani is an AI/ML leader driving enterprise innovation with expertise in AI, recommendation systems and evaluation frameworks.gettyAs agents move from answering questions to performing tasks, teams mu...
Chhaya Methani is an AI/ML leader driving enterprise innovation with expertise in AI, recommendation systems and evaluation frameworks.
getty
As agents move from answering questions to performing tasks, teams must continually rebuild their evaluation layer (commonly known as their “evals”) to match their system’s capabilities. That is easier said than done, because the process creates challenges many teams are not yet ready for.
Leaders need to stop asking, “Did the agent respond well?” Instead, they must ask, “Did it finish the task correctly, safely and efficiently?” Open benchmarks can hint at progress in solving complex problems, but they cannot reliably predict performance on business-specific scenarios. In practice, most companies need custom benchmarks that simulate the workflows they care about and measure the success metrics that matter.
The Core Shift
In my experience working on agentic systems (AI systems that can take actions) and chat assistants since the release of ChatGPT in 2022, my team has rebuilt our evals strategy multiple times to keep pace with how fast things are changing. We did this largely to identify issues before they impacted customers, but the process also helped us understand how models were improving. This, in turn, informed the redesign of our agentic system itself, during which we incorporated an evaluation harness (a controlled test environment) to improve our agents' capabilities.
Why This Matters Now
With effective harnesses that assist with running models in dedicated environments, agents can now carry out complex tasks over long periods of time. That makes it harder to measure their efficacy, since they can take different trajectories to accomplish the same goal. Evals must adapt by creating sandboxed environments, or “worlds,” where agents can explore multiple paths and be graded by the changes they make to that world.
That is a different game entirely. In this process, grading moves from evaluating steps to evaluating the final state. Often, that means looking at specific database updates, messages sent, tools called for and other relevant side effects to assess the system’s effectiveness.
How To Build The Harness
Before designing a harness, business leaders must clarify what success looks like for their agentic systems and the scale at which those agents must perform. This is surprisingly hard to do in productivity scenarios.
The more crisply you can define this goal, the better the ROI from agents. I focus on the following key criteria:
• Lead With Specs: Detailed specs become even more important for agentic evals. What workflow is the agent automating? How would humans approach it? Create a complete decision tree for each scenario.
• Define The Goal: What does success look like for the agent? Is it automating invoice processing, lead generation or HR processes? The tasks each of these agents needs to perform can look very different.
• Create A Benchmark: This is often the most critical step in producing reliable insights that all stakeholders can trust. The benchmark needs to capture the context of the users and the workflow they are in. The agent’s responses to the user query or task are known in advance and are recorded as ground truth (the expected correct outcomes). It is important that the ground truth not be too rigid, as reasoning systems are inherently adept at finding novel ways to solve tasks.
• Create An Evaluation Simulator Or Evaluation Harness: The agent needs a mock environment, complete with mock databases and mock tools, to create the agent’s world. For every test run, the agent receives a query, reasons over it and issues multiple calls to configured tools to complete the task.
• Interactive Agentic Scenarios: When building evals for interactive use cases, it is not uncommon to insert a different agent that simulates a user of the system. However, multiple agents increase the system’s unpredictability, so the scope of the user agent must be carefully defined.
Once the evaluation harness is running and the agent has completed the task, how do we measure performance?
How To Grade The Work
Since an agentic system takes many actions to complete a given task, you need multiple graders to assess various aspects of the agent’s work. I've found it's best to separate what the agent said it did from what it did and whether it did it correctly. This usually means having a variety of graders looking at specific aspects of the trajectory, database records, etc.
Consider designing around the following aspects:
• Assess The World State Using Automated, Rule-Based Checks: For example, if the agent was supposed to book a flight, did it call the tool to book the flight with the right location and date parameters—and update the database with a new row containing the correct booking details? Or did it delete the row when there was a flight cancellation?
• Required Tool Calls: For example, in a document search task, the agent can call the search tool repeatedly, but the grader should verify that the tool was called at least once in the trajectory. This helps separate a genuine response from a hallucinated response.
• Response Quality And Correctness: To assess response quality, define rubrics that check for a single aspect of the response, such as politeness or a key fact that the agent should have provided. For example, if a user asks a returns agent about the return window, did the agent say “30 days,” or did it invent another number?
Lastly, human evaluation is necessary for disambiguating samples. Each agent can have a slightly different interpretation of the terms, which can affect the evals. Human review is essential to remove ambiguity in wording and produce stable, reliable evals.
Why This Pays Off
Once the benchmark is ready, teams should continue adding every failure and customer complaint as a new sample in the eval. This ensures those cases are accounted for in system improvements—and that no regressions land in production later. Evals are created for hill-climbing on user-specific tasks, and as systems get better, benchmarks eventually get saturated with high accuracy. This usually means you need harder samples or more data from system users.
A scalable eval harness can ensure that regressions are caught before they reach production, helping teams ship with confidence. The organizations that benefit most from increasingly capable agents will be those willing to evolve their evals alongside them.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?