I loved reading Garrett Lord’s essay from two weeks ago about the strategic importance of evals. If you’re an agent-builder company, an AI deployment co, or the head of AI transformation at an enterprise, you’d have really felt what Garrett was trying to say. It’s a b*tch of an experience trying to eliminate edge cases, when your base product has a very broad premise and promises to be a lot of things for a lot of people. At that point, evals aren’t a part of the job - they’re the whole d*mn job.
Anyway, that essay inspired me to share how we’ve been thinking about evals at Ressl AI. If this is of interest, we’d love to chat -> https://calendly.com/arushi-gandhi
A crucial piece of any agent deployment is what we like to call as the staging environment. This is a step where you would try your best to simulate a world for the agent to operate in - which is as close to the real scenarios your agent would operate in post-deployment. You use this to evaluate the different components of the agent - prompts / context / models / harness etc. You’d obviously also use this to understand what the strengths and weaknesses of the software you’ve built are and where edge cases lie. With new models coming in every other week, a simulated environment becomes an important piece of the puzzle.
A simulated environment has 5 core pieces:
| Component | What it means |
|---|---|
| softwares | (these are mock versions of the applications or tools the agent will be interacting with. These need to be high-fidelity - basically a high quality replica of the real software.) |
| data | (this is the initial state of the data within the softwares - as in what will be returned on calling the APIs of the softwares we have simulated. This can be real if you’re working with your own data or programmatically generated through some grounding) |
| agent | (harness + model - this needs to be switchable with fair amount of use for a true simulation environment) |
| tasks | (set of prompts or queries that are domain relevant and economically valuable - ideally you also need a script-ish to be able to extrapolate a larger set of queries from few and where a lot of nuance lies - these can also be extracted from prod queries - which is where the line between online and offline evals starts to blur) |
| verifiers | (this is probably a place to spend a lot of artistic time on and really nail the rubric by which you’d want to judge your agent performance on) |
Through the 5 different components, realism and diversity is key. You need it to be as close to reality as possible without losing nuance of the breadth of possibilities your platform could be used for.
We believe the toughest part about simulating an environment is to have high-fidelity clones of the software that the agent will act on within those environments. You can't run quality evals / benchmark without having mocked softwares. The SaaS era understood this, which meant every software (or at-least the enterprise softwares) had a staging environment to test any configuration or customisation change within.
We don't have staging environments for agents yet. And the prev approach doesn't work now, because:
You'd need to run the eval repeatedly. Let’s say you run an eval and it alters the sandbox of SoR A. And, now you need to run it again for a different parameter - the amount of effort the get the sandbox data back to the state it was before run #1 is way too high, if not impossible. The hosted sandbox is just one live instance. The moment you want scale and the ability to run multiple parallel runs, you need it dockerized: offline, resettable, forkable.
The obvious reason why building software clones is tough is that it is operationally difficult - imagine having to make clones of all the services your agents will touch across all customers when you’re not even in the business of mocking software.
But, the other and larger reason is it’s actually pretty tough. You might be able to vibe code the surface level GUI details and API spec. But the mutations and second order effects are tough. How the hell would you teach your clone that when you convert an lead to an opportunity in Salesforce, it also creates an account and a contact. (We’re calling them mutations so let’s go with that!) And, getting those inherent second order effects of the software replicated in the clone is really tough.
What we’ve learnt to be the bar for a valid clone = runs in Docker, no internet, produces identical outputs to a licensed instance. Targeting ~96% match (clone vs. real Salesforce on same inputs).
(Like you get it, right? To get it 100% of the way through you’d have to have the codebase of the company whose software you’re cloning. Basically. LOL.)
We’ve heard from AI researchers and engineers that they end up spending a minimum of 1 week to clone a single piece of software which is really only 60% of the way there. Now imagine doing this for the 300+ softwares that your agent builder might want to touch. And you can’t vibe code a high-fidelity clone. Customer credentials and one-off sandbox access and buying licenses- all only goes so far.
All AI application layer companies we’ve spoken to about this have mentioned some flavour of the above to us. They only have agent traces and because they don’t actually know the production data / the metadata in the systems their agents touch in their customer’s instance - they have never been able to run a very high quality eval process.
It's hard to pinpoint why the agent failed. Mocks are getting outdated by the week as software companies are pushing updates / AI features. A LOT of engineering bandwidth is spent on just the infra to run simulation environments / evals.
We’ve heard that some application layer cos have been using ServiceNow's EnterpriseOps’s gym, tweaking it a little and using it for OSS model evaluation - which is a great early sign of where we’re headed.
Apart from continuous evals - another use-case we’ve seen for simulation environments is public benchmarks. If you’ve read TauBench by Sierra, or EnterpriseBench by Surge, you know what I’m talking about. A good public benchmark that gets cited by model labs starts with a good emulation of enterprise environments. Even with FDE teams, customers are incredibly expecting customer-specific benchmarks or a technical reporting of how the vendor’s software product will operate in the customer’s software / data etc. Benchmarks are a new procurement requirement.
So, how do you build a high-quality clone?
As we mentioned earlier, mutations are hard. Take, for example, a button in Salesforce called convert lead. When clicked, it marks the lead status, a field in the lead table, as converted and creates a new opportunity, another table entry. The hard question is: how can a clone ensure that this happens in the same way as the original software?
For every software system, we create three artifacts:
A map of the starting points, or the set of APIs the external world interacts with. The rough internal database schema of the software. The API to database table mapping.
The rough algorithm for cloning looks like this:
In the example above, convert_lead is the starting point. It is a POST endpoint that takes lead_id as an input parameter.
From our artifacts, we first figure out the list of tables this POST request could touch. Then we call the relevant GET endpoints for those tables and compare the before and after state to figure out what mutations actually occurred. This gives us L1 lineage.
Next, we find other endpoints that also take lead_id as an input parameter. We then figure out the database tables those endpoints touch and repeat the same process. This gives us L2 lineage.
In theory, this approach does not guarantee a 100% replica. But for most practical purposes, it gives us a realistic behavioral clone. We are working towards making the starting API state-conditioned, adding integration side effects, modeling permission systems, and handling other hidden behaviors to bring the clone as close to the actual software as possible.
Anyway, as always, reach me at arushi@ressl.ai / +14156035412 / https://calendly.com/arushi-gandhi for any feedback / questions!