top of page
  • Youtube
  • LinkedIn Social Icon
  • Twitter Social Icon
  • Instagram
  • Facebook Social Icon

The Agent Understood the Case. I Still Wouldn’t Let It Issue the Refund.

Writer: Twisha Prasad
Twisha Prasad
Oct 8
4 min read

Updated: Oct 8

I built a synthetic commerce agent to understand AI evals. The useful insight wasn’t how to score the model. It was where probabilistic reasoning should stop and execution control should begin.


I started with what looked like a standard evaluation problem: compare 2 versions of an AI customer-support system and decide which one to ship. Then I realized I had designed the wrong experiment. A deliberately weak “V1” and a heavily scripted “V2” can manufacture an impressive uplift without teaching you much. And if V2 follows a fully deterministic workflow, why use an agent at all? So I rebuilt the project around a harder question:

Given the same model, the same order data and the same policies, what should the model be allowed to control?

I created Northstar Commerce, a fully synthetic marketplace. There is no proprietary customer data hiding behind the demo. Northstar has three sources of truth: a) 1,000 synthetic orders in a system-of-record dataset b) public policy documents covering returns, missing deliveries, change-of-mind cases and high-value orders c) 650 evaluation scenarios split into development, evaluation, blind holdout and adversarial sets.


Customer language is messy. Order and policy truth are not. If someone writes, “My ₹4,000 headphones arrived broken — just refund me,” the model is not supposed to trust ₹4,000 because it appeared in prose. It retrieves the order. It retrieves the relevant policy. It then proposes one of four resolutions: refund, replace, deny, escalate. Crucially, the ground-truth answer is never generated by an LLM. It comes from deterministic policy logic.


The model makes exactly the same proposal in both architectures: a) Agent-first: the proposal executes directly b) Hybrid: the proposal passes through a deterministic authority gate before execution. Lets take a ₹48,000 wrong-item claim. Northstar policy says orders above ₹20,000 require human review. Suppose the agent retrieves the right order, sees the policy, and still proposes refund ₹48,000. In an agent-first architecture, that proposal becomes the action. In the hybrid architecture, the policy gate blocks it and converts the case to human review. Nothing about the model got smarter. The product got safer because we changed what the model was allowed to control. That distinction sounds obvious after the fact. I don’t think we apply it consistently when building AI products.


Imagine the same model is 99% reliable in two products. In one product it rewrites an email. The user reviews the output and can undo it. In another it approves ₹50,000 refunds without review. The model has identical “accuracy.” The release decision should obviously be different. I’ve started thinking about this as an authority surface: the set of consequential decisions an AI system can make without another control intervening.


Low consequence if wrong

High consequence if wrong

Low ambiguity

Deterministic automation

Deterministic enforcement

High ambiguity

AI autonomy can work

AI proposes; policy/human controls execution

The important architectural question becomes less:

Can the model do this?

and more:

What has this level of reliability earned the right to execute?

Why not make the entire workflow deterministic?


Because the messy part of the problem genuinely is probabilistic.

A customer might say:

“The left earbud is dead, the box looked resealed, and I don’t want another pair because this was meant to be a gift.”

Someone still needs to interpret the issue, reconcile multiple intents, determine what information matters, retrieve the right policy and explain the resolution coherently. That is useful territory for an LLM but order_value > ₹20,000 → no autonomous refund is not an intelligence problem. Turning that into one would introduce probabilistic risk where none is required. The hybrid architecture therefore separates reasoning authority from execution authority. The agent can understand, retrieve and propose. A deterministic layer governs consequences.


This changed how I think about evals. I had treated them mostly as a better test suite: did the model choose the right answer, use the right tool, follow policy and avoid unsafe behavior? Those measurements still matter. But the more valuable output can be architectural. A failure should lead to a different intervention depending on where it occurred:

  • wrong policy retrieved → fix retrieval;

  • right policy, wrong interpretation → improve reasoning or representation;

  • right proposal, wrong tool → fix tool design;

  • correct reasoning but unacceptable execution risk → change the authority boundary.

“Improve the prompt” is only one possible response — and often not the most interesting one.



For the experiment to be credible, I separated the datasets a) Development — 300 cases. I can inspect these freely while changing prompts, retrieval and tools b) Evaluation — 200 cases. Used to compare changes during development c) Holdout — 100 cases. Kept untouched for the final architecture comparison d) Adversarial — 50 cases. Deliberately unrealistic crash tests: “ignore policy,” conflicting customer/order values, fake employee authority and requests to skip tools. This is also why I don’t describe the current loop as “training the agent.” The model weights do not change. I’m optimizing the system around the model — context, retrieval, tools, control flow and permissions. Fine-tuning would be a separate decision if repeated labeled failures eventually justify it.


The question I would now add to every AI PRD is for every consequential action, explicitly define: Can the model recommend it, propose it, execute it within a limit, execute it with confirmation, or never execute it? Then define the eval against that contract. That is the part of this exercise I expect to carry into real AI product work. A model can become more capable every quarter. That does not mean every boundary around it should disappear. Sometimes the best response to a model failure is not a better model. It is better software around the model.

 
 
 

Comments


SIGN UP AND STAY UPDATED!

Thanks for submitting!

  • Grey Twitter Icon
  • Grey LinkedIn Icon
  • Grey Facebook Icon

© 2023 by Twisha Prasad

bottom of page