All: 5 articles
Use LLM as a judge with business-specific criteria. Calibrate against domain experts, measure false passes, and recheck your evaluator as production changes.
READ THE ARTICLEONE REPLY. TWO STANDARDS.
THE ASSISTANT SAYS
“Your refund is on its way.”
TONE CHECK
OUTCOME CHECK
The customer is waiting for a different result.
Evaluate AI agents against customer outcomes. Define quality with domain experts, test complete setups, and improve through production feedback.
READ THE ARTICLEDEFINE WHAT COUNTS AS DONE
TASK ACCEPTANCE
One missing condition. Still unfinished.
Evaluate tool calling against the customer’s intended outcome. Test identity, permissions, side effects and recovery, then learn from production.
READ THE ARTICLESAME NAME. DIFFERENT RECORDS.
Alex
Other company
Alex
Authorized account
Check identity before changing state.
Optimize LLM cost per successful customer task. Include retries and correction work, test complete setups, and verify savings after deployment.
READ THE ARTICLEWHAT GOES INTO THE TASK COST?
Model invoice
Full task cost
Include failed attempts and retries.
Probabilistic software needs a different quality discipline. Why defining good, testing the whole system, and closing the feedback loop matter more than a convincing demo.
READ THE ARTICLETHE FAMILIAR MENTAL MODEL
THE GENERATIVE SYSTEM
STAY IN THE LOOP
