Which model can
check your contracts?

They find the problems. Then they add some.

Spotting a real breach turned out to be the easy part. What separates one model from another is how often it flags a clause that was already fine.

Every model BENCH tried, on the same contracts.

Nothing changes between these cards except the model. A contract counts only when all six rules were called correctly on it, so a model that finds the breach and invents a second one does not score.

Which one to run, and what it costs.

Two questions, in this order. Can you get the outcome at all, and what is the least you can pay for it? Best value is the cheapest model within five points of the top score. The top scorer is marked separately so you can see what the extra money buys.

How BENCH makes a cheap model good enough.

This is the part a leaderboard cannot do for you. A cheap model that looks hopeless on score usually is not making many different mistakes. It is making one mistake repeatedly, and that is something you can fix.

One rule carries almost every failure.

It is not only the cheap one. Each model here has a single rule it reads as stricter than it is written, and that one rule accounts for nearly everything it gets wrong.

The same pattern holds for 9 more models: one rule, misread the same way, accounting for most of what it gets wrong.

Every wrong call, in plain words.

Grouped by model, with repeats counted rather than listed. This is the evidence behind the numbers above.

The harness BENCH ran.

A model is only part of what you ship. What you actually run is a model, a prompt and a harness around it: what it can see, what it can call, how many turns it gets, what happens when something fails. Every model here got this one, unchanged.

• ShapeSingle pass. One model call per contract, no agent loop, no follow-up question, no second look at its own answer. The simplest harness anyone would build first.
• What goes inTwo messages. The instruction above, then the company's six rules and the contract text together in one payload. Nothing else reaches the model: no case law, no past decisions, no company wiki.
• ToolsNone wired up. No search, no retrieval, no document store. Every answer comes from what was handed over in that one request.
• Output limit4,096 tokens. An empty answer aborts the run rather than being scored as a failure, because a truncated reply is a harness bug and not the model's.
• SamplingProvider defaults, no temperature tuning, one review per contract. No best-of, no retries, no voting between runs.
• Transport180 second timeout, transport retries off, and a cost ceiling reserved before every call so a runaway run stops instead of spending.
• Held constantEvery item above is identical on every row. The only thing that changes between models is which model id is called.

Where the contracts came from.

Written for this test, not taken from anyone. A benchmark is only as good as its documents, so here is how these were made and what is in them.

• Where they come fromWritten for this test against a fictional employer, Northstar Software. No client repository, no real employment agreement and no identifiable person is involved.
• How they were builtOne compliant base contract, then 38 variations of it. Changing one clause at a time means a difference in a verdict traces to that clause rather than to a different writing style.
• What is in the set12 contracts fully comply, including ones sitting exactly on the limit at six months' probation or 28 days' holiday. The other 27 carry at least one planted breach, up to one contract that breaks all six rules.
• The hard onesBreaches split between a numbered clause and an annex that replaces it, so neither half reads wrong on its own. The rules tell the model that annexes govern, so this is a reading test rather than a trick.
• Held out30 of the 39 were never looked at while the setup was being written, so the instruction could not be tuned against them.

How it was scored: one judge, rule by rule.

A score is only worth something if the same thing scored everybody. This is what did the scoring and what it did when it went wrong.

• The judgeOne fixed model scores every review on every row. Holding it still is what makes a row difference the model rather than the scoring.
• How it scoresTwo passes per review: general answer quality, then the six company rules one at a time. Rule by rule is what turns a score into a sentence you can go and fix.
• When the judge failsAn unparseable judge reply is retried once, then recorded as that single case rather than throwing away a finished suite.
• What counts as rightA contract scores only when all six rules were called correctly. Finding the real breach and inventing a second one does not count, because somebody still has to re-read it.
• CostProjected from the tokens each run reported, at published list prices. It is not a provider invoice, and it covers the review rather than the judging.

What BENCH would change next.

None of these has been run on this suite yet. Each would be scored on the same frozen contracts before anybody claims it worked.

1. PROMPTNOT RUN YET

   Say what the rule does not require.

   The dominant failure is a model reading “the employee must be paid for it” as “must be paid in full”, so compliant contracts paying half get flagged. One added sentence should remove it.

   Would address: almost every invented breach in the table

2. HARNESSNOT RUN YET

   Ask one rule at a time.

   Today one call covers all six rules at once. Splitting it into six smaller checks costs more calls but gives each rule the model's full attention, and makes a wrong call easier to localise.

   Would address: rules missed when a breach hides in an annex

3. TOOLSNOT RUN YET

   Let it look up past decisions.

   Give the review a lookup over contracts the company has already approved, so a borderline clause can be checked against how the same clause was treated before rather than judged cold.

   Would address: contracts sitting exactly on the limit

The rules, written down before anything ran.

• Notice periodAt least three months' notice once probation has ended.
• Probation lengthProbation may not run longer than six months.
• Non-competeTwelve months at most, and the employee must be paid for it.
• OvertimePaid, or given back as time off. Never just covered by salary.
• HolidayAt least 28 paid days a year, on top of public holidays.
• Invention ownershipWork an employee does in their own time, unrelated to the business, stays theirs.

Methodology: how this was done.

• The rules came firstThe six rules were written and frozen before any contract was drafted, so nothing could be tuned to a result.
• 39 contracts, hardest includedTwelve sit exactly on the limit: probation at precisely six months, holiday at exactly 28 days, a non-compete paying a token amount that still satisfies “must be paid”. The hardest split a breach across a clause and an annex, so neither reads wrong on its own.
• Same work, one judgeEvery model got the identical contract, rules and instructions. One judge scored all 507 reviews rule by rule, giving 3042 separate verdicts.
• What counts as rightA contract scores only when all six rules were called correctly. Finding a real breach and inventing a second one does not count, because somebody still has to re-read it.
• Cost and timingCost is projected from reported tokens at list prices, not an invoice, and covers the review rather than the judging. Timings are wall-clock on a single run.
• Projections are labelledWhere the page shows what a fix would be worth, it is arithmetic over verdicts already recorded, never a second run, and it says so on the card.
• No client dataEvery contract was drafted for this test. No real agreement and no identifiable person appears in the suite.