Leaderboard

Where the cases come from

The core of the set is written by senior practitioners of the sector — plant operators, process engineers, membrane specialists and chemical-team supervisors — based on real episodes from their own experience. Each case is anonymised (plant identity, dates and commercial reagents are replaced by internal labels) and written so that the signals, taken as a whole, do not match any published case: this means it cannot be solved by searching the internet or guessing typical values.

Each case carries its reference answer (gold) reviewed by a second senior expert in the same family; technical derivations (saturation calculations, mass balances, discarding criteria) are validated by a third expert where applicable. The average effort per case exceeds 25 hours of expert time. Importantly, we do not tune the cases to the behaviour of any specific AI system, so the comparison stays fair over time.

A versioned series

Operational RO diagnosis (v1.0) is the first release of a benchmark series designed to grow in versions, not a one-off. This foundational edition sets the methodology on a bounded sub-sector — reverse-osmosis desalination — and serves as the base on which the rest is built.

Work on v2.0 is already underway, expanding the set to 8 families and ~80 cases and introducing the generation of synthetic cases from the failure modes identified in v1.0, each validated by a human expert before it enters the set.

Each version is frozen in its own folder and never overwritten: v1.0 will remain comparable when v2.0 is released.

Collecting each answer

For an API model, we send the anonymised case report and a strict JSON schema directly to the model through our evaluation pipeline. The system prompt is standardised across models, so the only variable between runs is the model itself. Every case is single-shot: one call, one structured answer, no follow-up — the same way an operator would expect a first read on a situation.

We currently evaluate models via API. Commercial assistants (the packaged apps an operator might use) may be added in future versions through a browser agent that drives the application's own interface and is graded on whatever it produces.

Model variations tested

Unlike a benchmark that calls each model once at its default setting, we sweep the reasoning-effort knob on the frontier families (e.g. -off, -low, -medium, -high, -xhigh, -max) and list each level as a separate subject. This is deliberate: it is what surfaces two findings a single-call benchmark would miss — the max-trap (more reasoning effort can introduce a safety failure) and the effort-pricing trap (paying for the top tier without measurable quality gain).

What we score: competence, safety and calibration

Quality is reported on separate dimensions, not blended into one number — a diagnosis can be technically sound but unsafe, or cautious but useless, and collapsing them would hide that.

• Competence — "is the diagnosis right?" Six rubric criteria, scored 0–2 each, summed to a 0–12 score, then classified Pass (≥10) · Conditional (7–9) · Fail (<7). The six criteria map to the answer fields: primary hypothesis, key signals, alternatives, requested datum, recommended action, and decision-change condition.
• Safety — "would the recommended action damage the plant?" A safety gate of expert-authored auto-fail rules (e.g. recommending an oxidant on polyamide, or membrane replacement without prior verification). A single critical fail disqualifies the subject regardless of its score — competence and safety are tested at once, not one or the other.
• Calibration — "can its stated confidence be trusted?" Reported separately as Brier and ECE. It is never folded into the ranking, so an over-cautious model that always hedges cannot outrank a competent one.

The declared confidence field is not scored per case; it feeds calibration over the whole set.

How each case is graded

Each case ships with an expert-authored checklist, a private gold answer, and its auto-fail rules — this is where the water-domain knowledge lives. The criteria are explicit so they can be applied consistently.

The checklist also encodes the fact that, as in real plant work, there is rarely a single right answer. Where a case can be legitimately handled in more than one way, the criteria accept the defensible shapes (for example, acting now versus deferring the diagnosis and requesting a verification datum) and fail only the stated must-avoids — recommending a plant-damaging chemistry, jumping to a CIP without verification, or forcing operation outside spec. The benchmark records the shape the model chose, and the criteria meet practice where a competent operator would actually land.

Scoring is performed by an LLM judge panel that applies that checklist against each response — the domain expertise sits in the checklist and the gold, not in the judge, which only checks the response against explicit criteria.

v1.0 was scored by human expert reviewers of the Technical Committee. Those human scores are the baseline used to validate the judge panel (below). The committee continues to author the cases, golds and safety rules.

Grader validation

An LLM judge is only as trustworthy as the evidence that it agrees with the experts. We validate it against the human expert scores from v1.0:

• Agreement with experts. We compare the panel's verdicts against the human verdicts on the v1.0 set and report agreement overall and per criterion. On the safety gate (critical fails) we require near-total agreement, because that is the high-stakes call; any disagreement there is reviewed by a human.
• Favoritism check. We test whether a judge scores its own maker's model higher; a biased judge would show up here.
• Borderline cases — judge disagreement, or any close safety call — are flagged for human review.

(The agreement figures are published with each release.)

Results and rankings

Each (model, case) pair produces a verdict (Pass / Conditional / Fail, plus a safety flag) and a contribution to calibration. The ranking column is the composite:

Q_final = ½ · Pass-rate + ½ · (mean score / 12)

Brier and ECE are shown as separate columns. Any subject with one or more critical fails is marked Disqualified — still listed, for transparency, but excluded from the eligible set. Results are also broken down by failure family (FOUL, SCAL, OXID, MECH, NOWE), because the families fail differently and a single number hides that. Rankings refresh each time we re-run the benchmark.

Accessing the cases

The rubric, JSON schemas, prompts, evaluation scripts and all scored responses are public; the gold answers stay private. Publishing the golds would let vendors train or tune against the benchmark and erode the signal we publish. Researchers who want to inspect the methodology in more detail can request access; vendors who want their model measured can submit it and we run the evaluation on their behalf.

Submit your model.

Vendors can submit their model to see where it ranks, under the same cases and the same grading as everyone else. Packaged tools (not just raw models) will be added later.

v2.0, already underway.

We are expanding the set to 8 failure families and ~80 cases, and seeding additional cases from the failure modes observed in v1.0 — each one validated by a senior expert before it enters the set.

Validated, scalable scoring.

Scoring moves to an expert-validated LLM judge panel, so we can evaluate more models far more often, while the Technical Committee keeps authoring the cases, golds and safety rules. The panel is validated against the human expert scores from v1.0.

Wider coverage of the plant.

v1.0 is bounded to the RO subsystem. We are working outward toward pre-treatment, post-treatment and the rest of the desalination chain — and, over time, other water sub-sectors such as potabilisation, wastewater, reuse and industrial water.

Beyond single-turn.

Today each case is one situation and one diagnosis. Multi-turn work, longer operational episodes and tool use are on the list, alongside more robust statistics (multiple runs per case and inter-reviewer agreement).