SOR-Bench: a held-out benchmark for AI on enterprise systems of record

01/Abstract

Enterprise AI is increasingly pointed at systems of record such as Workday, yet mainstream benchmarks measure mathematics, coding, or general knowledge rather than operational competence on those systems. SOR-Bench is a held-out benchmark family that measures whether an AI system can correctly answer the questions required to operate an enterprise system. Its first instance, Workday TextQA, contains 2,214 real, customer-permissioned Workday administration questions across 31 modules and three difficulty tiers, each paired with an expert-verified solution. Responses are graded by rubric on correctness, specificity, evidence, and configuration safety. To prevent contamination, the question set and the answer key are withheld, and every system, including those built by the maintainer, is evaluated on the identical private corpus. Aggregate results are published on the leaderboard; full methodology, grader calibration, and failure analysis are documented in the technical report.

The gap this targets is well established in evaluation research: strong scores on general-capability suites that probe broad knowledge or reasoning do not transfer to the long-tail, configuration-specific knowledge that operating a system of record demands. SOR-Bench is therefore designed for two properties evaluation work treats as load-bearing. The first is construct validity, meaning every item reflects a task a practitioner actually performs, so the score estimates the capability of interest rather than a correlate of it. The second is resistance to contamination, the standard way public benchmarks decay once their data enters training corpora. A score here is meant to estimate deployable competence on the system, not familiarity with the test.

02/Benchmark and instance

SOR-Bench is the standard. A suite is one instance of it, defined by a single system, a single modality, and a single version, named accordingly: SOR-Bench (standard), Workday (system), TextQA (modality). Workday TextQA is the first instance; its version is listed in the spec.

The decomposition follows how durable benchmarks are structured: it separates the capability under test from the surface it is tested on, so results stay comparable as systems and models change. Modalities isolate distinct competencies rather than collapsing them into one number, with TextQA measuring grounded question answering and forthcoming agentic and tool-use suites measuring execution. Versioning is deliberate, because enterprise systems ship on a fixed release cadence; items that depend on release-sensitive behavior are rotated between versions to preserve temporal validity and limit overfitting to a frozen snapshot. Upcoming suites span further systems and modalities.

03/Method

Corpus

2,214 customer-permissioned Workday administration questions drawn from real support, implementation, and audit workflows rather than synthetic prompts. Items are stratified across modules, administrative domains, task types, and three difficulty tiers so aggregate scores are not dominated by any single area, and each is de-identified to remove tenant and employee data. Sampling for the genuine operational distribution is what gives the benchmark construct validity: it measures the work itself, not a convenient proxy for it.

Reference answers

Each question is paired with an expert-verified solution authored by Mando's network of 200+ certified Workday practitioners and checked by independent reviewers. The standard is operational correctness rather than plausibility: whether a practitioner could act on the answer in a live tenant without introducing risk. Disagreements are resolved by adjudication, and inter-reviewer agreement is tracked to keep the answer key internally consistent.

Scoring

Responses are graded against the reference on a multi-dimensional rubric spanning correctness, completeness, Workday specificity, evidence and citation quality, hallucination control, and configuration safety. Grading uses a model-based judge calibrated against human ratings, a now-standard evaluation technique, with rubric weights and calibration documented in the report. Fluent but ungrounded answers are penalized, and abstention is measured separately as coverage so a confident wrong answer never outscores honest uncertainty.

Held-out protocol

The scoring corpus and the answer key remain private, and no system, including the maintainer's, receives them before evaluation or fine-tunes on them. Public samples are representative of the distribution but are never part of the scored set, and release-sensitive items rotate between versions. This is what closes the train-test leakage that quietly inflates scores on published benchmarks.

04/Held-out protocol

A score is only meaningful if the system has not already seen the answers. Benchmark contamination, where evaluation data leaks into training corpora, is a documented and growing failure mode of public leaderboards: once a test set is published, models can memorize it, and the headline number then measures recall rather than capability. The effect is hard to detect from the outside and tends to overstate exactly the systems trained on the most web text.

SOR-Bench follows the logic of a professional certification exam. The full question set and the official answer key stay sealed, every system is scored on items it has never seen, and release-sensitive content is rotated between versions so the target cannot be quietly studied over time. Because the maintainer also submits systems, the rule is enforced internally: those systems are evaluated on the identical private corpus with no prior access and no fine-tuning on it. This is the condition under which a result reflects genuine ability rather than memorization.

05/Results

7 systems are scored and reported three ways: an overall accuracy figure, a breakdown by Workday module, and a breakdown by difficulty tier, alongside mean semantic similarity to the reference and a coverage figure that accounts for abstention. The disaggregation is deliberate. A single headline number hides where a system is actually reliable, and a model can look strong on common lookups while failing the harder, multi-step, release-sensitive questions that consume most administrative time, so module and difficulty cuts are where the signal lives.

We publish aggregate results but never raw questions or per-item answers, both to protect the held-out corpus and to keep the comparison contamination-resistant over time. See the full leaderboard for current standings, or the report for grader calibration, failure-mode analysis, and per-system detail.