Advancing AI with the
world's human expertise

We created the STEM Reasoning benchmark to evaluate frontier models on graduate-level scientific questions that require deep domain knowledge rather than pattern-matched recall. It comprises n=73 sample tasks spanning Physics, Biology, Chemistry, and Mathematics across 38 subareas, verified by domain experts. Samples include a prompt with the problem formulation, an expert-verified golden answer, and model evaluation across a variety of frontier models. Optionally we can provide a golden chain-of-thought reasoning path, CoT rubric and multimodal inputs, as exemplified in specific samples. Models are evaluated across 3 independent runs per prompt; we report both Pass@3 (at least one correct answer) and Pass^3 (correct on all three runs). Pass@3 ranges from 15.1% to 67.1% across the 16-model panel; Pass^3 rates fall as low as 4.1%, confirming real headroom across all sixteen frontier models evaluated.

• Prompt grounded in a specific research paper
• Expert-verified golden answer
• Golden chain-of-thought reasoning path
• Multi-model results at 3 turns per prompt
• Answer rubric + CoT rubric (optional)
• Multimodal variants available

We created Terminal Bench to evaluate AI agents on hard-tier sysadmin and DevOps terminal tasks grounded in real operational workflows. Here we present two sample tasks: filesystem-permissions-hard and postmortem-oncall-hard. Each ships with a deterministic verifier (test.sh + test_*.py) that scores 15 atomic rubric checks across three independent rollouts per model (45 binary outcomes per model) with a golden solution (solve.sh) validated to drive the reward signal to 1.0. Twelve of the fifteen panel models finish with pass@3 = 0; only Grok 4.5, GPT-5.6 Sol Pro, and Muse Spark 1.1 solve filesystem-permissions in one of three rollouts, and no model solves postmortem-oncall. Each task is graded on 15 atomic checks rather than a single pass/fail, so the eval still separates models sharply even when almost none reaches a complete solve (Grok 4.5 clears 82% of the checks, MiniMax-M3 just 13%), a dense, trainable signal.

• Task prompt + containerized environment (Harbor format)
• Automated test.sh verifier with XML reward output
• 15 atomic rubric checks (test_*.py modules)
• 3 rollouts per model with per-check pass/fail logs
• Golden solution (solve.sh) driving reward to 1.0
• Cross-model comparison and failure analysis

We created Office Reasoning to evaluate models on the document-grounded reasoning work of financial analysts and enterprise knowledge workers. Here we present three sample tasks that require locating evidence across multi-page PDFs, extracting precise values from dense tables, applying accounting and calendar conventions, and producing a single verifiable numeric answer. Each sample includes a structured evidence map with page and bounding-box citations, a human-authored gold reasoning trajectory, and a CoT rubric with atomic scoring criteria. Source documents are drawn from real U.S. Treasury Bulletins and SEC filings.

• Prompt grounded in one or more source PDFs
• Evidence map with page and bounding box references
• Verifiable ground-truth answer (exact-match checkable)
• Gold reasoning trajectory (human expert)
• CoT rubric with atomic criteria
• Multi-model evaluation results