Benchmarks That Can't Lie: Lessons from Building SOC-bench

Last year I got to do something most undergrads don't: design a benchmark task end to end and take it through arXiv publication. Task GOAT is one of five tasks in SOC-bench, a benchmark that evaluates multi-agent AI systems on blue-team security operations. I was the sole author of the task: five graded outcomes for file-system forensics, a ring-based scoring rubric, chain-of-custody evidence rules, and a JSON grading schema.

The follow-up work I'm doing now (ongoing, in the same lab) pushes further: can an LLM agent take a Windows event log from one environment and rewrite it to be forensically consistent with a network capture from a completely different one? Along the way I've collected a few rules about evaluation that I think generalize to anyone building benchmarks for AI systems.

1. Ground truth has to be real, not synthetic-looking

The cheapest way to build a benchmark is to perturb your answer key and grade the model on undoing the perturbation. It's also circular. We anchored on a natively co-sourced dataset pair - a host log and a network capture recorded from the same machine during the same session - so the answer key is physical reality, not something we authored.

2. Validate the grader with a negative control

A grader that gives high scores is useless unless you know what a bad submission scores. Our untransformed input scores ~12%. That floor is what makes a high score meaningful. If you don't know your benchmark's floor, you don't know anything.

3. Never trust a system's self-report

The single most valuable thing we did was run an independent validation pass that re-derived ground truth from scratch. It caught a dataset that our own self-checks had marked as correct - it had quietly borrowed identities from a canned sample. Closed loop, adversarial check, fix, re-validate. That cycle is now my default for anything I build.

4. Separate what's observable from what requires reasoning

The most interesting question in agent evaluation isn't "what's the score?" - it's "which parts of the task actually require intelligence?" Ablation and change-attribution studies let you isolate the constraints that are genuinely load-bearing. In our case, the gap between guided and autonomous runs falls almost entirely on the generative work: synthesizing events that must have happened but left no direct trace. That gap is the finding.

If you're working on agent evaluation or security benchmarks, I'd love to compare notes - links are in the footer.