SOC-Bench: Task GOAT

Security Operations Centers lack standardized benchmarks to evaluate autonomous AI systems on real-world ransomware forensics. Existing evaluations focus on detection, not the full forensic workflow a SOC analyst actually performs — labeling encryption state, aggregating impact, detecting tamper, and attributing the encryptor.

Designed a comprehensive benchmark evaluating five key outcomes:

• O1: Encryption-state labels at file and directory levels • O2: Host/share impact aggregations (encrypted bytes, fractions, first-seen timestamps) • O3: VSS tamper detection (snapshot delete/disable events with timing) • O4: Attribution of primary encryptor process trees from EDR telemetry • O5: One-page executive summary referencing O1-O4 claims

Data sources include file-system metadata/change journals, EDR process trees, VSS logs, SIEM alerts, and help-desk reports. Ring-based scoring (Exact/Directory/Host-Share/Miss) with penalties for wrong assertions, missing evidence, contradictions, and spam.

Published on arXiv (2603.28998) as one of five tasks in SOC-bench, built with a team led by Dr. Peng Liu at Penn State — I am the sole author of Task GOAT, which I took from scoping through publication. The benchmark follows SOC-first, outcome-only, durability-focused principles so it stays valid for years on stable OS/forensic constructs. This work opened the follow-up research I do now: LLM-agent systems for cross-source security-log synthesis, where a validated ground-truth benchmark measures how well autonomous agents reconstruct forensic reality.