SOC-Bench: Task GOAT

Security Operations Centers lack standardized benchmarks to evaluate autonomous AI systems on real-world ransomware forensics. Existing evaluations focus on detection, not the full forensic workflow SOC analysts perform.

Designed a comprehensive benchmark evaluating five key outcomes:

• O1: Encryption-state labels at file and directory levels • O2: Host/share impact aggregations (encrypted bytes, fractions, first-seen timestamps) • O3: VSS tamper detection (snapshot delete/disable events with timing) • O4: Attribution of primary encryptor process trees from EDR telemetry • O5: One-page executive summary referencing O1-O4 claims

Data sources include file-system metadata/change journals, EDR process trees, VSS logs, SIEM alerts, and help-desk reports. Ring-based scoring (Exact/Directory/Host-Share/Miss) with penalties for wrong assertions, missing evidence, contradictions, and spam.

Targeting arXiv publication. Benchmark follows SOC-first, outcome-only, and durability principles. Designed to remain valid for years using stable OS/forensic constructs. Part of the SOC-Bench suite (GOAT, PANDA, FOX, TIGER, MOUSE) for comprehensive SOC evaluation.