Part 1 was event logs only. This round I added a second evidence type, memory, and spent a lot of time on a problem I didn’t expect: the agent’s reports getting worse every time it rewrote them.
Everything below ran on gpt-oss:120b. I’ve moved
off 20B for day-to-day testing because accuracy is
what I care about, not speed.
Where it started today#
Event log scores on 120B, before any of this:
| Case | Average / fixed |
|---|---|
| APT29 day 1 | 96 / 95 |
| Lateral movement | 98 / 96 |
| Credential dumping | 99 / 99 |
| Multi-stage APT | 98 / 98 |
Good, but not clean. The APT29 runs had a few reports that were just broken, and a couple of claims kept sneaking in that nothing supported.
Reports that weren’t finished#
One APT29 run scored 14/24. Reading the log, the model had written a full report, got bounced by the validator for a couple of bad ATT&CK IDs, looked them up, then “rewrote” the report as just the Scope table. That’s it. Code accepted it because the only check on a final report was “is it longer than 200 characters”.
The same thing happened on my 3080 earlier in the day, except that time 20B got stuck counting event IDs (4808, 4809, 4810…) until it ran out of context.
Fix: code now checks the report has all six sections. If any are missing it asks once for the complete report, and if that fails too it falls back to the last complete draft and marks it.
APT29 went 94 → 97 straight away.
Every rewrite is a new chance to make stuff up#
This was the big one.
The validator bounces a report once if it cites an unverified ATT&CK ID or names something like Mimikatz or “C2” that no tool output shows. That worked. But after the bounce, a separate check (coverage) asks the model to add the detections it skipped and rewrite the report again. Nothing checked that rewrite.
That’s exactly where “C2” was coming back. 120B would drop it when told, then put it straight back in on the next full rewrite.
I tried checking every rewrite. That fixed C2 on APT29 (4/10 → 0/10), but on lateral movement the logs showed the real pattern:
- A report gets bounced for “Mimikatz”.
- The rewrite drops Mimikatz but adds “exfiltration”.
- Or it drops “coordinated campaign” and adds “command-and-control”.
Whatever you bounce, the next full rewrite is a fresh roll of the dice. More bounces just move the problem to whichever draft comes last.
Fix: stop asking for full rewrites at the end. After the report is done, code finds the specific lines that still make an unsupported claim and sends only those lines to the model in a separate, short call: “rewrite these to remove this claim, change nothing else”. Code then checks each returned line, and only swaps it in if the problem is actually gone and nothing new was added. The rest of the report can’t change.
A grading bug that looked like a model bug#
My new memory case failed the host-name check on 5 out of 5 runs. Every report clearly named both hosts.
Except it didn’t. gpt-oss writes ‑ (U+2011, a
non-breaking hyphen) instead of a normal -. So
DESKTOP‑SDN1RPT in the report doesn’t match
DESKTOP-SDN1RPT. That’s not just a grading
problem: if an analyst copies that hostname into a
SIEM search, they get nothing back.
Reports now get normalised to plain hyphens and
spaces, and that happens before any check runs,
not just before saving. I found that second part
the hard way: the claim check was missing
command‑and‑control for the same reason.
Memory analysis#
New tool: memory_triage(). Code runs the Volatility
3 plugins, caches the results, and hands the model a
short summary instead of raw output. netscan alone
was 5.4 MB on one image, which is never going in a
prompt.
The test case: The Stolen Szechuan Sauce#
For memory I needed a case where I already know the answers, otherwise there’s no way to grade the agent. I went with Case 001 from DFIR Madness, The Stolen Szechuan Sauce.
It’s a free, public practice case built around a small Windows domain that gets compromised. It’s well known in the DFIR community, plenty of people have written it up, and there’s a published answer key, so I can check the agent against what actually happened rather than what I think happened.
Why this one:
- It’s one intrusion across several evidence types. Memory, disk images, event logs, autoruns and a PCAP, all from the same incident. That means I can add disk and network to the agent later and test them against the same story, instead of finding a new case each time.
- Two hosts with different roles. A domain controller (Windows Server 2012 R2) and a Windows 10 desktop. The agent has to tell them apart and work out what links them.
- It has traps. The responder’s own tools are in the memory images (FTK Imager, PowerShell) right before capture. An agent that calls everything suspicious gets that wrong.
- It’s small. Two 2 GB images. Volatility gets through every plugin I use in a few minutes, so I can run it 10 times per change.
I only used the two memory images for this round. Disk comes next.
What code decides, so the model doesn’t have to:
| Section | What it does |
|---|---|
| Processes after boot | Grouped by logon session. Routine Windows processes collapsed into a count unless their parent is gone or they’re running from somewhere odd |
| Network | Connections to public addresses. If Volatility can’t tell which process owns one, it says so instead of guessing |
| Injected code | malfind hits classified by their first bytes: shellcode, a full PE in private memory, or a reflective loader stub (MZARUH). Anything else is listed as weak, because .NET does this constantly |
| Across hosts | Any outside address or unusual process seen on more than one host |
| Responder | Acquisition tools (FTK Imager etc.) labelled as the responder, plus anything they started in the few minutes before the capture |
On this case that surfaces:
coreupdater.exeon both hosts, started hours after boot, parent gone.- Both hosts talking to the same outside IP on 443. Volatility could name the process on the DC, but not on the Windows 10 box.
- The Print Spooler on the DC restarted mid-incident with shellcode, two PE images and a reflective loader in its memory.
- The responder’s PowerShell and FTK Imager at the end, labelled as the responder.
The responder bit mattered. Before I added it, one report flagged the responder’s own PowerShell as attacker activity.
netscan took over 15 minutes on the Windows 10
image. netstat does the same job in seconds by
walking the live tables instead of scanning all of
memory. The tool tries netstat first, and every
plugin has a time limit now so one slow plugin can’t
stall an investigation.First memory score was 88. Most of that was the hyphen bug. After the fixes: 98 / 99 over 10 runs.
Where it’s at#
gpt-oss:120b, average / fixed checks:
| Case | Score |
|---|---|
| APT29 day 1 | 99 / 99 |
| Lateral movement | 100 / 100 |
| Credential dumping | 100 / 100 |
| Multi-stage APT | 100 / 99 |
| Szechuan Sauce (memory) | 98 / 99 |
Lateral movement was the stubborn one. 120B really wants to call 47 unrelated sample files “a coordinated campaign”, and bouncing it only cut that from 5/10 to 4/10. The line-level fix took it to 100 / 100, with no unsupported claims across all ten runs.
The memory misses are mostly mine: the checker didn’t know about the ATT&CK IDs in the memory part of the guide, and the hyphen issue above. Both fixed for the next round.
Lessons from this round#
- A short report is not a finished report. Check for structure, not length.
- Every rewrite needs checking, not just the first. Better still, don’t ask for full rewrites when you only need to fix a line.
- Read the failing check before fixing the agent. 5/5 failures on host names was a Unicode hyphen.
- More parallel runs can be slower. I ran 25 at once and the GPU’s cache overflowed. Every call had to reprocess its whole prompt. About 10 at once was the sweet spot.
- Measure before buying hardware. I logged tokens per run (about 165k in, 11k out for APT29) before deciding what to run this on locally. Output speed matters far more than I expected.
Next#
- Disk images from the same case, so the agent can confirm what memory flagged (Prefetch, Amcache, registry, the binary itself) instead of just flagging it.
- Velociraptor collections as an evidence type.
- Moving off rented GPUs onto local hardware, so it can be used on real evidence.
