DFIR Agent Part 4: A Case It Hadn't Seen

Part 3 ended with me wanting Velociraptor collections as an evidence type. I got those working, and they scored 99 straight away. That told me nothing useful, because those zips came from the same dead-disk images I’d been tuning on for days.

So I built a test I couldn’t tune for. A Windows VM, attacked by me, collected live, with an answer key written before the VM existed.

It scored 94. Then I fixed things until it scored 100, and the 100 is the less interesting number.

The setup

#

The agent reads Velociraptor Collection-*.zip files through dissect’s loader, and copies the event logs out for Chainsaw. On the Szechuan collections that gave 99 / 99, the same as the full E01s, from 90 MB instead of 11 GB. That’s a nice result for the file format. It doesn’t say anything about a live host, because no live host was involved.

The new case is called FIN-WS01. It’s Windows 10 on an isolated bridge in Proxmox, with no internet and no LAN. Everything in it is fake, so the evidence is fine on a rented GPU.

I set three rules before starting:

• The key gets committed before the VM is built. The commit hash is the proof.
• I add hidden steps afterwards. AI wrote most of this tool and the key, so the only steps that can’t have been prepared for are ones it doesn’t know about. I kept those out of the repo and out of every chat until the first score was in.
• No agent, guide or checker changes until that score is recorded. Then fix what it shows, and keep the first score next to the later ones.

The attack

#

All times UTC on 3 November 2026, taken from the evidence afterwards:

The renamed curl is signed by Microsoft, so a no-publisher check won’t flag it. Only its path gives it away. And clearing the log takes the account creation events with it, which is the point of the last step. Then I waited over an hour so SRUM would write its row, and collected from the running host as a responder would.

94, or really 92

#

First run, 10 reports, blind: 94 / 92. With my two hidden steps added to the check, the same reports scored 92 / 88.5.

That’s a decent first result, and I’d rather have this table than a 99. The agent did find the intrusion. rdp_bad is a real gap though: the Security log was gone and no tool read accounts from the SAM hive, so it had no way of seeing it.

“The reports are calling it C2”

#

The exfiltration lead was being lost in rewrites. My notes said why: reports were calling the attacker’s server C2, the validator bounced them, and the lead went with it.

That was wrong. When I read the lines that actually tripped the check, most of them were T1041 Exfiltration Over C2 Channel. That’s the technique’s real name, and my guide tells the model to cite it. No tool output in this case says “C2”, so the validator treated the guide’s own answer as an unsupported claim. It never came up on Szechuan, because a label there happens to say “C2 address”.

A small fix, and I’d written down the wrong diagnosis without reading the evidence. That’s the third post in a row with a bug I blamed on the model.

The OneDriveHelper miss was worse than a missing word. One report gave its encoded command to SecurityHealth, the value on the row above. Code now decodes the payload and prints it under the value’s own name.

99 is not the same as reported

#

With those fixes the score jumped to 99 / 98. Every report contained rdp_bad, so the check passed 10 out of 10.

A batch later I counted how many reported it. Only 2 of 10 had it in Findings. The rest had it as a name in a list of users. An analyst reading those would never know a new account had turned up under four minutes before the log was cleared.

Same again with the firewall rule. The check only looked for the word firewall, and 2 of 10 reports never named the rule. So I made that check stricter.

The fix that worked was marking the row in code:

13:14:23Z  rdp_bad  rid 1004  last logon never
  logons 0  groups Users
  <- created after the first indicator
  <- its creation event went with the Security log

These are search counts, not every line read, and the last one could be missing other wordings. It’s also the one I care about most and it’s only half done.

I can’t fully explain why this worked. In Part 3, labels did nothing for the Timeline rows: 0 of 20. They’re still at about zero now. These marks worked on the first run. The obvious difference is that an account or a file is a thing you can write a finding about, and “opened Windows Security” isn’t, but that’s a guess.

Too many marks

#

Then I added a mark for one account opening a file in another account’s profile. FIN-WS01 improved, and the Szechuan cases got worse. Social Security Number.zip on the memory + disk case went from 1/10 misses to 5/10.

History, LNKs and jump lists all list the same files, so that case had 21 marked rows for 7 paths. That zip name isn’t marked at all. It only exists as a middle step in a USN rename chain. Marking each file once took it back to 2/10.

It’s at 4/10 in the latest run though, so I don’t think that’s fully settled. It’s also the same miss I flagged at the end of Part 3.

Removing a check found a bug

#

Nearly every run was bouncing at least once, and each bounce rewrites the whole report. A lot of those were only for a missing ATT&CK ID, which code can add at the end anyway. So I dropped that bounce reason:

Scores held, and a batch finished noticeably quicker. I checked what code was now writing, because code writing an ID is code making a claim. Code had added T1110.003 Password Spraying to 6 of 10 reports. One read “95 failed attempts for Administrator… indicates a prior password-spraying”.

One account hit 95 times is guessing, not spraying. The word came from the model, and my code stamped an ID on it. The guide now spells out the difference, and code won’t add the ID next to T1110.001. That took it to 1 of 10 on that case, and the credential dumping case, where it really is spraying, kept it in 9 of 10.

A false T1047 had a similar story. WmiPrvSE.exe in a firewall event made code tag it as WMI execution. I made that rule with New-NetFirewallRule, which fits WmiPrvSE.exe being the process, so that’s the tool working as normal, not WMI used as a technique. It took me three goes to catch every way a report can word that, because I’d tested the first fix on the wrong case. It’s at 2 of 10 now.

Where it’s at

#

gpt-oss:120b, average / fixed checks, 10 runs each (how these are scored is in Part 1). Last full batch, commit c56103c, 0 crashed, 0 degraded:

After that I ran the five cases the last fixes touch most, at a62b674: FIN-WS01 100 / 100, credential dumping 99 / 99, disk 99 / 98, memory + disk 98 / 98, Velociraptor 98 / 96. The other four haven’t been run on that commit yet.

The FIN-WS01 100 isn’t blind. I fixed the agent using its own misses, so it now shows the fixes work and nothing else. The number that measures new evidence is still 94 / 92.

Lessons from this round

#

• Build a test you can’t tune for. Weeks of 99s didn’t tell me what a first blind run did in an afternoon.
• A keyword can pass with the finding missing. Every report said rdp_bad and 2 of 10 reported it.
• Read what the check quoted before explaining it. My explanation for the C2 bounces was neat and wrong.
• When you remove a check, look at what it was hiding. The bounce had been covering a false ID.
• Test a fix on the case that showed the bug. The T1047 fix passed on one case’s wording and missed another’s.
• Marks cost attention. Marking more rows isn’t free, and the facts you didn’t mark pay for it.

Next

#

• A second blind test on a public case that I haven’t touched, of a very different kind. I write the checks from its published answers, and the AI doesn’t see them until the first score is in.
• The rest of the creation-events fix, and that zip name.
• The Mac Studio, still, so this can run on real evidence.