Building a Local AI DFIR Triage Agent

A local AI analyst agent. A local LLM (the brain) drives real forensic tools (the hands: Chainsaw, Sigma, ATT&CK lookup, authentication analysis) to triage Windows evidence and write a cited report.

It’s built so sensitive evidence never leaves local hardware. Rented cloud GPUs are only ever used with public sample data.

Why I built this

#

A lot of SaaS providers charge a boatload for DFIR agents, and most of them seem kind of bad. Talking to people across the DFIR industry, experience levels vary a lot. An open-source baseline for DFIR work, across a range of intelligence and threat types, sounds like something people would actually use, including me.

I’d like it to make people better at doing their own work, and to save professionals time without them losing capability. The agent shouldn’t know more than the analyst using it. I’m big on an AI agent being a junior or an assistant, not a replacement.

Building agents isn’t my forte, so I’ve used AI heavily to write the code. The DFIR side is mine: the test cases, what counts as a good report, and what the agent gets wrong.

Architecture

#

Brain (gpt-oss)
  Ollama (local GPU) or vLLM (A100)
        ▲              │
  tool requests      report
        │              ▼
SIFT VM (hands)
  triage.py + llm.py
  chainsaw_hunt, get_detections,
  attack_lookup, auth_summary,
  decode_powershell
        │
  evidence share (read-only)

• The brain only ever sees filtered tool output, never raw evidence.
• Swapping brains is a settings change, not a code change.

Tools the model can call

#

Two more summaries are attached automatically after every hunt: remote logons (type 3/10 by host, source and account, with source IPs named from Sysmon) and service installs (7045/4697 with installer, run-as account and binary path).

Design rule: enforce in code, not in the prompt

#

Small models drop prompt rules. Code doesn’t. Never make the model type what code already knows: paths, rule names, counts, timelines.

Guardrails that live in code:

• ATT&CK validator. Every cited ID must be active and backed by a lookup, the guide or a Sigma tag. Bounced once, then stripped to (ID unverified).
• Coverage check. If high-severity rules weren’t inspected, code pulls examples itself and hands them to the model to rewrite the report.
• No-tools recovery. If the model ends its first turn without calling a tool, code runs the hunt and carries on.
• Claim checks. Named claims (exfiltration, Mimikatz, C2) with no support in any tool output are flagged in an auto-generated section.
• ATT&CK names written by code, not the model.

Testing

#

Four test cases built from public datasets, each run 10 times per change and graded by a checker:

Current scores, gpt-oss:20b on vLLM (average / fixed checks):

The first version of the agent scored 48% on lateral movement.

Lessons so far

#

• Tools and guardrails beat model size. Most gains came from better tool output and code checks, not from moving 20B to 120B.
• Format beats data. The APT29 pivot IP was in the tool output every run, yet 7/10 reports dropped it. One line in the report format fixed it: 10/10.
• Test data can leak the answer. A folder named after the threat actor reached the model, and 7/10 reports named the actor with no evidence. Renaming the folder fixed it.
• Repeating the right answer doesn’t make the model use it. Putting the correct PsExec technique IDs in tool output 3–4 times didn’t change what it cited, and it crowded out other findings. One line in the report format did.
• One flat list can invent a campaign. A single host table made the model treat unrelated sample hosts as one coordinated attack. Grouping by domain fixed it.
• A low average can be one broken run. One score of 83 turned out to be a single run where the model called no tools and wrote a refusal. The other nine were fine. Always check the outliers before changing anything.
• Question the checker as hard as the agent. 5 runs swing ±5 points; 10 runs are the minimum for a keep/revert call.