# Audit Local LLM Agents Like Runtimes

Canonical URL: https://huecki.com/en/blog/local-llm-agent-runtime-audit/
Markdown URL: https://huecki.com/en/blog/local-llm-agent-runtime-audit.md
Language: English
Published: 2026-06-24
Updated: 2026-06-24
Author: Dominic Hückmann
Topic: AI Agent Security
- Agent topics: Agent Security, Context Engineering, Agent Evals
- Tags: AI Security, Local LLMs, Agents, Developer Workflow, Runtime Security
Content status: field-note

## Summary

Local LLM agents can touch shells, files, browsers, credentials, memory, and messaging tools. Treat their runtime layer as source code worth auditing, then turn static findings into a manual review queue instead of automatic verdicts.

## Description

CLAWAUDIT is a useful reminder that local agents are privileged runtimes, not chat widgets. Audit prompt assembly, parsers, tool dispatch, skill loading, memory writes, network clients, and permission gates.

## Body

## Short answer

Local LLM agents are not just chatbots with a few helpful tools.

They are privileged runtimes.

They sit between natural-language intent and host-level actions: shell commands, files, browser sessions, credentials, memory, network clients, and messages. That makes the runtime layer a real security boundary, not a prompt detail.

The useful idea in [*Local LLM Agents as Vulnerable Runtimes*](https://arxiv.org/abs/2606.21071) is to audit that layer as source code. Not "can the model be tricked?" but "where does this runtime assemble instructions, parse model output, grant tools, load skills, write memory, call the network, or check permissions?"

## The mistake

Most agent security talk still starts at prompt injection, malicious webpages, and poisoned skills. Good questions. Incomplete questions.

The runtime decides what model text becomes an action: how tool calls are parsed, which names are valid, whether skills can reshape priority, whether memory writes are trusted, and whether permission checks happen before side effects.

If that code is sloppy, a better prompt will not save you.

## A workflow to steal

Take one local agent runtime and make a four-column audit table:

```txt
Surface | Example risk | Evidence to inspect | Safe control
```

Use these rows:

1. **Prompt builder:** untrusted text entering high-priority instruction slots.
2. **Output parser:** fallback parsing, JSON repair, hidden command extraction, and partial parse success.
3. **Tool dispatcher:** stringly typed routing, broad allowlists, and tools that can call other tools.
4. **Skill loader:** remote install paths, weak provenance, and skill text that implies new authority.
5. **Memory writer:** unverified promotion, cross-user bleed, and attacker-controlled lessons.
6. **Network client:** implicit credential use, SSRF-shaped fetches, and weak destination checks.
7. **Permission gate:** checks after side effects, approval stored as plain text, and bypasses for "internal" calls.

For each finding, write the exploit path in boring language:

```txt
Untrusted source -> runtime transform -> privileged action -> missing gate
```

If you cannot write that path, keep the item as a hardening note, not a vulnerability claim.

## Static checks help, but they are noisy

CLAWAUDIT reports a clear signal: agent-specific static rules can catch things general rules miss. The abstract says held-out recall rose from 21.7% to 66.8% for Semgrep and from 13.8% to 75.1% for CodeQL.

That is useful.

It is also not the same as "run scanner, ship fix." Recall-oriented rules over-report by design. A preliminary live-code audit in the paper still required manual triage and semantic filtering before production use.

## Tiny audit prompt

Use this after a scan or source read:

```txt
Review this local-agent runtime finding.

Return:
1. Surface: prompt builder / parser / dispatcher / skill loader / memory writer / network client / permission gate.
2. Untrusted input source.
3. Privileged action it may influence.
4. Missing or late control.
5. Minimal exploit path in one line.
6. False-positive reason, if likely.
7. Best fix location in code.
8. Regression test that would catch it next time.
```

## Do and don't

## Where this fails

This is not a full security program. It will miss bugs that require live state, timing, auth context, or multi-step behavior. It also gets noisy if every tool-shaped string becomes a finding.

So keep the bar practical: a strong finding needs a source, a transform, a reachable privileged action, and a missing control. Local agents deserve that discipline because they run where the damage is close.

## Sources

- [Local LLM Agents as Vulnerable Runtimes: A Source-Code Audit of the Agent Runtime Layer](https://arxiv.org/abs/2606.21071)
- [arXiv HTML: Local LLM Agents as Vulnerable Runtimes](https://arxiv.org/html/2606.21071)
- Private Huecki radar artifact: `content-research/radar/2026-06-23.md`

## FAQ

### Why audit a local LLM agent runtime?

Because the runtime mediates between model text and host actions: shell commands, file writes, browser sessions, memory updates, stored credentials, and messages.

### Is static scanning enough for agent security?

No. Static rules are useful for recall and review queues, but agent-runtime findings need manual triage, semantic filtering, and runtime controls before production use.

### Where should a small team start?

Start with the seven runtime surfaces: prompt builder, output parser, tool dispatcher, skill loader, memory writer, network client, and permission gate.