A local agent in front of a cloud agent: gpt-oss-120b on llama.cpp

What stays local and what goes to the cloud?

The split is by task shape. Classification stays local. Anything that writes prose a human will send goes to the cloud agent.

That last row is the one worth stating explicitly. A large share of the decisions in this system are not model decisions at all. Putting a model behind a question that a substring match answers is the most common way to spend money on nothing.

How is llama-server configured?

One systemd unit, no container, no supervisor of our own. The model is the 3-part MXFP4 GGUF, about 63 GB on disk, on its own NVMe.

[Unit]
Description=llama.cpp server (gpt-oss-120b, Vulkan)
After=network-online.target
Wants=network-online.target
StartLimitIntervalSec=120
StartLimitBurst=3

[Service]
Type=simple
User=coop
Group=coop
SupplementaryGroups=render video
ExecStart=/home/coop/llama.cpp/build/bin/llama-server \
  -m /data/models/gpt-oss-120b/gpt-oss-120b-mxfp4-00001-of-00003.gguf \
  -ngl 999 -c 131072 --jinja -fa on -ub 512 -b 2048 \
  --host 0.0.0.0 --port 8080
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

When does the local agent hand off to the cloud agent?

A scheduled job wakes frequently. The expensive part is handing two mailboxes to the cloud agent. Three gates decide whether that happens, ordered cheapest first, and all three fail open: when in doubt, hand off, because the cost of skipping real mail is much higher than the cost of a wasted run.

The call itself is deliberately boring. Temperature zero, because this is a classifier and run-to-run variation on the same message is a defect rather than a feature.

POST http://127.0.0.1:8080/v1/chat/completions

{
  "model": "gpt-oss-120b-mxfp4-00001-of-00003.gguf",
  "temperature": 0,
  "max_tokens": 200,
  "chat_template_kwargs": { "reasoning_effort": "low" }
}
# 60s timeout; any error or unparseable body counts as "genuine"

Two details that are not obvious from the config. Verdicts are cached per message id, because a skip does not advance the last-run watermark, so without a cache the same junk gets re-judged every cycle all day. And there is a heartbeat backstop: if the last real run is older than 24 hours, or predates this morning, the gates are bypassed regardless of what they think. A filter that can bury a message forever is a worse failure than a filter that wastes a run.

What is the second local job?

An inbox janitor, running every 15 minutes on new unread mail. Two passes: move obvious noise out of the inbox, and rescue real mail that the provider filter put in Junk. Deletes are soft, into Deleted Items, so a wrong call is one drag-and-drop from undone.

Same endpoint, same model, different settings:

{
  "temperature": 0,
  "max_tokens": 400,
  "chat_template_kwargs": { "reasoning_effort": "medium" }
}

reasoning_effort is a per-job setting, not a global one. The gate runs at low because it fires constantly and a wrong answer costs one wasted run. The janitor runs at mediumbecause a wrong answer moves someone's mail. We measured the tiers on a 22-case battery: 101, 103, and 107 out of 110 for low, medium, and high, at roughly 1x, 1.6x, and 2.9x latency. Higher tiers buy little and cost real time, so nothing here runs at high.

Why do two jobs on the same model have opposite fail directions?

Because the cost of being wrong is inverted, and that belongs in configuration rather than in a prompt.

How do you stop an email body from steering the classifier?

Judge one message per call. A mail body is untrusted input written by whoever sent it, and batching ten of them into a single prompt means any one of those bodies can talk about the other nine.

With per-message isolation, a body carrying injected instructions can only misclassify itself. It cannot suppress the verdict on someone else's real mail. And because the gate fails open, self-misclassification lands on the safe side: unsure becomes genuine, which hands off to the cloud agent.

This costs more calls than batching. Against a local agent there is no per-call bill, so that trade is easy.

Why does the local agent not draft anything?

Before assigning it work we ran a scored battery against it: 22 cases, 5 trials each, 110 runs. The tool mechanics were clean. Zero schema violations and zero malformed tool calls across 110 trials. Single calls 40/40, compound commands 25/25 under a real execute-then-continue loop, four-hop chains 21/25.

A separate 180-trial run tested whether it invents facts. With tools available it fetched the answer 57 times out of 60. With tools stripped it fabricated nothing in 60 trials, reaching for tools that did not exist instead. Asked about entities that do not exist, it reported not-found 60 times out of 60.

The failures it did have were arithmetic. Asked to multiply 410.75 by 0.91 in its head it produced a different wrong product on each trial, while multiplying 2890 by 0.78 correctly every time. It cannot tell a hard mantissa from an easy one, and states both with the same confidence.

The relevant result for this system: fabrication showed up in generative tasks, not in retrieval-shaped question answering. An earlier drafting test had it invent invoice details and a signature. So it screens and it gates, and it does not write anything anyone will read.

Why a hand-rolled harness?

We started on existing agent harnesses and moved off them. The reason was fit rather than quality. This box needs per-job control: effort low for the gate and medium for the janitor, opposite fail directions for two jobs sharing one model, and a loop we can reshape when a benchmark result says to.

A small harness we own turned out to be easier to tweak and tune than configuring someone else's. It is about 2,400 lines of Python across five files plus 27 systemd units. There is no daemon and no scheduler of our own, because systemd is already both.

One measured result shaped this more than anything else. The model is strictly sequential: zero parallel tool-call events in 110 trials. A harness that expects parallel calls makes it look like it drops the second half of compound instructions. Under a plain execute, feed the result back, continue loop, it handles them 25 out of 25. The loop decides that, not the model.

How do you know the local model is actually up?

Not by asking it. The /health endpoint returns 200 on a server that is reachable and wedged, which is exactly the state a GPU watchdog reset leaves behind. And Restart=on-failure only catches a process that exited.

A separate timer issues a real one-token completion every three minutes and restarts the unit if that hangs. Every restart is appended to a log and counted in a daily digest, so self-repair cannot quietly mask a crash loop. Silent recovery is still a fault worth seeing.

What does it cost to run?

Read from the box on 2026-08-10: 17 timers active, no failed units. In the preceding 24 hours the gate ladder handed 3 cycles to the cloud agent and stopped 84, which is roughly 95% of cycles never leaving the box. That figure is the whole ladder, not the local model alone: Gates 0 and 0.5 run first on every cycle and use no model at all, so the local model only judges what they let through. A cloud run costs about $0.58.

The electricity is a rounding error next to the inference bill it displaces, and the box was bought for other work anyway. The point of the ladder is not that local inference is free. It is that the cheapest gate that can answer a question should be the one that answers it, and for most questions here that gate is an integer comparison.

Sources and further reading

• llama.cpp - the inference server, built here with GGML_VULKAN=ON.
• llama.cpp server documentation - the flag reference for -ub, -b, and the OpenAI-compatible endpoints.
• gpt-oss-120b model card - including the reasoning_effort levels used above.
• systemd.timer - the scheduler this harness does not reimplement.
• systemd.service - StartLimitBurst, Restart, and SupplementaryGroups.