Setup guide · public edition

Home AI Box Setup — MINISFORUM AI X1 Pro-370 (Ryzen AI 9 HX 370, 64GB)

MINISFORUM AI X1 Pro-370 on Windows: measured tok/s, Tailscale, a Cloudflare tunnel, and what the box will not do.

Printed October 22, 2026 from rabinai.com — check the site for the current version.

Public edition. Real hostnames and addresses have been replaced with placeholders (llm.example.com, 100.x.y.z). Everything else — the commands, the measurements, the reasoning — is unchanged from the working copy this was built from. 100.64.0.0/10 is left intact on purpose: that is Tailscale's public CGNAT range, identical for every tailnet, and the firewall rules mean nothing without it.

Goal: a part-time local inference box — asleep most of the day, awake when something needs it. Windows 11, no Linux.

The machine this was built on: MINISFORUM AI X1 Pro-370 — Ryzen AI 9 HX 370 (12C/24T), 64GB DDR5, 1TB SSD, Radeon 890M iGPU.

Buy the 64GB configuration, not a smaller one. Section 1 hands roughly half of that to the integrated GPU as VRAM before Windows boots, and that allocation is what lets a 30B-class model sit resident at all — ~17GB for Qwen3.6-35B-A3B at a 32k context, or 12.8GB for gpt-oss-20b with room left over for an image model beside it. On a 32GB machine the same split leaves too little for the OS, and you would be choosing between the language model and everything else. Nothing later in this guide can recover from that decision.

There is no integrated-GPU-specific magic here otherwise: the reason this box works is memory bandwidth and a large unified pool, so any machine with a comparable APU and 64GB would follow the same path.

What it serves is rabinforest.com. The assistant on that site answers from this box whenever the box is awake, and falls back to a cloud model when it is not — so the box going to sleep degrades the answer rather than breaking the site. One language model and one image model share the single integrated GPU, and the language model can be swapped for the other one while the site stays up; section 4 covers both, and the swap is a live control at rabinai.com.

Section 8 covers wiring the box into a second, batch workload. That adapter is written up here but is not in production, and section 8 says so where it matters — worth knowing before you follow it.

Build order — the linear path

The guide has grown optional branches; this is the spine. The first build is five sections, in order, skipping everything else:

Step	Section	You have, when done
1	section 1 BIOS	VRAM allocated, visible in Task Manager
2	section 2 Drivers	GPU actually usable
3	section 3 Toolchain	git/node/python/Claude Code installed
4	section 4 LM Studio	a model answering on `localhost:1234`
5	section 6 Tailscale	that endpoint reachable from your dev machine

Do each section's Done when check before moving on — that's the whole method. Stop after step 5: that's the finished first version, a working private LLM server, one afternoon of which only ~90 minutes is hands-on (the rest is downloads).

Second session — production: section 7 (Cloudflare Tunnel + Access), section 8 (the adapter in bfoster-services), then the section 11 checklist including the local-vs-Gemini quality spot-check.

Later, à la carte, any order: section 5 image models, section 9 IDE assistant, section 10 TTS/voice. Nothing in the spine depends on any of them.

Once it is serving real traffic: section 12 is day two — keeping it up, what actually breaks, and why. Nothing in it is hypothetical; it is the part this guide did not have for its first month.

1. BIOS: set the VRAM allocation first

Before installing anything, reboot into BIOS (Del or F7 at startup):

Check MINISFORUM's support page for the X1 Pro-370 and update to the latest BIOS if one's available — memory-controller fixes are common.
Find the GPU/VRAM allocation setting (usually under Advanced → AMD CBS) and set a fixed size. For your 64GB unit running a ~20GB text model plus an ~8GB image model, 32-48GB allocated to GPU leaves plenty of headroom. Don't max it out — Windows and background services need system RAM too.

Done when: after booting into Windows, Task Manager → Performance → GPU shows your chosen amount as Dedicated GPU memory.

2. Install AMD GPU drivers

Download the latest AMD Adrenalin driver for the Ryzen AI 9 HX 370 (Radeon 890M) from amd.com/en/support (auto-detect tool or pick Ryzen AI 9 HX 370). This is what enables Vulkan/ROCm acceleration — without it, models silently fall back to slow CPU-only inference and you won't get an error telling you why.

Done when: Device Manager → Display adapters shows AMD Radeon 890M (not "Microsoft Basic Display Adapter"), and the AMD Adrenalin app launches.

3. Install the support toolchain

Windows 11 ships with winget, so this is a few one-liners from an elevated PowerShell (right-click Start → Terminal (Admin)):

winget install -e Git.Git
winget install -e CoreyButler.NVMforWindows
winget install -e Python.Python.3.12

What each is for:

Git — cloning ComfyUI (Section 5) and pulling any repo you need on the box.
nvm-windows — the Windows counterpart to nvm (the original nvm is a bash script and doesn't run on Windows; Corey Butler's nvm-windows is the standard equivalent, same commands). Node itself comes through it, never installed directly, so you can switch versions without reinstalling.
Python 3.12 — ComfyUI's venv in Section 5.

Then close and reopen the terminal (winget installs don't update the current session's PATH) and install Node LTS:

nvm install lts
nvm list          # note the version it installed
nvm use 22.11.0   # or whatever `nvm list` showed; needs the admin terminal

nvm use needs an elevated terminal — it works by swapping a symlink in a protected location.

Note: the production adapter code (Section 8) runs in bfoster-services, not on this box — Node here is for testing the endpoint and running utility scripts locally.

Claude Code on the box

Install Claude Code so it can help drive the rest of this setup from the box itself — checking drivers, running the verification curls, debugging whatever goes sideways — instead of you relaying error messages from another machine. Use the native installer rather than npm install -g: a global npm package lives inside the active nvm Node version and vanishes when you switch versions.

irm https://claude.ai/install.ps1 | iex

Then, so it has this guide as context, put a copy of this document in a folder and start it there:

mkdir home-ai-box; cd home-ai-box
# drop this guide in as README.md, then:
claude

If you keep your own notes in a private repo, cloning it instead triggers a GitHub sign-in — Git for Windows ships Git Credential Manager, which opens a browser window for it; that's expected, not an error.

Log in with your claude.ai account when prompted. From there you can ask it to walk you through the remaining sections and verify each Done-when check as you go.

If Node got installed directly first (it did here)

This box took the wrong path before the right one: Node was installed straight to C:\Program Files\nodejs, then reinstalled under nvm-windows later. Two things to know if you're repeating that repair:

Installing nvm-windows moves Node to C:\nvm4w\nodejs (a symlink to the active version under %LOCALAPPDATA%\nvm). The old directory is left empty.
Every already-open shell keeps the stale PATH. node and npx vanish mid-session and work fine in a new terminal, which looks exactly like a broken install and isn't. Open a new terminal before diagnosing anything.

Done when: git --version, node --version, py --version, and claude --version all print versions in a fresh (non-admin) terminal.

As built here: git 2.55.0, Node v24.19.0 (nvm 1.2.2), Python 3.12.10, Claude Code 2.1.241. Note py is not present on this box — Python answers as python from its install directory, which is all ComfyUI's venv needs.

4. Install LM Studio and pull your models

Download LM Studio for Windows from lmstudio.ai — GUI-based, no command line needed.

Two models are worth having on this box: qwen3.6-35b-a3b, the default (since 2026-10-22), and openai/gpt-oss-20b, the other seat of the public model switch (the default from 2026-09-12 until then). Both are mixture-of-experts models with ~3B parameters active per token, which is the shape the bandwidth math below says this box can run fluidly.

Qwen3.6 was brought in (as the second seat, 2026-10-09) on one measured property: how prefill degrades with prompt length. The production prompt is ~11.5k tokens, and a model that primes it slowly is a model visitors meet through the cloud fallback. Measured with one script on the idle box, cold, 2026-10-09:

	3.2k tokens	5.5k	11.5k	decode
`openai/gpt-oss-20b`	6.3s	11.5s	35s	23.5 tok/s
`qwen3.6-35b-a3b`	7.7s	13.7s	34s	19.8 tok/s
Qwen3-30B-A3B-2507 (the seat it replaced)	10.6s	25.6s	107s	26 tok/s
GLM-4.7-Flash (tried, rejected)	67s	190s	~14 min	26 tok/s

Quant was not the lever — a Q4_0 of Qwen3 was 4% faster than its Q4_K_M. What decided it is attention design: gpt-oss alternates full attention with 128-token sliding-window layers, and 30 of Qwen3.6's 40 layers are linear (Gated DeltaNet), so neither pays the full quadratic cost as the prompt grows. Qwen3's full attention on every layer lost two thirds of its prefill rate between 3k and 11.5k tokens; GLM's MLA attention did worse. Filter candidates on long-context attention design, not on active parameter count — three models with the same ~3B active differed by 20x on the number that matters.

Both seats pass the same 15-question suite against the production assistant (15/15 each, every answer from the box). The remaining difference is decode: Qwen3.6 writes at ~20 tok/s to gpt-oss's ~23.5, so a long answer streams a little slower. It also needs less memory than the 30B it replaced (~16.7GB at 32k context, parallel 3) because the linear layers carry almost no KV cache.

Qwen3.6 became the default on 2026-10-22, after thirteen days as the second seat: the suite could not separate the two, but the answers read better to the person who knows the material, and with the prime-speed objection gone there was nothing operational left to hold gpt-oss in the seat. The one thing it costs is decode, and the server's length budget already accounts for that.

Install gpt-oss first — it is the easier download, the GUI handles it end to end — then Qwen3.6, which needs the import below. Whether you keep both loaded is a separate question with a measured answer: see 4a.

The trap with Qwen3.6: it thinks by default, and only one switch turns that off under LM Studio. reasoning_effort: "none" in the request body works (0 reasoning tokens, answer starts at once). The two switches the model card documents do not: chat_template_kwargs: { enable_thinking: false } is ignored, and /no_think in the prompt is ignored — both measured 2026-10-09, each leaving ~1,300 characters of reasoning in reasoning_content and no answer inside a 300-token budget. Match the control to the model family, every time; the wrong one is silently inert.

The other seat, and the easier install: openai/gpt-oss-20b, GGUF, MXFP4, ~12.8GB. OpenAI's open-weights release — 21B total, ~3.6B active per token. Open LM Studio → Discover tab → search for it → download. MXFP4 is not a quant you pick in the GUI; it is how the weights ship, so there is no Q4/Q8 choice to make here.

Expect any large download to be interrupted at least once — the source throttles, and an 18GB pull that starts at 9MB/s can decay to ~1MB/s and time out partway. This is safe. The partial file (downloading_*.gguf.part) survives, and starting the download again resumes from it rather than restarting — from the Downloads panel in the GUI, or by re-running the same lms get command. Just repeat until it completes.
The default: Qwen3.6-35B-A3B, from unsloth/Qwen3.6-35B-A3B-GGUF, file Qwen3.6-35B-A3B-UD-IQ4_XS.gguf, 17.7GB. There is no Instruct/Thinking split for this generation — thinking is a request-time switch, see the trap above. lms get stalled on this file twice; what worked was pulling the single GGUF with curl into a holding directory, checking its sha256 against the Hugging Face LFS oid, then importing it (commands below). LM Studio names an imported file from the repo and filename — the id came out as qwen3.6-35b-a3b, no org prefix, and that is the string everything downstream must use.

Check the runtime before you download: the architecture is qwen35moe and needs a llama.cpp build that knows it (the Vulkan runtime 2.54.0 did, and had a Gated DeltaNet kernel). An unknown architecture fails at load, after the 25-minute download.
Load the model in the Chat tab. Check the GPU offload slider is set to max (all layers on GPU). Run a test prompt and watch Windows Task Manager's GPU graph — if usage stays flat, it's running on CPU instead and something upstream (driver, BIOS setting) needs fixing.
Go to the Local Server tab and start it. Default endpoint: http://localhost:1234/v1 — OpenAI-compatible, which is what lets the Node.js adapter in Section 8 work with almost no rewriting.

Driving all of this from the command line instead

Everything above is also scriptable, which is the point of installing Claude Code on the box in Section 3 — it can run these itself and read the output, instead of you narrating a GUI to it. LM Studio ships a CLI (lms), but it only registers after the app has been run once, so launch LM Studio normally the first time, then:

lms get "openai/gpt-oss-20b" -y --gguf          # MXFP4, ~12.8GB, no quant to choose; re-run to resume
lms load openai/gpt-oss-20b --gpu max -y --context-length 32768 --parallel 3
lms server start --port 1234
lms ps                                          # what's loaded
lms status                                      # server on/off

# the default: lms get stalled on this one, so fetch the file and import it
curl -L -o Qwen3.6-35B-A3B-UD-IQ4_XS.gguf "https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF/resolve/main/Qwen3.6-35B-A3B-UD-IQ4_XS.gguf"
Get-FileHash .\Qwen3.6-35B-A3B-UD-IQ4_XS.gguf -Algorithm SHA256   # compare with the LFS oid on the HF file page
lms import --yes --user-repo unsloth/Qwen3.6-35B-A3B-GGUF .\Qwen3.6-35B-A3B-UD-IQ4_XS.gguf   # still prompts Y on stdin
lms ls                                          # confirm the id it was given: qwen3.6-35b-a3b
lms load qwen3.6-35b-a3b --gpu max -y --context-length 32768 --parallel 3

# swapping rather than stacking: unload first, then load (here, to the other seat)
lms unload --all
lms load openai/gpt-oss-20b --gpu max -y --context-length 32768 --parallel 3

Pass --context-length explicitly on every load. LM Studio does not remember it across a swap, and a model that comes back at the default context silently truncates long prompts — section 12d is the whole story of how that bites.

The model's API identifier is whatever lms ls shows, and the two seats differ: a model pulled through LM Studio keeps its org prefix (openai/gpt-oss-20b, not gpt-oss-20b), while an imported file gets a bare id (qwen3.6-35b-a3b). That exact string is what /v1/models returns and what LOCAL_LLM_MODEL should be set to in Section 8. Getting it wrong is sneakier than it sounds — see the failure note below.

Two operational facts that matter for a box meant to sit unattended:

Closing the LM Studio window kills the server. The GUI and the server share a process tree, so quitting the app takes the endpoint down with it — requests then fail with ECONNREFUSED, and the adapter falls back to Gemini.
lms server start doesn't need the GUI. Run from a cold machine it prints Waking up LM Studio service... and starts the endpoint headlessly. That's the right way to bring the box up for unattended work — but note it starts with no model loaded, so pair it with lms load unless you're relying on JIT (and see the cold-start timeout note in Section 8).

Picking models: the bandwidth math

Token rate on this box is bounded by memory bandwidth, not compute: every generated token reads the model's active weights from RAM, and dual-channel DDR5-5600 gives ~90GB/s theoretical (~60-70GB/s realistic). So: tokens/sec ≈ effective bandwidth ÷ bytes read per token. Estimated rates (bandwidth math, not benchmarks — verify with LM Studio's tok/s readout). The A3B row has since been measured on this box at 32 tok/s (2026-09-11; an earlier reading was 31.5), mid-range of its predicted band, so treat the other rows as trustworthy too:

Model	Reads/token	Est. rate	Character
Qwen2.5-32B (dense, Q4)	~19GB	3-5 t/s	max quality, batch-only pace
Qwen2.5-Coder-14B (dense, Q4)	~9GB	7-10 t/s	coding-tuned, tolerable
8B-class (dense, Q4)	~4.5GB	13-18 t/s	snappy, less capable
Qwen3-30B-A3B (MoE, replaced 2026-10-09)	~2GB	25-40 t/s (measured: 32)	30B-class quality, fluid; slow to prime long prompts
Qwen3.6-35B-A3B (MoE, linear attention)	~1.6GB	20-30 t/s (measured: 20)	primes long prompts as fast as gpt-oss
gpt-oss-20b (MoE)	~2-3GB	25-35 t/s (measured: 25)	OpenAI open-weights, fast to prime

The mixture-of-experts (MoE) rows are the trick: all ~30B parameters sit in VRAM, but each token routes through only ~3B of them — the model reads like a 3B and thinks like a 30B. On a bandwidth-limited box that's roughly 10x the speed of a dense 32B at comparable quality. For calibration: ~10 t/s is comfortable reading pace, 20+ feels fluid.

The other half of "fast," measured separately: prefill. The table above is generation speed — how fast words stream once they start. Before that, the model must process the whole prompt (prefill). An earlier draft said prefill is compute-bound, so a mixture of experts barely helps it. The first half is right and the second is not. Large-batch prefill on this box is compute-bound, and active parameters set that cost the same way they set generation speed — section 12g has the experiment. Short prompts are still cheap. Long ones are not, and the cost grows faster than the length: a ~16k-token prompt took minutes. When testing, record BOTH numbers LM Studio reports — generation tok/s AND prompt eval tok/s — and judge any latency-sensitive use (chat serving, IDE @codebase queries) on the second one. The lever that actually removes the wait is a stable prefix: an identical prompt comes back from cache in a fraction of a second. Batch extraction doesn't care; residents waiting on a chat bubble do.

Two habits that follow:

When browsing Discover, check active parameters, not total size — on this box, that one number is your token rate.
A3B is the default build; the dense 32B exists as its quality benchmark. If A3B holds up in the Section 11 spot-check (vs the 32B and vs Gemini), it keeps the job and every batch run is ~5x faster than it would have been.

Also worth a look as the ecosystem moves: Gemma 3 27B and Mistral Small 24B (strong dense all-rounders), DeepSeek-R1 distills (step-by-step reasoning, slower). New open-weight generations land every couple of months and supersede rather than patch — each roughly matches the previous generation one size class up, so re-run the comparison occasionally.

Qwen3 thought by default — and it cost 5x

Measured on Qwen3-30B-A3B, the model this box ran until 2026-10-09. The lesson holds for its replacement, the lever does not: Qwen3.6 ignores /no_think and enable_thinking and answers only to reasoning_effort: "none" — see the trap at the top of this section.

Qwen3-30B-A3B is a hybrid reasoning model: unless told otherwise it emits a chain of thought before answering, returned in a separate reasoning_content field and billed as reasoning_tokens in the usage block. On a schema-extraction call, measured on this box, that was 360 of 452 completion tokens spent thinking — reasoning was the majority of the work.

Appending /no_think to the system prompt turns it off:

Mode	Time	Reasoning tokens	Valid JSON
default (thinking on)	15.1s	360	yes
`/no_think`	3.05s	0	yes

Same extraction, 5x faster, still valid JSON. Two caveats worth knowing before you reach for it everywhere:

The documented API switch does not work here. Passing chat_template_kwargs: { enable_thinking: false } is silently ignored by LM Studio — in testing it reasoned more, not less. The /no_think prompt token is the only lever that actually works.
It costs some judgment. On the same county-notice sample, thinking produced the title "Gwinnett County Board of Commissioners Regular Meeting"; /no_think produced "NOTICE OF PUBLIC MEETING" — the literal page header. Date, time, and venue were correct either way. So the loss shows up in fields needing inference, not transcription.

That tradeoff is exactly the Section 11 spot-check question, so don't settle it from this table — run extraction both ways on real inputs and decide per field. If quality holds without thinking, the extraction path gets 5x cheaper on top of the 10x the MoE architecture already bought.

Measured on this box (2026-08-24, Qwen3-30B-A3B Q4)

Real numbers over Tailscale, replacing the estimates above. This is the model the box ran until 2026-10-09; the current pair's prefill figures are in the table at the top of this section, and the lessons below — prefill cost, prefix caching, thinking — are what led to measuring prefill against length when choosing its replacement.

Case	Prompt	Prefill	Generation	Time to answer
Casual chat	20 tok	0.5s	31 tok/s	14.7s (thinking)
Grounded RAG	3.5k tok	15.0s	26 tok/s	26.6s
Grounded RAG	5.9k tok	33.4s	26 tok/s	50.1s
Same RAG prompt, repeated	3.5k tok	0.2s	26 tok/s	11.7s

Generation speed matched the estimate (26-31 tok/s). The three things that didn't show up in the estimates:

Prefill runs ~176-234 tok/s, so long prompts are expensive: 15s at 3.5k tokens, 33s at 5.9k. This is the number that decides whether an interactive use is viable, and it confirms live chat serving is not.
Qwen3 is a reasoning model and thinks by default — on a trivial 20-token question it produced ~2,300 characters of internal reasoning before the first visible word, turning a 0.5s prefill into a 14.7s wait. Append /no_think to the user message to disable it (verified: 0 thinking characters, answer begins the moment prefill ends). Note chat_template_kwargs: {enable_thinking: false} was silently ignored — use /no_think.
Prefix caching is dramatic: re-sending an identical prompt dropped prefill from 15.0s to 0.2s. Put the static parts of a prompt (system prompt, stable context) FIRST and the variable part last, so repeated calls reuse the cached prefix.

Tool calling works, but only with thinking on. Qwen3 emits proper OpenAI-style tool_calls through LM Studio. However, asked "what is today's date?" with /no_think, it skipped the tool it had been given and answered "October 25, 2023" — its training cutoff — with complete confidence. With thinking enabled, it called the tool and answered correctly. Reasoning is what makes the model check itself, so any tool-using setup should leave thinking on and pay the ~10-15s. (This is the doctrine's model-agnostic rule in miniature: never rely on the model choosing to call a tool when correctness matters — force it in code.)

Practical readings: batch extraction doesn't care about any of this. Interactive use wants short prompts plus /no_think. Anything resident-facing stays on a cloud model.

4a. Running both, or one at a time

Both models can be loaded at once. This is the question everyone asks and the answer is less obvious than either "yes" or "no", so here is what was actually measured on this box.

An idle model costs memory, not bandwidth. It is not answering, so it is not reading its weights, so it does not compete for the bus that sets generation speed. Measured 2026-09-10: Qwen benchmarked 36.5 tok/s with gpt-oss sitting loaded beside it — no slower than running alone. The box ran that way for hours without incident.

So if you expected two resident models to halve your throughput: they do not. That intuition comes from thinking of a GPU as compute-bound, and this box is bound by memory bandwidth, which only the answering model consumes.

The cost is memory, and it is image generation that pays it.

Qwen3.6-35B-A3B  IQ4_XS   ~17GB (incl. its small KV cache at 32k)
gpt-oss-20b      MXFP4    ~12.8GB
KV cache                  gpt-oss's, ~3GB at 32k context
                          --------
                          ~33GB against a 32GB carve-out

Over the carve-out, it spills into system RAM. UMA handles that and nothing crashes — but SDXL needs ~7GB when it actually renders, and it draws from the same pool. Two resident LLMs plus a render is where this box has hung.

So the rule is about your workload, not about the models:

If you	Keep
generate images on this box	one LLM at a time — swap deliberately
only serve text	both is fine, and saves you the swap wait
are not sure	one — the swap costs seconds, a starved render costs minutes

The non-obvious failure mode, worth internalising now: if image generation starts failing or crawling, check how many LLMs are resident before blaming ComfyUI, the checkpoint, or the GPU. That is the actual cause more often than any of them, and it is one command away:

lms ps                                          # two entries when you expected one

How you end up with two by accident is the subject of the next subsection: JIT loading does not evict. Call a model that is not loaded and LM Studio brings it up alongside whatever was already there. Nothing warns you, and both keep answering correctly — which is exactly why this goes unnoticed until a render slows down.

To get back to one, unload everything and load the one you want. Unloading only the model you do not want works, but --all is the habit that survives being half-remembered at 11pm.

Playing with other models won't break anything

The plan is to load and unload all kinds of models on this box — that's fine, and the extraction path is built to survive it:

API requests name their model in the model field, and LM Studio just-in-time loads whatever's requested (JIT is on by default). So an extraction call still works even if you've been playing with something else all evening — the named model loads on demand as long as it's still downloaded. Verified: calling an unloaded model returned a correct answer in 2.8s, without evicting the model that was already resident.
The failure case is quieter than you'd want. If the name doesn't resolve — model deleted, renamed, typo — LM Studio doesn't error. It answers with whatever model happens to be loaded, and returns 200. So a stale LOCAL_LLM_MODEL doesn't fall back to Gemini; it runs your extraction on some other model entirely, and the output looks fine. The response's model field is the only place the truth shows up — log it (Section 8). The one loud failure is having nothing loaded, which returns a 400.
Storage math for the collection habit: 30B-class models run ~19GB each, 14B ~9GB, on a 1TB drive shared with Windows. When it gets tight, prune from LM Studio's My Models tab rather than the filesystem.

Done when: a test prompt streams a reply while Task Manager's GPU graph visibly spikes, and this returns the loaded model's name:

curl http://localhost:1234/v1/models

If you'd rather verify the GPU claim with a number than by eye, sample the compute engine while a generation runs — this should read 90%+, and near zero if it's secretly on CPU:

(Get-Counter '\GPU Engine(*)\Utilization Percentage').CounterSamples |
  Where-Object { $_.CookedValue -gt 1 } |
  Sort-Object CookedValue -Descending | Select-Object -First 3 InstanceName, CookedValue

5. Image models (optional): generation and vision

Two different capabilities live under "image model" — this section covers both:

Generation (making pictures): diffusion models via ComfyUI — a separate architecture, runs alongside LM Studio.
Vision (understanding pictures): LLMs with eyes — another Discover-tab download served through the same LM Studio endpoint.

5a. Generation with ComfyUI (for the `ai.js` image-gen routes)

ai.js currently calls gpt-image-1.5 / Imagen for image generation. To self-host that piece too, ComfyUI is the standard runner, works on Windows:

git clone https://github.com/comfyanonymous/ComfyUI
cd ComfyUI
python -m venv venv
.\venv\Scripts\activate
pip install -r requirements.txt
python main.py --listen 0.0.0.0 --port 8188

As built, this box runs a ROCm PyTorch stack instead — torch 2.9.1+rocm7.2.1, HIP 7.2, gfx1150 native, rather than the generic pip install above. It is measurably faster than the estimate this section originally carried. Those install steps were never recorded, which makes this the least reproducible part of the whole guide — see build-log.md. Capture them from whoever ran it.

Getting a checkpoint ComfyUI can actually load

The original version of this section said "download an SDXL checkpoint into models\checkpoints\," which is where the first attempt went wrong and cost 18GB. Be specific:

ComfyUI's Load Checkpoint node wants one single-file .safetensors sitting directly in models\checkpoints\.
A git clone of the HuggingFace SDXL repo gives you diffusers format instead — weights split across unet/, vae/, text_encoder/, with fp16, fp32 and ONNX copies of each. ComfyUI cannot read it, and says so only by showing an empty dropdown — no error, no log line.
Fetch the single file directly, with resume enabled, because large pulls on this box do get throttled:

cd ComfyUI\models\checkpoints
curl -L -C - --retry 5 -o sd_xl_base_1.0.safetensors `
  "https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/resolve/main/sd_xl_base_1.0.safetensors"

That's 6.46GB. ComfyUI picks up a new checkpoint live — no restart needed, though you may need the refresh icon in the menu.

Settings and speed

Set Empty Latent Image to 1024×1024. The 512×512 default is an SD1.5-era value and visibly degrades SDXL output. Steps 20-30 and cfg 7-8 are sane starting points for base SDXL only.

Lightning checkpoints use different settings, and the base ones fry them. RealVisXL_V4.0_Lightning.safetensors (installed 2026-08-29, the RabinAI imagery checkpoint) wants 8 steps, CFG 1.5–2.0, euler, sgm_uniform. Measured on this box at 1024×1024: 19.87s with euler, 34.13s with dpmpp_sde — the two are not interchangeable, and only euler meets the imagery page's ~18s claim.

Measured on this box at 1024×1024, 25 steps, euler:

Run	Time
Cold (includes ~7GB checkpoint load)	68.3s
Warm	52.2s (~2s per step)

Better than the 60-90s this section originally estimated, thanks to the ROCm build. Still ~a minute per image, which is why the RabinAI design is async with a one-job-at-a-time guard: two simultaneous visitors put the second one past two minutes.

Running LM Studio's 30B model (17.35 GiB) and SDXL (~7GB) at once is comfortable. Note the two runtimes report VRAM differently — ComfyUI's ROCm layer reports 55GB available where BIOS allocated 48GB, because it counts shared memory too. Neither number is wrong; don't try to reconcile them.

Done when: http://localhost:8188 in a browser shows the ComfyUI graph, and the default workflow renders an image end to end.

5b. Vision model experiment: extracting from flyers

Text-only extraction can't touch county event flyers, scanned notices, or image-based PDFs. A vision model can — and it's the same schema-extraction pattern as events-ai-extract.js, just with an image in the payload.

LM Studio → Discover → Qwen2.5-VL-7B-Instruct, Q4 GGUF (~5GB — small enough to sit alongside everything else).
Warm-up test: load it in the Chat tab, attach any photo, ask for a description.
The real experiment — flyer in, event JSON out, through the same OpenAI-compatible endpoint the adapter already uses:

const b64 = fs.readFileSync('flyer.jpg').toString('base64');
const completion = await c.chat.completions.create({
  model: 'qwen2.5-vl-7b-instruct',
  messages: [{
    role: 'user',
    content: [
      { type: 'text',
        text: `Extract the event as JSON matching: ${JSON.stringify(schema)}` },
      { type: 'image_url',
        image_url: { url: `data:image/jpeg;base64,${b64}` } },
    ],
  }],
  response_format: {
    type: 'json_schema',
    json_schema: { name: 'event', strict: true, schema: eventSchema },
  },
});

(eventSchema here is a real JSON Schema object — { type: 'object', properties: {...}, required: [...] } — not the loose shape hint used in the prompt text. See the note in Section 8 on why json_object doesn't work.)

If the output holds up on a handful of real flyers, wiring it into production is the Section 8 adapter pattern verbatim — try local, fall back on any failure (here the fallback is "skip the flyer," since the Gemini path never handled images in the first place).

Done when: a real event flyer through the snippet above returns JSON with the right date, time, and venue.

6. Networking: don't port-forward — use Tailscale

Since this box isn't always on and you don't want to expose an inference endpoint to the open internet, put it on a private mesh network instead.

Download and install Tailscale for Windows from tailscale.com/download, sign in, and it's live — no router config, no port forwarding.

Do the same on whatever you're calling it from. Tailscale is the right tool for testing from your own dev machine. For the production Heroku leg, use Cloudflare Tunnel instead (Section 7) — Heroku dynos can't easily run a persistent Tailscale client.

One gotcha: LM Studio's server binds to localhost only by default, so other machines — even over Tailscale — get connection refused. In the server settings, enable Serve on Local Network so it listens on all interfaces. (The Cloudflare leg in Section 7 doesn't need this — cloudflared runs on the box itself and reaches localhost directly.)

The bind does not stay put — pass `--bind`, don't rely on the env var

This is the trap that cost the most time here, because it fails quietly. From the CLI:

lms server start --port 1234 --bind 0.0.0.0

works, but does not persist. LM Studio writes networkInterface: "0.0.0.0" into %USERPROFILE%\.lmstudio\.internal\http-server-config.json, and then a plain lms server start ignores that file and reverts the listener to 127.0.0.1. Verified by restarting: config still said 0.0.0.0, the actual socket came back on localhost.

So after any restart the tailnet endpoint is silently closed while the config still claims it's open — and because the Section 8 adapter fails open, every extraction would quietly run on Gemini instead. Nothing errors. You would not notice.

Set the environment variable instead, which survives restarts and reboots:

[Environment]::SetEnvironmentVariable('LMS_SERVER_HOST','0.0.0.0','User')

Then verify with a stop/start and no --bind flag:

lms server stop; lms server start
Get-NetTCPConnection -LocalPort 1234 -State Listen |
  Select-Object LocalAddress, LocalPort     # must show 0.0.0.0

Windows Firewall needed no change here — installing LM Studio created inbound Allow rules for it on Private and Public, and Tailscale's interface registers as Private. (Those permissive rules are what section 6a later replaces with Tailscale-scoped ones.)

The env var is a backstop, not the mechanism — pass --bind at every call site. An environment variable fails silently in three ways here: a Scheduled Task running as SYSTEM never sees a User-scope variable, any shell opened before you set it doesn't have it, and neither does anything launched from those. Each of those silently rebinds the server to localhost the next time something runs lms server start.

That is not hypothetical: on 2026-08-26 port 1234 went unreachable over Tailscale while ComfyUI on 8188 stayed fine, and it looked exactly like a firewall bug — the rules were audited twice and were correct. The listener had reverted to 127.0.0.1, because a startup script called lms server start without the variable in scope.

lms server start --port 1234 --bind 0.0.0.0

Diagnostic worth internalising: "unreachable" has two causes that look identical from a client — a firewall dropping the packet, and nothing listening on that interface. Check the second one first; it is one command and rules out the more tedious audit:

Get-NetTCPConnection -LocalPort 1234 -State Listen |
  Select-Object LocalAddress, LocalPort

0.0.0.0 means every interface. 127.0.0.1 means only this machine can reach it, no matter how permissive the firewall is.

Also worth knowing: closing the LM Studio window kills the server — the GUI and server share a process tree, and requests then fail ECONNREFUSED. lms server start brings it back headlessly ("Waking up LM Studio service..."), but with no model loaded, so pair it with lms load or accept the cold-start delay on the first call.

Done when: from your dev machine, with the box's Tailscale IP from tailscale status:

curl http://<box-tailscale-ip>:1234/v1/models

returns the model list.

6a. Lock the port to Tailscale only

"Serve on Local Network" is all-or-nothing: it binds every interface, so LM Studio now answers to your whole home network with no authentication — guests, a kid's laptop, any IoT device. Nobody can administer the box that way, but they can consume it, and the inference runtime itself becomes attack surface (LM Studio's real CVEs have been file-parsing bugs).

You don't need LAN access: Tailscale reaches the box from anywhere, including from the couch. So allow the port only from Tailscale's address range (100.64.0.0/10) and block it everywhere else. In an elevated PowerShell on the box:

# 1. See what currently allows these ports (installer/prompt-created rules)
Get-NetFirewallPortFilter |
  Where-Object { $_.LocalPort -in @('1234','8188') } |
  ForEach-Object { $_ | Get-NetFirewallRule } |
  Select-Object DisplayName, Direction, Action, Enabled, Profile

# 2. Disable the permissive ones it lists, by name, e.g.
#    Get-NetFirewallRule -DisplayName "LM Studio*" | Disable-NetFirewallRule

# 3. Allow ONLY the Tailscale range. Windows blocks inbound by default,
#    so anything not matching these is denied without a block rule.
New-NetFirewallRule -DisplayName "LM Studio (Tailscale only)" `
  -Direction Inbound -Protocol TCP -LocalPort 1234 `
  -RemoteAddress 100.64.0.0/10 -Action Allow -Profile Any

New-NetFirewallRule -DisplayName "ComfyUI (Tailscale only)" `
  -Direction Inbound -Protocol TCP -LocalPort 8188 `
  -RemoteAddress 100.64.0.0/10 -Action Allow -Profile Any

# 4. Confirm the default really is deny
Get-NetFirewallProfile | Select-Object Name, DefaultInboundAction

Do not add a blanket -Action Block rule for these ports. Windows evaluates block rules before allow rules, so a blanket block would override the Tailscale allow and cut off remote access too. Default-deny plus a scoped allow is the correct shape.

Windows applies Block rules before Allow rules of the same specificity, so order doesn't matter here — but the allow rule's narrower RemoteAddress is what lets tailnet traffic through.

Done when: the Tailscale-IP curl above still works, and the same request to the box's LAN IP (192.168.x.x, from ipconfig on the box) times out:

curl -m 5 http://<box-lan-ip>:1234/v1/models   # should fail
curl http://<box-tailscale-ip>:1234/v1/models  # should still work

If you later run ComfyUI (8188) or a TTS server (8880) and reach them remotely, give each the same pair of rules.

7. Production leg: Cloudflare Tunnel + Access service token

Cloudflare Tunnel is outbound-only: cloudflared on the box dials out to Cloudflare, and requests to your public hostname ride back down that connection. No inbound ports, no router config, Windows Firewall stays fully closed to the internet.

The catch: LM Studio's local server has no authentication, so the tunnel hostname must not be left open — anyone who finds it could run inference on your box. Cloudflare Access with a service token locks it at Cloudflare's edge; unauthenticated requests never reach your house. This is Cloudflare's standard machine-to-machine service-auth pattern, not a homegrown scheme.

Do these subsections out of order: 7b → 7c → 7a.

Publishing the hostname is what puts an unauthenticated endpoint on the public internet, and the Access application is what closes it. Doing them in written order leaves a gap between the two — and 7c is a multi-screen wizard that is easy to get wrong on the first attempt, so that gap is measured in however long it takes you to get the dashboard right. On this box that was ~40 minutes of open, unauthenticated inference (build-log.md).

Build the lock first. Create the service token (7b) and the Access application (7c) with the hostname typed in before it resolves — Cloudflare accepts that without complaint — and publish the hostname (7a step 3) last. The door is then locked before it exists.

If you find yourself with a published hostname and no policy, Stop-Service cloudflared from an elevated shell takes every public hostname dark in about a second, without touching Cloudflare config or your Tailscale access.

7a. Create the tunnel

Prerequisite: a domain on Cloudflare. A tunnel hostname needs a domain whose nameservers point at Cloudflare — the account and the Zero Trust features used here are free. Use a domain that is currently empty — pointing its nameservers at Cloudflare then breaks nothing. Do not repurpose the domain serving your live site: moving nameservers means re-creating every existing record with production at stake.

Add the domain in the Cloudflare dashboard, change the nameservers at your registrar to the pair Cloudflare shows, and wait for it to go active — usually minutes, occasionally hours.

Cloudflare Zero Trust dashboard → Zero Trust → Networks → Tunnels → Create a tunnel (Cloudflared connector type).
Copy the Windows install command it shows — it installs cloudflared as a Windows service with the tunnel token baked in, so it starts on boot with no logged-in user needed.
Add a Public Hostname: something on a domain you have in Cloudflare (e.g. llm.example.com) → service type HTTP, URL localhost:1234.
(For the planned RabinAI image toy on Rabin Forest) add a second public hostname while you're here — e.g. img.example.com → HTTP, localhost:8188 (ComfyUI) — and include it in the same Access application in 7c. The backend adapter and load safeguards come later; having the hostname ready costs nothing.

7b. Create the service token

Zero Trust → Access → Service Auth → Service Tokens → Create.
Name it after the caller (bfoster-services), pick a duration (default 1 year — put the expiry on a calendar; rotation means generating a new token and swapping two env vars).
Copy the Client ID and Client Secret now — the secret is shown once.

7c. Gate the hostname with an Access application

Zero Trust → Access → Applications → Add → Self-hosted, with the application domain set to the tunnel hostname from 7a.
Add a policy with action Service Auth (not Allow — Allow is for humans with identity logins), and under the include rule pick the service token from 7b.

Now every request must carry the token headers or Cloudflare rejects it at the edge. Verify both halves:

CF_ID='<client-id>'         # ends in .access
CF_SECRET='<client-secret>'

# 1. no headers → 403 from Cloudflare, never touches the box
curl -s -o /dev/null -w "%{http_code}\n" https://llm.example.com/v1/models

# 2. with headers → 200 with LM Studio's model list
curl -s https://llm.example.com/v1/models \
  -H "CF-Access-Client-Id: $CF_ID" -H "CF-Access-Client-Secret: $CF_SECRET"

# 3. right ID, WRONG secret → must also be 403
curl -s -o /dev/null -w "%{http_code}\n" https://llm.example.com/v1/models \
  -H "CF-Access-Client-Id: $CF_ID" -H "CF-Access-Client-Secret: deadbeef"

Run all three. The first two passing only proves the two calls differ somehow; the third is what proves Access is validating the token rather than just noticing that headers exist.

What a failure tells you:

Result	Meaning
`200` unauthenticated	No policy in force — application never saved, action is `Allow` instead of `Service Auth`, or its destinations omit this hostname
`403` both ways	Headers don't match the token — truncated paste, or the policy includes a different one
`403` bare, `502` authenticated	Access works, routing doesn't — the connector can't reach the origin
`530`	No connector running

On Heroku, set LOCAL_LLM_URL=https://llm.example.com/v1 plus the two LOCAL_LLM_CF_ACCESS_* vars from Section 8 — the adapter attaches the headers automatically when they're present.

Done when: curl 1 returns 403, curl 2 returns 200 with the model list, and curl 3 returns 403 — the lock, the key, and proof the lock inspects the key.

Two things that surprise people afterwards:

The hostname stops working in a browser. A Service Auth policy permits no identity login, by design. Tailscale (http://my-box:1234) stays your human-facing path to the box.
cloudflared service install needs an elevated shell, as do Stop-Service / Start-Service cloudflared. Without it you get "Cannot establish a connection to the service control manager: Access is denied."

8. Wiring it into `extract.js` and `events-ai-extract.js`

Follow the same fail-open pattern semantic.js already uses: try local, fall back to the existing Gemini path on any error, missing config, or timeout. One small adapter, not a rewrite.

This is not the only adapter, and not the one that ships today.

bfoster-services already calls the box from server/ai.js for the Rabin Forest assistant chat, using RABINAI_* environment variables. That code is built and deployed. The LOCAL_LLM_* adapter below is for AskGwinnett ingest extraction and has not been written — nothing in that repo reads LOCAL_LLM_*.

Two integrations, two prefixes, two purposes. Setting LOCAL_LLM_* config vars does nothing today; the assistant needs RABINAI_URL, RABINAI_CF_ACCESS_ID, RABINAI_CF_ACCESS_SECRET, RABINAI_MODEL and RABINAI_ASSISTANT=on. When you build this section, decide deliberately whether to keep the prefixes separate or converge them — but don't assume one set configures both.

Before you add auth to an endpoint, find every caller. Enabling Cloudflare Access on llm.example.com instantly broke two of the four existing fetches in server/ai.js, which had been sending no headers because none were needed. One of them was the assistant's warm-up ping, whose failure pinned the whole feature to Gemini permanently while logging a single line. Adding authentication is a breaking change for every caller you forgot about — grep for them first (build-log.md has the full account).

Two things about LM Studio's OpenAI compatibility to get right before writing any of this, both verified against LM Studio 0.4.21:

response_format: { type: 'json_object' } is rejected, with 'response_format.type' must be 'json_schema' or 'text'. This is the one place its OpenAI compatibility stops short. Use json_schema and pass a real JSON Schema object ({ type: 'object', properties: {...}, required: [...] }) — which is stricter and better anyway, since the server constrains generation to the schema instead of trusting the model to follow a prompt.

But the schema constrains shape, not meaning. A field typed string will always be a string and always be present; nothing makes it a useful string. The same schema and input returned "date": "2026-10-06" on one run and "date": "October 6, 2026" on another. Keep stating formats in the prompt ("ISO date YYYY-MM-DD, 24-hour HH:MM") — the schema stops malformed JSON, not sloppy values — and normalize dates downstream rather than trusting either.
The model field is a suggestion, not a contract. If the name resolves to a downloaded model, LM Studio serves (and JIT-loads) it. If it doesn't resolve, there's no error — the server quietly answers with whatever model is currently loaded. Requesting totally-not-a-model returns a cheerful 200 from Qwen. You only get a 400 ("No models loaded") when nothing is resident at all.

So the failure mode isn't a broken pipeline, it's a silently wrong one: after an evening of playing with other models, a typo in LOCAL_LLM_MODEL means extraction runs on whatever you left loaded, and the output looks plausible either way. Copy the id from /v1/models or lms ls, exactly. Then don't trust the response to tell you if you got it wrong: LM Studio echoes the id you asked for in the response's model field even when another model answered (measured 2026-08-27 and again 2026-10-09, when a bench model served production traffic for half an hour with every response labelled as the model that was not loaded). The only honest check is the catalogue: ask /v1/models what is loaded before you trust an answer, and refuse rather than substitute. Log what you sent, not what came back (see the end of this section).

// server/askgwinnett/local-llm.ts
import OpenAI from 'openai';

const LOCAL_LLM_URL = process.env.LOCAL_LLM_URL; // e.g. http://localhost:1234/v1
const LOCAL_LLM_MODEL = process.env.LOCAL_LLM_MODEL || 'openai/gpt-oss-20b'; // exact id from /v1/models — org prefix included
// Reasoning models think by default (~5x slower) and each family has its own off-switch; see Section 4.
const LOCAL_LLM_NO_THINK = process.env.LOCAL_LLM_NO_THINK === 'true';
// 30s is not enough for a cold call: a JIT model load is ~16-19s on its own,
// and a reasoning-mode extraction then takes ~15s more. See the note below.
const LOCAL_LLM_TIMEOUT_MS = Number(process.env.LOCAL_LLM_TIMEOUT_MS) || 90_000;
// Cloudflare Access service token (Section 7) — unset for Tailscale/local use
const LOCAL_LLM_CF_ACCESS_ID = process.env.LOCAL_LLM_CF_ACCESS_ID;
const LOCAL_LLM_CF_ACCESS_SECRET = process.env.LOCAL_LLM_CF_ACCESS_SECRET;

let client: OpenAI | null = null;
function getLocalClient(): OpenAI | null {
  if (!LOCAL_LLM_URL) return null;
  if (!client) {
    client = new OpenAI({
      baseURL: LOCAL_LLM_URL,
      apiKey: 'not-needed',
      defaultHeaders:
        LOCAL_LLM_CF_ACCESS_ID && LOCAL_LLM_CF_ACCESS_SECRET
          ? {
              'CF-Access-Client-Id': LOCAL_LLM_CF_ACCESS_ID,
              'CF-Access-Client-Secret': LOCAL_LLM_CF_ACCESS_SECRET,
            }
          : undefined,
    });
  }
  return client;
}

/**
 * Attempts extraction against the home box. Returns null on any failure —
 * missing config, timeout, box offline — so callers fall back to Gemini
 * exactly like semantic.js does for embeddings.
 */
export async function tryLocalExtraction(
  prompt: string,
  schema: Record<string, unknown> // a JSON Schema: { type: 'object', properties, required }
): Promise<string | null> {
  const c = getLocalClient();
  if (!c) return null;

  const controller = new AbortController();
  const timer = setTimeout(() => controller.abort(), LOCAL_LLM_TIMEOUT_MS);

  // The schema guarantees shape; the prompt still has to ask for sane values.
  const system =
    'Extract the requested fields as JSON. Use ISO dates (YYYY-MM-DD) and 24-hour times (HH:MM).' +
    (LOCAL_LLM_NO_THINK ? ' /no_think' : '');

  try {
    const completion = await c.chat.completions.create(
      {
        model: LOCAL_LLM_MODEL,
        messages: [
          { role: 'system', content: system },
          { role: 'user', content: prompt },
        ],
        response_format: {
          type: 'json_schema',
          json_schema: { name: 'extraction', strict: true, schema },
        },
      },
      { signal: controller.signal }
    );
    return completion.choices[0]?.message?.content ?? null;
  } catch (err) {
    console.warn('Local extraction unavailable, falling back to Gemini:', (err as Error).message);
    return null;
  } finally {
    clearTimeout(timer);
  }
}

Then in extract.js, the change is additive — try local first, keep the existing Gemini call as the fallback path:

const { tryLocalExtraction } = require('./local-llm');

async function extractMeeting(rawText, schema) {
  const local = await tryLocalExtraction(rawText, schema);
  if (local) return JSON.parse(local);

  // existing Gemini extraction call — unchanged
  return extractMeetingViaGemini(rawText, schema);
}

Set LOCAL_LLM_URL only in your local .env (not on Heroku) until you've decided whether to wire the box into production. This keeps today's Gemini-based pipeline completely untouched — the local box is purely additive, and unplugging it later means deleting one env var.

The first call after the box wakes is the slow one

The 30-second timeout this guide originally suggested is too short in two separate ways.

Cold start. When the named model isn't resident, LM Studio JIT-loads it before generating — ~16-19s for the 30B. Measured cold, the call aborted at 30s; the identical call, warm, returned in 2.7s. In production that reads as a flaky box: the first job after the box wakes silently goes to Gemini, every job after it runs locally.

Reasoning-mode variance. How long thinking takes depends on how hard the model decides the problem is, and it is not stable. The same extraction took 15s on one run and 38s on another, both warm — the second would have blown a 30s budget with the model already loaded. If you leave thinking on, budget for the tail, not the average.

Three ways to deal with it, and they combine:

Raise the timeout past a cold start — 90s, as above.
Keep the model resident. lms load without a --ttl keeps it loaded indefinitely; JIT-loaded models expire after an hour by default.
Warm it deliberately at the start of a batch run — one throwaway request, or lms load — so the first real extraction is already hot.

Failing open means a misconfigured box and a healthy one produce the same extraction results — Gemini's — so nothing in the output tells you the local path never ran. An expired Access token, a box that's off, a tunnel that's down: all identical from the caller's side.

The console.warn above is the only signal, so make it countable rather than decorative. Log both facts you can't otherwise recover — which path served the job, and which model actually did it (remember the model field you sent is not authoritative):

// have tryLocalExtraction return the whole completion, not just content
const local = await tryLocalExtraction(rawText, schema);
console.info(`extract: ${local ? `local/${local.model}` : 'gemini'}`);

Then "did the box do any work last night, and with which model?" is a log grep instead of a guess. Worth having before the first unattended batch run — both a silently idle box and a silently wrong model can otherwise hide for weeks behind perfectly plausible output.

Done when: with the box on, tryLocalExtraction returns valid JSON for a real sample; with the box off (or LOCAL_LLM_URL unset), the same call quietly returns null and the Gemini path produces the result — test both directions before trusting the fallback.

9. Alternate build: coding assistant for your IDE (optional)

Same box, same LM Studio server, same Tailscale leg — the only changes are the model and what's on the other end. Sections 1-3 and 6 are prerequisites.

Two 30B models do not coexist. Corrected 2026-08-28, refined 2026-09-10 — an earlier version of this section said both models could be resident at once (~9GB + ~19GB) and that you need not unload anything. That was written against a 48GB carve-out and a 14B coder, and it is wrong for this build: two 30B-class models need ~37.3GB of a 32GB carve-out, so the coder and the chat model are genuinely one-at-a-time.

That is a statement about these two models, not a law about the box. An earlier version of this line said flatly that the box runs one LLM at a time, which 4a now measures more precisely: an idle model costs memory but no bandwidth, and a 30B plus the smaller 20B can both stay resident if you are not generating images. The binding constraint is the memory budget and whether SDXL needs its ~7GB — not a rule about counting models. Size the pair you actually want against the carve-out, the way 4a does.

9a. Load a coding model

Use qwen/qwen3-coder-30b at GGUF Q4_K_M (~18.6GB) — same MoE shape as the chat model, so the same ~31 t/s rather than a dense model's 3-6.

lms get qwen/qwen3-coder-30b@q4_k_m --gguf -y
lms load qwen/qwen3-coder-30b --gpu max -y --context-length 32768

At the 32GB carve-out the arithmetic is 18.6 (coder) + ~7 (SDXL) = ~25.6GB, leaving ~6GB spare. Two 30B models would need 37.3GB and do not fit — that is the whole reason this is one-at-a-time.

Swapping the resident model is a production change, not a local one — and where that choice lives has changed.

Originally bfoster-services read the model name from RABINAI_MODEL, so swapping meant editing a config var. That is the wrong mechanism for this, for a reason worth internalising: changing a config var restarts the process, and the restart kills the very swap you are performing. It also puts the authority somewhere the box cannot read, which forces the box to keep its own copy of the answer — and a second copy of a decision is a stale copy waiting to happen (section 12b).

The choice now lives in a small persisted document the service owns, is set from a control page, and is published on a status endpoint that anything — including the box's own warm-up script — can read. RABINAI_MODEL is no longer even the fallback: as of 2026-09-12 it names only the qwen entry in the swap allowlist, and the boot default is its own setting (RABINAI_DEFAULT_MODEL; the built-in default is Qwen3.6 since 2026-10-22, gpt-oss before that). It had been doing both jobs, which meant pointing it at gpt-oss to change the default would have silently made a swap request for "qwen" load gpt-oss. Any documentation that describes RABINAI_MODEL as the mechanism, including older parts of this guide, is describing history.

The rule generalises past this implementation: put the switch somewhere you can change without a restart, and somewhere every participant can read. Anything forced to keep its own copy will eventually enforce a stale one with total confidence.

Related, and newer still: context length is becoming per-model rather than one global, because the two models are no longer interchangeable — see the KV figures in section 12c. Anything on the box that pins a single context length for all models will fight that the moment the two diverge.

Get this wrong and nothing errors — LM Studio 404s the unknown model, server/ai.js fails open, and the site serves the cloud model indefinitely with one log line as the only symptom. That is the same failure shape as the missing Access headers on 08-25. Check with:

curl -s https://your-api.example.com/ai/gemini-assistant   -H 'Content-Type: application/json' -H 'Origin: https://your-site.example.com'   -d '{"prompt":"what does Brian do?"}'      # engine= is the whole diagnostic

Also turn off JIT loading (justInTimeModelLoading: false). With it on, a production request naming the other model makes LM Studio load a second copy alongside the resident one — 37GB against a 32GB carve-out, which is how the box hung on 2026-08-28.

9b. Point your IDE at it

Continue.dev (VS Code or JetBrains) is the standard client for local models. Install the extension on your dev machine (not the box) and add the box to its config:

models:
  - name: home-box-coder
    provider: lmstudio
    model: qwen2.5-coder-14b-instruct
    apiBase: http://<box-tailscale-ip>:1234/v1
    roles: [chat, edit]

Cline / Roo Code work too: pick the "OpenAI Compatible" provider, set the base URL to the same address, and enter anything as the API key (LM Studio ignores it).

Skip tab-autocomplete for now — it needs sub-second latency that this box can't deliver over the network. Chat and edit-selection are the usable modes.

9c. Why not Cursor

Cursor routes model calls through its own servers, not your machine, so a Tailscale address is unreachable — you'd have to point it at the public Cloudflare hostname, and Cursor can't send the Access service-token headers, so the only way to make it work is stripping the auth off your tunnel. Don't weaken the lock for it — Cursor's marquee features are restricted to its own models anyway. Use Continue or Cline on this box, and Cursor with its own models.

Expectation check: a Coder-14B is a competent assistant, not a frontier one — good for privacy-sensitive code, offline work, and unlimited usage; not a replacement for Claude Code or Cursor's hosted models on hard problems.

Done when: a Continue chat on your dev machine answers a coding question while the box's GPU graph spikes.

10. TTS: give the box a voice (optional)

Text-to-speech is its own model class — LLMs don't speak, and LM Studio doesn't serve audio — but open TTS models are tiny next to everything else on this box and run faster than real time on CPU alone.

10a. Kokoro — the quick win

Kokoro (~82M params, Apache) is the current open-weight standout: excellent quality at a thousandth the size of your text model. The community server kokoro-fastapi wraps it in an OpenAI-compatible /v1/audio/speech endpoint — the same API shape the rest of this guide already speaks, which is why the backend can treat it as a drop-in for OpenAI TTS rather than a second integration.

Why this one is easy on this hardware. The box's binding constraint everywhere else is memory bandwidth — a ~3B-active MoE reads ~1.6-2GB per token, which is what caps it at 20-31 tok/s. Kokoro is about a thousandth that size, never approaches the ceiling, and is compute-bound instead, which is the axis with headroom. It runs comfortably faster than real time on CPU; the 890M iGPU is not needed and not worth fighting for.

Install

espeak-ng is optional on current builds — skip it and come back only if synthesis complains. The kokoro package ships espeakng_loader in the venv and phonemises through that.

Measured on the box 2026-09-16: the winget install below failed (MSI 1603, an unanswered UAC prompt) and synthesis worked regardless — a 1.78s MP3 returned in 1.0s with no system-wide espeak-ng present. Older releases did need it, which is why it is still written down:

winget install -e eSpeak-NG.eSpeak-NG   # only if you hit a phonemizer error

Note it needs admin. Do not go chasing elevation for it before you have seen an actual phonemizer error at first synthesis.

Then the server itself, same venv pattern as ComfyUI in Section 5:

cd $HOME
git clone https://github.com/remsky/Kokoro-FastAPI.git
cd Kokoro-FastAPI
python -m venv venv
.\venv\Scripts\Activate.ps1

# CPU torch EXPLICITLY. The default index pulls a CUDA build — ~2.5GB of
# wheels that cannot run on an AMD iGPU and will silently fall back to
# CPU anyway, if it imports at all.
pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install -r requirements.txt

The repo ships its own start scripts, and their names have changed between releases — check the README for the current CPU entrypoint rather than trusting a name written here. Whatever it is called, it should bind port 8880. First run downloads the model (~350MB).

If the native install fights you, the project also publishes Docker images and that route is more deterministic — at the cost of adding Docker Desktop to a box that does not otherwise need it.

Open the port to Tailscale only

Same rule as everything else here: reachable from your machines, closed to the internet (Section 6).

New-NetFirewallRule -DisplayName "Kokoro TTS (Tailscale only)" `
  -Direction Inbound -Protocol TCP -LocalPort 8880 `
  -RemoteAddress 100.64.0.0/10 -Action Allow -Profile Any

Verify

curl -X POST http://localhost:8880/v1/audio/speech `
  -H "Content-Type: application/json" `
  -d '{"model":"kokoro","input":"RabinAI, online.","voice":"af_heart","response_format":"mp3"}' `
  --output test.mp3

Ask for mp3: it is what the browser gets served and what the backend requests, so testing the format you actually ship avoids discovering a wav/mp3 mismatch later.

Kokoro ships a fixed set of voices (af_*/am_* female/male American, bf_*/bm_* British) — pick one as RabinAI's default.

Wire it to the site

bfoster-services already prefers Kokoro and falls back to OpenAI TTS on any failure, so this is configuration, not code:

KOKORO_URL=http://100.x.y.z:8880/v1     # Tailscale, or the tunnel hostname
KOKORO_VOICE=af_heart

Leave it unset and TTS still works on OpenAI — the box stays genuinely optional, the same way it is for the assistant. For anything reaching it from outside the mesh (Heroku), add a tunnel hostname pointing at localhost:8880 and reuse the same Access token, exactly like ComfyUI in 7a; the backend already sends those headers.

Done when: test.mp3 plays back the sentence, and the Rabin Forest read-aloud button returns X-TTS-Engine: kokoro instead of openai.

Watch for: narration and chat now share the CPU. Chunked read-aloud synthesises while the language model is still generating the answer it is speaking, and this box is run at three concurrent requests. A fourth, on a long prompt, failed everything already in flight (section 12c). If answers get slower once TTS is live, that contention is the first thing to measure — not the model.

10b. Mapping your own voice (eventually)

Kokoro cannot clone voices — fixed roster only. Cloning is a different model family, and the current pick is Chatterbox (Resemble AI, MIT-licensed): zero-shot cloning from a short reference clip, no training run needed. (The older alternative, Coqui XTTS-v2, clones well but its license forbids commercial use — Chatterbox's MIT is cleaner.)

The recipe when you get to it:

Record ~10-20 seconds of clean speech. The clip's quality caps the clone's quality, so it's worth five minutes of care:
- Room beats microphone. A phone mic is fine; echo is not. Record in a soft-furnished room (or facing a closet full of clothes) — never a bathroom or kitchen. If you can hear the room in the recording, so will every clone.
- Kill the hum. HVAC, fans, fridge, the box itself — background noise gets baked into the voice. Pause the loud stuff.
- Mic at 6-8 inches, slightly off to the side of your mouth so P's and B's don't pop, and speak at normal volume — no clipping, no whispering.
- Delivery is what gets cloned, not words. Speak the way you want RabinAI to sound: natural conversational energy, complete sentences, some rise and fall. A monotone read produces a monotone clone. The actual content doesn't matter.
- One continuous take, no edits, no music, nobody else audible. Save as WAV (or the highest-quality format your recorder offers) — re-record rather than clean up a bad take.
Follow the Chatterbox README to stand it up (Python, same venv habits as ComfyUI) and pass the clip as the reference audio.
Expect slower-than-Kokoro generation on this hardware — cloning models are ~1-2GB and heavier per sentence. Fine for playground use, not for narrating books.

Note: Chatterbox watermarks its audio output (inaudible, detectable) — sensible default for cloned voices, and no obstacle for RabinAI duty.

Done when: the box says something you never recorded, in your voice, and it's only medium-creepy.

10c. ElevenLabs — the honest alternative

Everything above keeps the voice on this hardware. This section does not, and it is here because for one specific job — your voice, good enough that a stranger doesn't wince — the hosted option is still meaningfully ahead of anything self-hostable, and pretending otherwise in a guide you wrote for yourself would be silly.

Read this before spending anything: you cannot buy a portable voice. ElevenLabs (and PlayHT, and Resemble) keep the model on their servers. There is no weights file to download and serve from the box. You are renting a voice, not acquiring an asset — which is the exact opposite of what 10b/10c-local are for. If "runs on my hardware" is the point, stop here and fine-tune Piper instead.

Two clone grades:

Instant Voice Clone — a couple of minutes of audio, ready immediately, noticeably better than any local zero-shot option.
Professional Voice Clone — closer to 30 minutes to a few hours of audio, trained over hours, and the one that actually sounds like you. Gated behind a paid tier and an identity-verification recording, which is a reasonable thing for them to require and a real step to plan for.

Either way the recording rules in 10b are unchanged and still the highest-leverage part. A hosted model does not fix a bad room.

The shape that matters: do NOT synthesise live per answer

The instinct is to slot ElevenLabs in as another engine ahead of OpenAI in /readaloud. Do the arithmetic first. Billing is per character, and a typical assistant answer is ~1,000 characters — so a 100k-character monthly allowance is roughly 100 answers. One curious visitor clicking through a conversation can eat a day's worth. On a public portfolio that is not a cost risk, it is a certainty.

Two shapes fix it, and they compose:

Pre-render the set pieces. The intro, the project blurbs, the greeting — a fixed list, synthesised once, committed as static MP3s and served from Vercel. Zero runtime cost, instant playback, and you can re-roll a take until it sounds right, which is where a good voice earns its money anyway.
Cache by answer hash. For arbitrary answers, key the audio on a hash of the text and store the MP3. The first person to ask a question pays for it; everyone after gets it free and instantly. The assistant's answers to common questions repeat far more than they vary, so the hit rate is high — this is the same trick as AskGWINnett's first-turn answer cache, applied to audio.

With both in place, ElevenLabs is affordable. With neither, it is a metered API on a public button, which is the one configuration to avoid.

Wiring

The backend already picks an engine and falls back, so this is a third branch rather than a new integration: prefer a cached/pre-rendered clip, then ElevenLabs when configured, then Kokoro, then OpenAI. The client is unchanged — it asks for audio and plays what comes back.

ELEVENLABS_API_KEY=...
ELEVENLABS_VOICE_ID=...        # the cloned voice
ELEVENLABS_MODEL=...           # a low-latency model for live use

Leave them unset and nothing changes.

Done when: the pre-rendered intro plays in your voice with no network call at all, and a fresh answer is spoken once and served from cache the second time.

The honest summary of section 10: Kokoro for live narration because it is fast and local; a fine-tuned Piper voice if owning the asset matters more than polish; ElevenLabs if polish matters more than owning it. Those are three different goals and there is no configuration that serves all three.

11. Quick sanity checklist before first real extraction run

BIOS updated, VRAM allocation set (32-48GB)
AMD Adrenalin driver installed
Toolchain in place: git --version, node --version, py --version, claude --version all answer (Section 3)
LM Studio loads the language model with GPU offload confirmed in Task Manager (not silently running on CPU)
Local server running at http://localhost:1234/v1
Tailscale connects from your dev machine to the box
Port 1234 locked to Tailscale: LAN-IP curl fails, Tailscale-IP curl works (Section 6a)
(Production leg) tunnel hostname returns 403 without the Access token headers and 200 with them — both curls from Section 7c
LOCAL_LLM_MODEL matches /v1/models exactly (openai/gpt-oss-20b keeps its org prefix; an imported file like qwen3.6-35b-a3b has none) — an unresolvable name doesn't error, it silently runs on whatever model is loaded, and the response claims otherwise
Timeout is long enough for a cold JIT load (~16-19s) plus generation — 30s is not; 90s is (Section 8)
LOCAL_LLM_URL set locally, tryLocalExtraction returns valid JSON on a real meeting-minutes sample
Logging records both the path taken and the model the response says served it, so an idle box and a wrong model are both visible (Section 8)
Run one extraction both ways (local vs. Gemini) on the same input and spot-check the output quality before trusting it at scale
Run that same comparison a third way — local with /no_think — and decide whether the 5x speedup costs you any field quality (Section 4)

12. Operating it: what the first month teaches you

The build above is one afternoon. Keeping the box serving is the part no guide warned us about, and every subsection here is a failure that actually happened on this machine, with the date attached. If you read one section after getting the box working, read 12c.

The theme, stated once: almost every outage here came from something being checked against a value that had stopped being true — not from hardware, and not from the model.

12a. It does not stay up by itself

A model that is loaded is not a box that is serving. The model can be unloaded by a TTL, the server can rebind to localhost, a swap can load the wrong thing. Something has to notice.

Before any of that, though: THE BOX SLEEPS, and for a long time we did not know it. This cost an hour twice, both times chasing the wrong cause, so it goes first.

This hardware supports no S1/S2/S3 — only Modern Standby (S0 Low Power Idle). That is not a sleep you have opted into; it is the system's idle state, entered whenever the machine goes quiet, regardless of your sleep timeouts. Measured here: STANDBY IDLE TIMEOUT AC = 0, HIBERNATE TIMEOUT AC = 0, no Group Policy — and the box still entered Modern Standby 14 times in 10 days. powercfg /change standby-timeout-ac 0 returns success, changes nothing, and looks exactly like a fix landing.

Symptom to recognise: the machine stops answering, and the event log shows Kernel-Power 41 with BugcheckCode = 0, no crash dump, no WHEA faults. That is a hang in low-power idle — not a reboot, not a power cut. Do not read a long uptime as proof it is fine; a 59-hour session ended cleanly here hours before the one that never woke up.

The fix is a firmware-behaviour change, and it takes TWO steps. The second is the one everybody misses:

# 1. Disable S0 Low Power Idle outright. Elevated, and needs a reboot.
#    HKLM\SYSTEM\CurrentControlSet\Control\Power\PlatformAoAcOverride = 0 (DWORD)

# 2. THEN kill the hibernate idle timer. Works unelevated.
powercfg /change hibernate-timeout-ac 0

Step 2 exists because step 1 worked. With S0 gone and no S3 available, Hibernate becomes the only low-power state left — and the default HIBERNATEIDLE on AC is three hours. Between the two fixes this box hibernated every ~3 idle hours and needed someone to physically wake it, which is worse than what it replaced. Say the consequence out loud before you start: after step 1 the machine has no standby state at all.

Verification, and both halves are required:

powercfg /a   # must NOT list Standby (S0 Low Power Idle)
powercfg /q   # HIBERNATEIDLE on AC must be 0x0

A timeout reading zero proves nothing on its own — that is the trap in the first paragraph. Re-check both after Windows feature updates and after installing any vendor power utility.

With that settled, the box stays up. Everything below is about noticing when the software falls over, which is a different problem.

Two Scheduled Tasks, both running the same script:

Task	Trigger	Repeats	Job
warm	at boot, +2 min	no	bring it up after a reboot
watchdog	boot +1 min, and a time trigger	every 10 min	notice when it has fallen over

The watchdog does not actually run every 10 minutes. Both triggers carry their own independent repetition and they are offset from each other, so they interleave. Measured from the log on 2026-09-09, the gaps alternate:

20:02:00 -> 20:08:20   6m20s
20:08:20 -> 20:11:59   3m39s
20:11:59 -> 20:18:20   6m21s

Real worst case is ~6m20s, not 10 minutes. That is only ever good news for recovery time, but it is not what the registration code reads as — and it means a change you make by hand can be reverted before you have finished testing it.

The watchdog must be conditional. Re-priming is not free to repeat: it invalidates the cached prefix, so visitors get the cloud fallback while it re-warms. An unconditional timer hands every visitor a cloud answer on a schedule — a slower, quieter version of the outage it exists to prevent.

12b. "Healthy" is whatever your health check says it is

Health here means four things, not one, because each has failed alone:

the right model is loaded
at the right context length
bound to the right interface, not localhost
exactly one copy of it

A server that answers on localhost is invisible over the tailnet. A model at the wrong context throws errors only on long conversations. A second copy of a 19 GB model fits, succeeds quietly, and takes the memory with it.

The trap, and it caught us twice in two days: a health check compares reality against something, and the moment that something is a hardcoded constant, the check becomes a bug as soon as the constant stops being the source of truth.

On 2026-09-09 the resident model was chosen by a constant in the script. The authoritative choice had moved elsewhere. So the watchdog measured the box against a model nobody had asked for, declared it "not loaded", and helpfully reloaded it — every few minutes, underneath a deliberate swap. Nothing was broken. Everything worked exactly as written.

The fix is not a better constant. It is for the check to ask whoever owns the decision. If a value can be changed somewhere else, your checker must read it from there, or it will eventually enforce a stale answer with total confidence.

A corollary worth its own line: a check that detects a fault and then does not act on it is worse than no check, because it looks like monitoring. When a slot-count check was first added here, it correctly logged parallel is 4, want 3 — and the repair path, which compared only the context length, skipped the reload as a no-op. It would have reported the problem forever while fixing nothing. That was caught by running it, not by reading it.

12c. Concurrency has a budget, and going over it fails everything in flight

This is the one that will bite you, because nothing warns you and the failure is loud, total, and looks like something else.

The server admits some number of concurrent predictions — a slot count. Each in-flight request needs room for its prompt. If your assistant sends a large system instruction on every call, concurrency multiplies it.

What happened here on 2026-09-10: the slot count was 4, the system instruction was ~9,400 tokens, the context was 32,768. Four concurrent requests failed with:

Context size has been exceeded

Not the fourth request. Every request in flight. From the outside it looked like the box had died: the knowledge-base prime never completed, the tunnel timed out, the prime retried, and it looped for hours while every visitor got the cloud fallback. Three concurrent requests of the same size had been completing fine minutes earlier.

Dropping the slot count to 3 restored it.

Be careful what you conclude from that — we were not careful enough. The obvious reading is that the context is one shared pool, and 3 x 9,400 fits where 4 x 9,400 does not. That arithmetic works, and it is not established. Contradicting evidence from the same log: requests of 9,389 tokens completed successfully while the slot count was still 4, which a strict per-slot division of 32,768 across four slots would have made impossible. Other things changed in the same window, so even the fix's causation is not cleanly attributed.

So take the operational rule, which is well supported, and leave the mechanism alone:

know your prompt size — it is a budget, not a detail
keep the slot count low unless you have measured that you can afford more; on one integrated GPU serving one thing at a time, extra slots buy little
treat "Context size has been exceeded" as a concurrency symptom first, not a prompt-too-long symptom — the single request that triggers it is usually innocent
if you need headroom, raising the context length is a cheaper lever than trimming content, and the KV cost is linear in context length

What the headroom actually costs. LM Studio logs a load-size estimate when it loads a model. Measured on this box, 2026-09-10:

Model	Weights	KV @ 8k	KV @ 32k	KV @ 64k
20B dense (MXFP4)	12.47 GB	0.83 GB	~3.3 GB	~6.6 GB
30B MoE (Q4)	19.11 GB	1.23 GB	~4.9 GB	~9.8 GB

Two traps in that table. First, the figure LM Studio reports is computed at 8,192 tokens regardless of the context you actually loaded — it says so in the log — so you have to scale it yourself. The 32k and 64k columns are linear extrapolations, not readings. Second, the two models are not interchangeable at the top end: with the image stack also resident, 64k was comfortable for the 20B and did not fit for the 30B. A single global context length raised to help the small model will fail to load the big one, which is why that setting wants to be per-model.

KV allocates lazily, so measure it warm. You can read the real cost off the inference process's working set, but only after traffic has actually used the slots. Immediately after a load, the same process here showed 12.93 GB against 12.47 GB of weights — barely half a gigabyte of cache, which looks like the arithmetic above is wrong. Warm, the same process reads 16.44 GB: weights plus ~4 GB, which matches. A reading taken seconds after a reload will tell you the cache is free. It is not; it has not been allocated yet.

Measure your headroom against peak, not idle. This matters more than the KV numbers. The image services on this box sit at roughly 0.03 GB each while idle and take several gigabytes each the moment they render. Any "free memory" figure captured while they are quiet is systematically optimistic, and a context length chosen against it will fit perfectly right up until someone asks for a picture.

And the real ceiling is the split, not the total. This machine takes 2 x 32 GB and both slots are populated, so there is no upgrade path short of replacing it. What is adjustable, for free, is the BIOS carve-out from section 1 — the boundary between what Windows sees and what the GPU gets. Before trading prompt content away to fit a budget, check whether the budget is where you want it. The honest framing of the capacity question is not "bigger context or retrieval", it is "where do you want the 64 GB, and is image generation worth more to you than a long context". That is a decision about the box.

Not established as of 2026-09-11. The rule above is from observation. The pool arithmetic is a plausible explanation that has failed at least one test. Do not make an architecture decision on it without measuring.

12d. A model swap bypasses your loader

The loader script passes deliberate load options — slot count, context length, GPU offload. A swap triggered from the application does not go through that script. It loads the model directly, and gets the defaults.

So the box asserts the right settings on its own path and cannot assert them on anyone else's. Every swap silently re-introduces whatever the default is, and the setting appears to have "changed by itself". It had not: it was never applied.

Two fixes, and you probably want both:

send the same load options in the swap request, so the right thing is loaded the first time
have the watchdog check them, so anything loaded by any other route is corrected within a few minutes

If you only do (2), expect an extra reload and re-prime a few minutes after every swap. That is the watchdog doing its job, not a bug — but document it, or someone will chase it as one.

12e. The GUI can become impossible to open

If you start the server from a Scheduled Task that runs whether or not anyone is logged in, every process it starts lives in the non-interactive session — which has no desktop. The app runs headless there and holds a machine-wide single-instance lock.

The symptom is specific and very confusing: double-clicking the application does nothing at all. No window, no error, no new process. The launch finds the lock, hands off to the instance in the invisible session, and exits.

You cannot fix it by launching it again, and you cannot kill it from an ordinary shell — a normal-privilege session gets "Access is denied" against the task's processes. The recovery is: disable both scheduled tasks (the watchdog will re-grab the lock otherwise), kill the headless processes from an elevated shell, relaunch through the shell so the app lands on the desktop at normal privilege, then re-enable the tasks. tools/lmstudio-to-desktop.ps1 does exactly that, in that order, with the re-enable in a finally so a failure never leaves the box unwatched.

12f. Warm and cold are different machines

With a large system instruction, the first request after a load pays a full prefill and later ones do not. On this box that is the difference between tens of seconds and about a second — the single biggest lever on perceived speed, larger than the choice of model.

Two consequences:

Something must warm it deliberately, or your first real visitor pays that cost.
The warmer must not rebuild the prompt itself. Prefix caching is an exact-token match. A warmer that assembles its own copy of the instruction diverges at the first token the moment the real one changes, caches a prefix nobody asks for, and reports success every time. That version shipped here: it "warmed" in 0.6s — its own prefix, cached — while the first genuine question still paid a full cold prefill. Have the system that owns the string do the priming.

12g. Active parameters rule both phases — a theory we got wrong

An earlier version of this section said generation rate follows active parameters while prefill follows total parameters, so a big sparse model would generate fast and prefill slowly, and a small dense model would be the reverse. "Same box, opposite winners" was the phrase.

That is wrong on this hardware, and we ran an experiment on the strength of it before finding out. The reasoning behind it treats prefill as bandwidth- bound: reading the prompt touches every weight, so total size should set the pace. Measured on this box, 2026-09-13, cold prefill of the same ~24k-token prompt:

Model	Params	Active	Cold prefill
20B MoE (MXFP4)	20B	~3.6B	137 s / 151 s
7B dense (Q4_K_M)	7B	7B	287 s

The model with three times the total parameters prefilled roughly twice as fast. The prediction was not merely off, it had the sign backwards.

The coherent explanation is that large-batch prefill here is compute-bound, not bandwidth-bound. Weights stream once and amortise across the whole batch, so what sets the rate is per-token arithmetic — and that is governed by active parameters, exactly as it is for generation. Active parameters rule both phases. A dense 7B has 7B active and was never going to beat 3.6B active, whatever the totals say.

Two caveats kept deliberately, because the record is more useful honest than tidy. The two runs were not measured under identical conditions — the smaller model was competing with the larger one for compute — so the 2x factor is soft. And a keep-warm ping was landing intermittently during both. The direction is not in doubt; a contaminated 2x-slower result cannot become the predicted 3-4x-faster in a clean room. The magnitude is.

Prefill cost is also strongly superlinear, which matters more than the headline rate. Same model, same prompt file, one cold run per size:

Prompt tokens	Seconds	tok/s
1,106	6.7	165
4,462	20.9	214
8,287	49.0	169
16,198	144.4	112
23,907	287.0	83
23,907 repeated	0.1	cache hit

Throughput falls as the prompt grows — 214 tok/s at 4k, 83 tok/s at 24k. So doubling the prompt costs much more than double. Any figure quoted without the length it was measured at is meaningless.

The last row is the one to design around. An identical prompt returns in 0.1s. That is the prefix cache, and it is worth more than any model choice: a workload with a stable prefix pays the table above once and is instant thereafter. A workload that rebuilds its context every turn pays it every turn. This is why an append-only chat against this box feels fast while an agentic coding tool that reassembles its context does not — the same hardware, the same model, two completely different experiences.

Where the model choice lives matters too. Persist it somewhere you can change without restarting the application. Putting it in an environment variable means changing it restarts the process — which kills the very swap you are performing.