RabinAI

Docs

Home AI Box Setup — MINISFORUM AI X1 Pro-370 (Ryzen AI 9 HX 370, 64GB)

Public edition. Real hostnames and addresses have been replaced with placeholders (llm.example.com, 100.x.y.z). Everything else — the commands, the measurements, the reasoning — is unchanged from the working copy this was built from. 100.64.0.0/10 is left intact on purpose: that is Tailscale's public CGNAT range, identical for every tailnet, and the firewall rules mean nothing without it.

Goal: a part-time local inference box that AskGwinnett's backend can call for background extraction work (extract.js, events-ai-extract.js), falling back to Gemini automatically when the box is off — the same fail-open pattern semantic.js already uses. Windows 11, no Linux.

Build order — the linear path

The guide has grown optional branches; this is the spine. The first build is five sections, in order, skipping everything else:

Step Section You have, when done
1 section 1 BIOS VRAM allocated, visible in Task Manager
2 section 2 Drivers GPU actually usable
3 section 3 Toolchain git/node/python/Claude Code installed
4 section 4 LM Studio Qwen3-30B-A3B answering on localhost:1234
5 section 6 Tailscale that endpoint reachable from your dev machine

Do each section's Done when check before moving on — that's the whole method. Stop after step 5: that's the finished first version, a working private LLM server, one afternoon of which only ~90 minutes is hands-on (the rest is downloads).

Second session — production: section 7 (Cloudflare Tunnel + Access), section 8 (the adapter in bfoster-services), then the section 11 checklist including the local-vs-Gemini quality spot-check.

Later, à la carte, any order: section 5 image models, section 9 IDE assistant, section 10 TTS/voice. Nothing in the spine depends on any of them.

1. BIOS: set the VRAM allocation first

Before installing anything, reboot into BIOS (Del or F7 at startup):

  1. Check MINISFORUM's support page for the X1 Pro-370 and update to the latest BIOS if one's available — memory-controller fixes are common.
  2. Find the GPU/VRAM allocation setting (usually under Advanced → AMD CBS) and set a fixed size. For your 64GB unit running a ~20GB text model plus an ~8GB image model, 32-48GB allocated to GPU leaves plenty of headroom. Don't max it out — Windows and background services need system RAM too.

Done when: after booting into Windows, Task Manager → Performance → GPU shows your chosen amount as Dedicated GPU memory.

2. Install AMD GPU drivers

Download the latest AMD Adrenalin driver for the Ryzen AI 9 HX 370 (Radeon 890M) from amd.com/en/support (auto-detect tool or pick Ryzen AI 9 HX 370). This is what enables Vulkan/ROCm acceleration — without it, models silently fall back to slow CPU-only inference and you won't get an error telling you why.

Done when: Device Manager → Display adapters shows AMD Radeon 890M (not "Microsoft Basic Display Adapter"), and the AMD Adrenalin app launches.

3. Install the support toolchain

Windows 11 ships with winget, so this is a few one-liners from an elevated PowerShell (right-click Start → Terminal (Admin)):

winget install -e Git.Git
winget install -e CoreyButler.NVMforWindows
winget install -e Python.Python.3.12

What each is for:

  • Git — cloning ComfyUI (Section 5) and pulling any repo you need on the box.
  • nvm-windows — the Windows counterpart to nvm (the original nvm is a bash script and doesn't run on Windows; Corey Butler's nvm-windows is the standard equivalent, same commands). Node itself comes through it, never installed directly, so you can switch versions without reinstalling.
  • Python 3.12 — ComfyUI's venv in Section 5.

Then close and reopen the terminal (winget installs don't update the current session's PATH) and install Node LTS:

nvm install lts
nvm list          # note the version it installed
nvm use 22.11.0   # or whatever `nvm list` showed; needs the admin terminal

nvm use needs an elevated terminal — it works by swapping a symlink in a protected location.

Note: the production adapter code (Section 8) runs in bfoster-services, not on this box — Node here is for testing the endpoint and running utility scripts locally.

Claude Code on the box

Install Claude Code so it can help drive the rest of this setup from the box itself — checking drivers, running the verification curls, debugging whatever goes sideways — instead of you relaying error messages from another machine. Use the native installer rather than npm install -g: a global npm package lives inside the active nvm Node version and vanishes when you switch versions.

irm https://claude.ai/install.ps1 | iex

Then, so it has this guide as context, put a copy of this document in a folder and start it there:

mkdir home-ai-box; cd home-ai-box
# drop this guide in as README.md, then:
claude

If you keep your own notes in a private repo, cloning it instead triggers a GitHub sign-in — Git for Windows ships Git Credential Manager, which opens a browser window for it; that's expected, not an error.

Log in with your claude.ai account when prompted. From there you can ask it to walk you through the remaining sections and verify each Done-when check as you go.

If Node got installed directly first (it did here)

This box took the wrong path before the right one: Node was installed straight to C:\Program Files\nodejs, then reinstalled under nvm-windows later. Two things to know if you're repeating that repair:

  • Installing nvm-windows moves Node to C:\nvm4w\nodejs (a symlink to the active version under %LOCALAPPDATA%\nvm). The old directory is left empty.
  • Every already-open shell keeps the stale PATH. node and npx vanish mid-session and work fine in a new terminal, which looks exactly like a broken install and isn't. Open a new terminal before diagnosing anything.

Done when: git --version, node --version, py --version, and claude --version all print versions in a fresh (non-admin) terminal.

As built here: git 2.55.0, Node v24.19.0 (nvm 1.2.2), Python 3.12.10, Claude Code 2.1.241. Note py is not present on this box — Python answers as python from its install directory, which is all ComfyUI's venv needs.

4. Install LM Studio and pull a 30B model

Download LM Studio for Windows from lmstudio.ai — GUI-based, no command line needed.

  1. Open LM Studio → Discover tab → Qwen3-30B-A3B, GGUF format, 4-bit quant (Q4_K_M), 18.63GB. It's the mixture-of-experts pick from the bandwidth math below — 30B-class quality at fluid speed on this box. (Grab the dense Qwen2.5-32B-Instruct too at some point — it's the quality benchmark for the section 11 spot-check — but A3B is the day-one model.)

    Expect the download to be interrupted at least once — the source throttles, and an 18GB pull that starts at 9MB/s can decay to ~1MB/s and time out partway. This is safe. The partial file (downloading_*.gguf.part) survives, and starting the download again resumes from it rather than restarting — from the Downloads panel in the GUI, or by re-running the same lms get command. Just repeat until it completes.

  2. Load the model in the Chat tab. Check the GPU offload slider is set to max (all layers on GPU). Run a test prompt and watch Windows Task Manager's GPU graph — if usage stays flat, it's running on CPU instead and something upstream (driver, BIOS setting) needs fixing.

  3. Go to the Local Server tab and start it. Default endpoint: http://localhost:1234/v1 — OpenAI-compatible, which is what lets the Node.js adapter in Section 8 work with almost no rewriting.

Driving all of this from the command line instead

Everything above is also scriptable, which is the point of installing Claude Code on the box in Section 3 — it can run these itself and read the output, instead of you narrating a GUI to it. LM Studio ships a CLI (lms), but it only registers after the app has been run once, so launch LM Studio normally the first time, then:

lms get "qwen/qwen3-30b-a3b@q4_k_m" -y --gguf   # re-run to resume if it times out
lms load qwen/qwen3-30b-a3b --gpu max -y        # --estimate-only to check VRAM first
lms server start --port 1234
lms ps                                          # what's loaded
lms status                                      # server on/off

The model's API identifier keeps its org prefix: qwen/qwen3-30b-a3b, not qwen3-30b-a3b. That exact string is what /v1/models returns and what LOCAL_LLM_MODEL should be set to in Section 8. Getting it wrong is sneakier than it sounds — see the failure note below.

Two operational facts that matter for a box meant to sit unattended:

  • Closing the LM Studio window kills the server. The GUI and the server share a process tree, so quitting the app takes the endpoint down with it — requests then fail with ECONNREFUSED, and the adapter falls back to Gemini.
  • lms server start doesn't need the GUI. Run from a cold machine it prints Waking up LM Studio service... and starts the endpoint headlessly. That's the right way to bring the box up for unattended work — but note it starts with no model loaded, so pair it with lms load unless you're relying on JIT (and see the cold-start timeout note in Section 8).

Picking models: the bandwidth math

Token rate on this box is bounded by memory bandwidth, not compute: every generated token reads the model's active weights from RAM, and dual-channel DDR5-5600 gives ~90GB/s theoretical (~60-70GB/s realistic). So: tokens/sec ≈ effective bandwidth ÷ bytes read per token. Estimated rates (bandwidth math, not benchmarks — verify with LM Studio's tok/s readout). The A3B row has since been measured on this box at 31.5 tok/s, mid-range of its predicted band, so treat the other rows as trustworthy too:

Model Reads/token Est. rate Character
Qwen2.5-32B (dense, Q4) ~19GB 3-5 t/s max quality, batch-only pace
Qwen2.5-Coder-14B (dense, Q4) ~9GB 7-10 t/s coding-tuned, tolerable
8B-class (dense, Q4) ~4.5GB 13-18 t/s snappy, less capable
Qwen3-30B-A3B (MoE) ~2GB 25-40 t/s (measured: 31.5) 30B-class quality, fluid
gpt-oss-20b (MoE) ~2-3GB 25-35 t/s OpenAI open-weights, fast

The mixture-of-experts (MoE) rows are the trick: all ~30B parameters sit in VRAM, but each token routes through only ~3B of them — the model reads like a 3B and thinks like a 30B. On a bandwidth-limited box that's roughly 10x the speed of a dense 32B at comparable quality. For calibration: ~10 t/s is comfortable reading pace, 20+ feels fluid.

The other half of "fast," measured separately: prefill. The table above is generation speed — how fast words stream once they start. Before that, the model must read the whole prompt (prefill), which is compute-bound, so MoE barely helps it. Short prompts: negligible. Long RAG prompts (thousands of tokens of retrieved chunks): the silent gap before the first word can reach 10+ seconds. When testing, record BOTH numbers LM Studio reports — generation tok/s AND prompt eval tok/s — and judge any latency-sensitive use (chat serving, IDE @codebase queries) on the second one. Batch extraction doesn't care; residents waiting on a chat bubble do.

Two habits that follow:

  • When browsing Discover, check active parameters, not total size — on this box, that one number is your token rate.
  • A3B is the default build; the dense 32B exists as its quality benchmark. If A3B holds up in the Section 11 spot-check (vs the 32B and vs Gemini), it keeps the job and every batch run is ~5x faster than it would have been.

Also worth a look as the ecosystem moves: Gemma 3 27B and Mistral Small 24B (strong dense all-rounders), DeepSeek-R1 distills (step-by-step reasoning, slower). New open-weight generations land every couple of months and supersede rather than patch — each roughly matches the previous generation one size class up, so re-run the comparison occasionally.

Qwen3 thinks by default — and it costs 5x

Qwen3-30B-A3B is a hybrid reasoning model: unless told otherwise it emits a chain of thought before answering, returned in a separate reasoning_content field and billed as reasoning_tokens in the usage block. On a schema-extraction call, measured on this box, that was 360 of 452 completion tokens spent thinking — reasoning was the majority of the work.

Appending /no_think to the system prompt turns it off:

Mode Time Reasoning tokens Valid JSON
default (thinking on) 15.1s 360 yes
/no_think 3.05s 0 yes

Same extraction, 5x faster, still valid JSON. Two caveats worth knowing before you reach for it everywhere:

  • The documented API switch does not work here. Passing chat_template_kwargs: { enable_thinking: false } is silently ignored by LM Studio — in testing it reasoned more, not less. The /no_think prompt token is the only lever that actually works.
  • It costs some judgment. On the same county-notice sample, thinking produced the title "Gwinnett County Board of Commissioners Regular Meeting"; /no_think produced "NOTICE OF PUBLIC MEETING" — the literal page header. Date, time, and venue were correct either way. So the loss shows up in fields needing inference, not transcription.

That tradeoff is exactly the Section 11 spot-check question, so don't settle it from this table — run extraction both ways on real inputs and decide per field. If quality holds without thinking, the extraction path gets 5x cheaper on top of the 10x the MoE architecture already bought.

Measured on this box (2026-08-24, Qwen3-30B-A3B Q4)

Real numbers over Tailscale, replacing the estimates above:

Case Prompt Prefill Generation Time to answer
Casual chat 20 tok 0.5s 31 tok/s 14.7s (thinking)
Grounded RAG 3.5k tok 15.0s 26 tok/s 26.6s
Grounded RAG 5.9k tok 33.4s 26 tok/s 50.1s
Same RAG prompt, repeated 3.5k tok 0.2s 26 tok/s 11.7s

Generation speed matched the estimate (26-31 tok/s). The three things that didn't show up in the estimates:

  1. Prefill runs ~176-234 tok/s, so long prompts are expensive: 15s at 3.5k tokens, 33s at 5.9k. This is the number that decides whether an interactive use is viable, and it confirms live chat serving is not.
  2. Qwen3 is a reasoning model and thinks by default — on a trivial 20-token question it produced ~2,300 characters of internal reasoning before the first visible word, turning a 0.5s prefill into a 14.7s wait. Append /no_think to the user message to disable it (verified: 0 thinking characters, answer begins the moment prefill ends). Note chat_template_kwargs: {enable_thinking: false} was silently ignored — use /no_think.
  3. Prefix caching is dramatic: re-sending an identical prompt dropped prefill from 15.0s to 0.2s. Put the static parts of a prompt (system prompt, stable context) FIRST and the variable part last, so repeated calls reuse the cached prefix.

Tool calling works, but only with thinking on. Qwen3 emits proper OpenAI-style tool_calls through LM Studio. However, asked "what is today's date?" with /no_think, it skipped the tool it had been given and answered "October 25, 2023" — its training cutoff — with complete confidence. With thinking enabled, it called the tool and answered correctly. Reasoning is what makes the model check itself, so any tool-using setup should leave thinking on and pay the ~10-15s. (This is the doctrine's model-agnostic rule in miniature: never rely on the model choosing to call a tool when correctness matters — force it in code.)

Practical readings: batch extraction doesn't care about any of this. Interactive use wants short prompts plus /no_think. Anything resident-facing stays on a cloud model.

Playing with other models won't break anything

The plan is to load and unload all kinds of models on this box — that's fine, and the extraction path is built to survive it:

  • API requests name their model in the model field, and LM Studio just-in-time loads whatever's requested (JIT is on by default). So an extraction call still works even if you've been playing with something else all evening — the named model loads on demand as long as it's still downloaded. Verified: calling an unloaded model returned a correct answer in 2.8s, without evicting the model that was already resident.
  • The failure case is quieter than you'd want. If the name doesn't resolve — model deleted, renamed, typo — LM Studio doesn't error. It answers with whatever model happens to be loaded, and returns 200. So a stale LOCAL_LLM_MODEL doesn't fall back to Gemini; it runs your extraction on some other model entirely, and the output looks fine. The response's model field is the only place the truth shows up — log it (Section 8). The one loud failure is having nothing loaded, which returns a 400.
  • Storage math for the collection habit: 30B-class models run ~19GB each, 14B ~9GB, on a 1TB drive shared with Windows. When it gets tight, prune from LM Studio's My Models tab rather than the filesystem.

Done when: a test prompt streams a reply while Task Manager's GPU graph visibly spikes, and this returns the loaded model's name:

curl http://localhost:1234/v1/models

If you'd rather verify the GPU claim with a number than by eye, sample the compute engine while a generation runs — this should read 90%+, and near zero if it's secretly on CPU:

(Get-Counter '\GPU Engine(*)\Utilization Percentage').CounterSamples |
  Where-Object { $_.CookedValue -gt 1 } |
  Sort-Object CookedValue -Descending | Select-Object -First 3 InstanceName, CookedValue

5. Image models (optional): generation and vision

Two different capabilities live under "image model" — this section covers both:

  • Generation (making pictures): diffusion models via ComfyUI — a separate architecture, runs alongside LM Studio.
  • Vision (understanding pictures): LLMs with eyes — another Discover-tab download served through the same LM Studio endpoint.

5a. Generation with ComfyUI (for the ai.js image-gen routes)

ai.js currently calls gpt-image-1.5 / Imagen for image generation. To self-host that piece too, ComfyUI is the standard runner, works on Windows:

git clone https://github.com/comfyanonymous/ComfyUI
cd ComfyUI
python -m venv venv
.\venv\Scripts\activate
pip install -r requirements.txt
python main.py --listen 0.0.0.0 --port 8188

As built, this box runs a ROCm PyTorch stack instead — torch 2.9.1+rocm7.2.1, HIP 7.2, gfx1150 native, rather than the generic pip install above. It is measurably faster than the estimate this section originally carried. Those install steps were never recorded, which makes this the least reproducible part of the whole guide — see build-log.md. Capture them from whoever ran it.

Getting a checkpoint ComfyUI can actually load

The original version of this section said "download an SDXL checkpoint into models\checkpoints\," which is where the first attempt went wrong and cost 18GB. Be specific:

  • ComfyUI's Load Checkpoint node wants one single-file .safetensors sitting directly in models\checkpoints\.
  • A git clone of the HuggingFace SDXL repo gives you diffusers format instead — weights split across unet/, vae/, text_encoder/, with fp16, fp32 and ONNX copies of each. ComfyUI cannot read it, and says so only by showing an empty dropdown — no error, no log line.
  • Fetch the single file directly, with resume enabled, because large pulls on this box do get throttled:
cd ComfyUI\models\checkpoints
curl -L -C - --retry 5 -o sd_xl_base_1.0.safetensors `
  "https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/resolve/main/sd_xl_base_1.0.safetensors"

That's 6.46GB. ComfyUI picks up a new checkpoint live — no restart needed, though you may need the refresh icon in the menu.

Settings and speed

Set Empty Latent Image to 1024×1024. The 512×512 default is an SD1.5-era value and visibly degrades SDXL output. Steps 20-30 and cfg 7-8 are sane starting points for base SDXL only.

Lightning checkpoints use different settings, and the base ones fry them. RealVisXL_V4.0_Lightning.safetensors (installed 2026-08-29, the RabinAI imagery checkpoint) wants 8 steps, CFG 1.5–2.0, euler, sgm_uniform. Measured on this box at 1024×1024: 19.87s with euler, 34.13s with dpmpp_sde — the two are not interchangeable, and only euler meets the imagery page's ~18s claim.

Measured on this box at 1024×1024, 25 steps, euler:

Run Time
Cold (includes ~7GB checkpoint load) 68.3s
Warm 52.2s (~2s per step)

Better than the 60-90s this section originally estimated, thanks to the ROCm build. Still ~a minute per image, which is why the RabinAI design is async with a one-job-at-a-time guard: two simultaneous visitors put the second one past two minutes.

Running LM Studio's 30B model (17.35 GiB) and SDXL (~7GB) at once is comfortable. Note the two runtimes report VRAM differently — ComfyUI's ROCm layer reports 55GB available where BIOS allocated 48GB, because it counts shared memory too. Neither number is wrong; don't try to reconcile them.

Done when: http://localhost:8188 in a browser shows the ComfyUI graph, and the default workflow renders an image end to end.

5b. Vision model experiment: extracting from flyers

Text-only extraction can't touch county event flyers, scanned notices, or image-based PDFs. A vision model can — and it's the same schema-extraction pattern as events-ai-extract.js, just with an image in the payload.

  1. LM Studio → Discover → Qwen2.5-VL-7B-Instruct, Q4 GGUF (~5GB — small enough to sit alongside everything else).
  2. Warm-up test: load it in the Chat tab, attach any photo, ask for a description.
  3. The real experiment — flyer in, event JSON out, through the same OpenAI-compatible endpoint the adapter already uses:
const b64 = fs.readFileSync('flyer.jpg').toString('base64');
const completion = await c.chat.completions.create({
  model: 'qwen2.5-vl-7b-instruct',
  messages: [{
    role: 'user',
    content: [
      { type: 'text',
        text: `Extract the event as JSON matching: ${JSON.stringify(schema)}` },
      { type: 'image_url',
        image_url: { url: `data:image/jpeg;base64,${b64}` } },
    ],
  }],
  response_format: {
    type: 'json_schema',
    json_schema: { name: 'event', strict: true, schema: eventSchema },
  },
});

(eventSchema here is a real JSON Schema object — { type: 'object', properties: {...}, required: [...] } — not the loose shape hint used in the prompt text. See the note in Section 8 on why json_object doesn't work.)

If the output holds up on a handful of real flyers, wiring it into production is the Section 8 adapter pattern verbatim — try local, fall back on any failure (here the fallback is "skip the flyer," since the Gemini path never handled images in the first place).

Done when: a real event flyer through the snippet above returns JSON with the right date, time, and venue.

6. Networking: don't port-forward — use Tailscale

Since this box isn't always on and you don't want to expose an inference endpoint to the open internet, put it on a private mesh network instead.

Download and install Tailscale for Windows from tailscale.com/download, sign in, and it's live — no router config, no port forwarding.

Do the same on whatever you're calling it from. Tailscale is the right tool for testing from your own dev machine. For the production Heroku leg, use Cloudflare Tunnel instead (Section 7) — Heroku dynos can't easily run a persistent Tailscale client.

One gotcha: LM Studio's server binds to localhost only by default, so other machines — even over Tailscale — get connection refused. In the server settings, enable Serve on Local Network so it listens on all interfaces. (The Cloudflare leg in Section 7 doesn't need this — cloudflared runs on the box itself and reaches localhost directly.)

The bind does not stay put — pass --bind, don't rely on the env var

This is the trap that cost the most time here, because it fails quietly. From the CLI:

lms server start --port 1234 --bind 0.0.0.0

works, but does not persist. LM Studio writes networkInterface: "0.0.0.0" into %USERPROFILE%\.lmstudio\.internal\http-server-config.json, and then a plain lms server start ignores that file and reverts the listener to 127.0.0.1. Verified by restarting: config still said 0.0.0.0, the actual socket came back on localhost.

So after any restart the tailnet endpoint is silently closed while the config still claims it's open — and because the Section 8 adapter fails open, every extraction would quietly run on Gemini instead. Nothing errors. You would not notice.

Set the environment variable instead, which survives restarts and reboots:

[Environment]::SetEnvironmentVariable('LMS_SERVER_HOST','0.0.0.0','User')

Then verify with a stop/start and no --bind flag:

lms server stop; lms server start
Get-NetTCPConnection -LocalPort 1234 -State Listen |
  Select-Object LocalAddress, LocalPort     # must show 0.0.0.0

Windows Firewall needed no change here — installing LM Studio created inbound Allow rules for it on Private and Public, and Tailscale's interface registers as Private. (Those permissive rules are what section 6a later replaces with Tailscale-scoped ones.)

The env var is a backstop, not the mechanism — pass --bind at every call site. An environment variable fails silently in three ways here: a Scheduled Task running as SYSTEM never sees a User-scope variable, any shell opened before you set it doesn't have it, and neither does anything launched from those. Each of those silently rebinds the server to localhost the next time something runs lms server start.

That is not hypothetical: on 2026-08-26 port 1234 went unreachable over Tailscale while ComfyUI on 8188 stayed fine, and it looked exactly like a firewall bug — the rules were audited twice and were correct. The listener had reverted to 127.0.0.1, because a startup script called lms server start without the variable in scope.

lms server start --port 1234 --bind 0.0.0.0

Diagnostic worth internalising: "unreachable" has two causes that look identical from a client — a firewall dropping the packet, and nothing listening on that interface. Check the second one first; it is one command and rules out the more tedious audit:

Get-NetTCPConnection -LocalPort 1234 -State Listen |
  Select-Object LocalAddress, LocalPort

0.0.0.0 means every interface. 127.0.0.1 means only this machine can reach it, no matter how permissive the firewall is.

Also worth knowing: closing the LM Studio window kills the server — the GUI and server share a process tree, and requests then fail ECONNREFUSED. lms server start brings it back headlessly ("Waking up LM Studio service..."), but with no model loaded, so pair it with lms load or accept the cold-start delay on the first call.

Done when: from your dev machine, with the box's Tailscale IP from tailscale status:

curl http://<box-tailscale-ip>:1234/v1/models

returns the model list.

6a. Lock the port to Tailscale only

"Serve on Local Network" is all-or-nothing: it binds every interface, so LM Studio now answers to your whole home network with no authentication — guests, a kid's laptop, any IoT device. Nobody can administer the box that way, but they can consume it, and the inference runtime itself becomes attack surface (LM Studio's real CVEs have been file-parsing bugs).

You don't need LAN access: Tailscale reaches the box from anywhere, including from the couch. So allow the port only from Tailscale's address range (100.64.0.0/10) and block it everywhere else. In an elevated PowerShell on the box:

# 1. See what currently allows these ports (installer/prompt-created rules)
Get-NetFirewallPortFilter |
  Where-Object { $_.LocalPort -in @('1234','8188') } |
  ForEach-Object { $_ | Get-NetFirewallRule } |
  Select-Object DisplayName, Direction, Action, Enabled, Profile

# 2. Disable the permissive ones it lists, by name, e.g.
#    Get-NetFirewallRule -DisplayName "LM Studio*" | Disable-NetFirewallRule

# 3. Allow ONLY the Tailscale range. Windows blocks inbound by default,
#    so anything not matching these is denied without a block rule.
New-NetFirewallRule -DisplayName "LM Studio (Tailscale only)" `
  -Direction Inbound -Protocol TCP -LocalPort 1234 `
  -RemoteAddress 100.64.0.0/10 -Action Allow -Profile Any

New-NetFirewallRule -DisplayName "ComfyUI (Tailscale only)" `
  -Direction Inbound -Protocol TCP -LocalPort 8188 `
  -RemoteAddress 100.64.0.0/10 -Action Allow -Profile Any

# 4. Confirm the default really is deny
Get-NetFirewallProfile | Select-Object Name, DefaultInboundAction

Do not add a blanket -Action Block rule for these ports. Windows evaluates block rules before allow rules, so a blanket block would override the Tailscale allow and cut off remote access too. Default-deny plus a scoped allow is the correct shape.

Windows applies Block rules before Allow rules of the same specificity, so order doesn't matter here — but the allow rule's narrower RemoteAddress is what lets tailnet traffic through.

Done when: the Tailscale-IP curl above still works, and the same request to the box's LAN IP (192.168.x.x, from ipconfig on the box) times out:

curl -m 5 http://<box-lan-ip>:1234/v1/models   # should fail
curl http://<box-tailscale-ip>:1234/v1/models  # should still work

If you later run ComfyUI (8188) or a TTS server (8880) and reach them remotely, give each the same pair of rules.

7. Production leg: Cloudflare Tunnel + Access service token

Cloudflare Tunnel is outbound-only: cloudflared on the box dials out to Cloudflare, and requests to your public hostname ride back down that connection. No inbound ports, no router config, Windows Firewall stays fully closed to the internet.

The catch: LM Studio's local server has no authentication, so the tunnel hostname must not be left open — anyone who finds it could run inference on your box. Cloudflare Access with a service token locks it at Cloudflare's edge; unauthenticated requests never reach your house. This is Cloudflare's standard machine-to-machine service-auth pattern, not a homegrown scheme.

Do these subsections out of order: 7b → 7c → 7a.

Publishing the hostname is what puts an unauthenticated endpoint on the public internet, and the Access application is what closes it. Doing them in written order leaves a gap between the two — and 7c is a multi-screen wizard that is easy to get wrong on the first attempt, so that gap is measured in however long it takes you to get the dashboard right. On this box that was ~40 minutes of open, unauthenticated inference (build-log.md).

Build the lock first. Create the service token (7b) and the Access application (7c) with the hostname typed in before it resolves — Cloudflare accepts that without complaint — and publish the hostname (7a step 3) last. The door is then locked before it exists.

If you find yourself with a published hostname and no policy, Stop-Service cloudflared from an elevated shell takes every public hostname dark in about a second, without touching Cloudflare config or your Tailscale access.

7a. Create the tunnel

Prerequisite: a domain on Cloudflare. A tunnel hostname needs a domain whose nameservers point at Cloudflare — the account and the Zero Trust features used here are free. Use a domain that is currently empty — pointing its nameservers at Cloudflare then breaks nothing. Do not repurpose the domain serving your live site: moving nameservers means re-creating every existing record with production at stake.

Add the domain in the Cloudflare dashboard, change the nameservers at your registrar to the pair Cloudflare shows, and wait for it to go active — usually minutes, occasionally hours.

  1. Cloudflare Zero Trust dashboard → Zero Trust → Networks → Tunnels → Create a tunnel (Cloudflared connector type).
  2. Copy the Windows install command it shows — it installs cloudflared as a Windows service with the tunnel token baked in, so it starts on boot with no logged-in user needed.
  3. Add a Public Hostname: something on a domain you have in Cloudflare (e.g. llm.example.com) → service type HTTP, URL localhost:1234.
  4. (For the planned RabinAI image toy on Rabin Forest) add a second public hostname while you're here — e.g. img.example.com → HTTP, localhost:8188 (ComfyUI) — and include it in the same Access application in 7c. The backend adapter and load safeguards come later; having the hostname ready costs nothing.

7b. Create the service token

  1. Zero Trust → Access → Service Auth → Service Tokens → Create.
  2. Name it after the caller (bfoster-services), pick a duration (default 1 year — put the expiry on a calendar; rotation means generating a new token and swapping two env vars).
  3. Copy the Client ID and Client Secret now — the secret is shown once.

7c. Gate the hostname with an Access application

  1. Zero Trust → Access → Applications → Add → Self-hosted, with the application domain set to the tunnel hostname from 7a.
  2. Add a policy with action Service Auth (not Allow — Allow is for humans with identity logins), and under the include rule pick the service token from 7b.

Now every request must carry the token headers or Cloudflare rejects it at the edge. Verify both halves:

CF_ID='<client-id>'         # ends in .access
CF_SECRET='<client-secret>'

# 1. no headers → 403 from Cloudflare, never touches the box
curl -s -o /dev/null -w "%{http_code}\n" https://llm.example.com/v1/models

# 2. with headers → 200 with LM Studio's model list
curl -s https://llm.example.com/v1/models \
  -H "CF-Access-Client-Id: $CF_ID" -H "CF-Access-Client-Secret: $CF_SECRET"

# 3. right ID, WRONG secret → must also be 403
curl -s -o /dev/null -w "%{http_code}\n" https://llm.example.com/v1/models \
  -H "CF-Access-Client-Id: $CF_ID" -H "CF-Access-Client-Secret: deadbeef"

Run all three. The first two passing only proves the two calls differ somehow; the third is what proves Access is validating the token rather than just noticing that headers exist.

What a failure tells you:

Result Meaning
200 unauthenticated No policy in force — application never saved, action is Allow instead of Service Auth, or its destinations omit this hostname
403 both ways Headers don't match the token — truncated paste, or the policy includes a different one
403 bare, 502 authenticated Access works, routing doesn't — the connector can't reach the origin
530 No connector running

On Heroku, set LOCAL_LLM_URL=https://llm.example.com/v1 plus the two LOCAL_LLM_CF_ACCESS_* vars from Section 8 — the adapter attaches the headers automatically when they're present.

Done when: curl 1 returns 403, curl 2 returns 200 with the model list, and curl 3 returns 403 — the lock, the key, and proof the lock inspects the key.

Two things that surprise people afterwards:

  • The hostname stops working in a browser. A Service Auth policy permits no identity login, by design. Tailscale (http://my-box:1234) stays your human-facing path to the box.
  • cloudflared service install needs an elevated shell, as do Stop-Service / Start-Service cloudflared. Without it you get "Cannot establish a connection to the service control manager: Access is denied."

8. Wiring it into extract.js and events-ai-extract.js

Follow the same fail-open pattern semantic.js already uses: try local, fall back to the existing Gemini path on any error, missing config, or timeout. One small adapter, not a rewrite.

This is not the only adapter, and not the one that ships today.

bfoster-services already calls the box from server/ai.js for the Rabin Forest assistant chat, using RABINAI_* environment variables. That code is built and deployed. The LOCAL_LLM_* adapter below is for AskGwinnett ingest extraction and has not been written — nothing in that repo reads LOCAL_LLM_*.

Two integrations, two prefixes, two purposes. Setting LOCAL_LLM_* config vars does nothing today; the assistant needs RABINAI_URL, RABINAI_CF_ACCESS_ID, RABINAI_CF_ACCESS_SECRET, RABINAI_MODEL and RABINAI_ASSISTANT=on. When you build this section, decide deliberately whether to keep the prefixes separate or converge them — but don't assume one set configures both.

Before you add auth to an endpoint, find every caller. Enabling Cloudflare Access on llm.example.com instantly broke two of the four existing fetches in server/ai.js, which had been sending no headers because none were needed. One of them was the assistant's warm-up ping, whose failure pinned the whole feature to Gemini permanently while logging a single line. Adding authentication is a breaking change for every caller you forgot about — grep for them first (build-log.md has the full account).

Two things about LM Studio's OpenAI compatibility to get right before writing any of this, both verified against LM Studio 0.4.21:

  • response_format: { type: 'json_object' } is rejected, with 'response_format.type' must be 'json_schema' or 'text'. This is the one place its OpenAI compatibility stops short. Use json_schema and pass a real JSON Schema object ({ type: 'object', properties: {...}, required: [...] }) — which is stricter and better anyway, since the server constrains generation to the schema instead of trusting the model to follow a prompt.

    But the schema constrains shape, not meaning. A field typed string will always be a string and always be present; nothing makes it a useful string. The same schema and input returned "date": "2026-10-06" on one run and "date": "October 6, 2026" on another. Keep stating formats in the prompt ("ISO date YYYY-MM-DD, 24-hour HH:MM") — the schema stops malformed JSON, not sloppy values — and normalize dates downstream rather than trusting either.

  • The model field is a suggestion, not a contract. If the name resolves to a downloaded model, LM Studio serves (and JIT-loads) it. If it doesn't resolve, there's no error — the server quietly answers with whatever model is currently loaded. Requesting totally-not-a-model returns a cheerful 200 from Qwen. You only get a 400 ("No models loaded") when nothing is resident at all.

    So the failure mode isn't a broken pipeline, it's a silently wrong one: after an evening of playing with other models, a typo in LOCAL_LLM_MODEL means extraction runs on whatever you left loaded, and the output looks plausible either way. The canonical id here is qwen/qwen3-30b-a3b (org prefix included) — copy it from /v1/models. Then don't trust it: the response's own model field reports what actually served the request, so log it (see the end of this section).

// server/askgwinnett/local-llm.ts
import OpenAI from 'openai';

const LOCAL_LLM_URL = process.env.LOCAL_LLM_URL; // e.g. http://localhost:1234/v1
const LOCAL_LLM_MODEL = process.env.LOCAL_LLM_MODEL || 'qwen/qwen3-30b-a3b'; // exact id from /v1/models — org prefix included
// Qwen3 reasons by default (~5x slower). '/no_think' disables it; see Section 4.
const LOCAL_LLM_NO_THINK = process.env.LOCAL_LLM_NO_THINK === 'true';
// 30s is not enough for a cold call: a JIT model load is ~16-19s on its own,
// and a reasoning-mode extraction then takes ~15s more. See the note below.
const LOCAL_LLM_TIMEOUT_MS = Number(process.env.LOCAL_LLM_TIMEOUT_MS) || 90_000;
// Cloudflare Access service token (Section 7) — unset for Tailscale/local use
const LOCAL_LLM_CF_ACCESS_ID = process.env.LOCAL_LLM_CF_ACCESS_ID;
const LOCAL_LLM_CF_ACCESS_SECRET = process.env.LOCAL_LLM_CF_ACCESS_SECRET;

let client: OpenAI | null = null;
function getLocalClient(): OpenAI | null {
  if (!LOCAL_LLM_URL) return null;
  if (!client) {
    client = new OpenAI({
      baseURL: LOCAL_LLM_URL,
      apiKey: 'not-needed',
      defaultHeaders:
        LOCAL_LLM_CF_ACCESS_ID && LOCAL_LLM_CF_ACCESS_SECRET
          ? {
              'CF-Access-Client-Id': LOCAL_LLM_CF_ACCESS_ID,
              'CF-Access-Client-Secret': LOCAL_LLM_CF_ACCESS_SECRET,
            }
          : undefined,
    });
  }
  return client;
}

/**
 * Attempts extraction against the home box. Returns null on any failure —
 * missing config, timeout, box offline — so callers fall back to Gemini
 * exactly like semantic.js does for embeddings.
 */
export async function tryLocalExtraction(
  prompt: string,
  schema: Record<string, unknown> // a JSON Schema: { type: 'object', properties, required }
): Promise<string | null> {
  const c = getLocalClient();
  if (!c) return null;

  const controller = new AbortController();
  const timer = setTimeout(() => controller.abort(), LOCAL_LLM_TIMEOUT_MS);

  // The schema guarantees shape; the prompt still has to ask for sane values.
  const system =
    'Extract the requested fields as JSON. Use ISO dates (YYYY-MM-DD) and 24-hour times (HH:MM).' +
    (LOCAL_LLM_NO_THINK ? ' /no_think' : '');

  try {
    const completion = await c.chat.completions.create(
      {
        model: LOCAL_LLM_MODEL,
        messages: [
          { role: 'system', content: system },
          { role: 'user', content: prompt },
        ],
        response_format: {
          type: 'json_schema',
          json_schema: { name: 'extraction', strict: true, schema },
        },
      },
      { signal: controller.signal }
    );
    return completion.choices[0]?.message?.content ?? null;
  } catch (err) {
    console.warn('Local extraction unavailable, falling back to Gemini:', (err as Error).message);
    return null;
  } finally {
    clearTimeout(timer);
  }
}

Then in extract.js, the change is additive — try local first, keep the existing Gemini call as the fallback path:

const { tryLocalExtraction } = require('./local-llm');

async function extractMeeting(rawText, schema) {
  const local = await tryLocalExtraction(rawText, schema);
  if (local) return JSON.parse(local);

  // existing Gemini extraction call — unchanged
  return extractMeetingViaGemini(rawText, schema);
}

Set LOCAL_LLM_URL only in your local .env (not on Heroku) until you've decided whether to wire the box into production. This keeps today's Gemini-based pipeline completely untouched — the local box is purely additive, and unplugging it later means deleting one env var.

The first call after the box wakes is the slow one

The 30-second timeout this guide originally suggested is too short in two separate ways.

Cold start. When the named model isn't resident, LM Studio JIT-loads it before generating — ~16-19s for the 30B. Measured cold, the call aborted at 30s; the identical call, warm, returned in 2.7s. In production that reads as a flaky box: the first job after the box wakes silently goes to Gemini, every job after it runs locally.

Reasoning-mode variance. How long thinking takes depends on how hard the model decides the problem is, and it is not stable. The same extraction took 15s on one run and 38s on another, both warm — the second would have blown a 30s budget with the model already loaded. If you leave thinking on, budget for the tail, not the average.

Three ways to deal with it, and they combine:

  • Raise the timeout past a cold start — 90s, as above.
  • Keep the model resident. lms load without a --ttl keeps it loaded indefinitely; JIT-loaded models expire after an hour by default.
  • Warm it deliberately at the start of a batch run — one throwaway request, or lms load — so the first real extraction is already hot.

Fail-open's blind spot: watch the logs, not the output

Failing open means a misconfigured box and a healthy one produce the same extraction results — Gemini's — so nothing in the output tells you the local path never ran. An expired Access token, a box that's off, a tunnel that's down: all identical from the caller's side.

The console.warn above is the only signal, so make it countable rather than decorative. Log both facts you can't otherwise recover — which path served the job, and which model actually did it (remember the model field you sent is not authoritative):

// have tryLocalExtraction return the whole completion, not just content
const local = await tryLocalExtraction(rawText, schema);
console.info(`extract: ${local ? `local/${local.model}` : 'gemini'}`);

Then "did the box do any work last night, and with which model?" is a log grep instead of a guess. Worth having before the first unattended batch run — both a silently idle box and a silently wrong model can otherwise hide for weeks behind perfectly plausible output.

Done when: with the box on, tryLocalExtraction returns valid JSON for a real sample; with the box off (or LOCAL_LLM_URL unset), the same call quietly returns null and the Gemini path produces the result — test both directions before trusting the fallback.

9. Alternate build: coding assistant for your IDE (optional)

Same box, same LM Studio server, same Tailscale leg — the only changes are the model and what's on the other end. Sections 1-3 and 6 are prerequisites.

These builds do not coexist. Corrected 2026-08-28 — an earlier version of this section said both models could be resident at once (~9GB + ~19GB) and that you need not unload anything. That was written against a 48GB carve-out and a 14B coder. It is no longer the plan: the box runs one LLM at a time, plus SDXL — they do not fit resident together.

9a. Load a coding model

Use qwen/qwen3-coder-30b at GGUF Q4_K_M (~18.6GB) — same MoE shape as the chat model, so the same ~31 t/s rather than a dense model's 3-6.

lms get qwen/qwen3-coder-30b@q4_k_m --gguf -y
lms load qwen/qwen3-coder-30b --gpu max -y --context-length 32768

At the 32GB carve-out the arithmetic is 18.6 (coder) + ~7 (SDXL) = ~25.6GB, leaving ~6GB spare. Two 30B models would need 37.3GB and do not fit — that is the whole reason this is one-at-a-time.

Swapping the resident model is a production change, not a local one. bfoster-services asks for RABINAI_MODEL by name, so the moment the coder replaces the chat model, the Heroku config var has to change with it:

RABINAI_MODEL=qwen/qwen3-coder-30b

Get this wrong and nothing errors — LM Studio 404s the unknown model, server/ai.js fails open, and the site serves the cloud model indefinitely with one log line as the only symptom. That is the same failure shape as the missing Access headers on 08-25. Check with:

curl -s https://your-api.example.com/ai/gemini-assistant   -H 'Content-Type: application/json' -H 'Origin: https://your-site.example.com'   -d '{"prompt":"what does Brian do?"}'      # engine= is the whole diagnostic

Also turn off JIT loading (justInTimeModelLoading: false). With it on, a production request naming the other model makes LM Studio load a second copy alongside the resident one — 37GB against a 32GB carve-out, which is how the box hung on 2026-08-28.

9b. Point your IDE at it

Continue.dev (VS Code or JetBrains) is the standard client for local models. Install the extension on your dev machine (not the box) and add the box to its config:

models:
  - name: home-box-coder
    provider: lmstudio
    model: qwen2.5-coder-14b-instruct
    apiBase: http://<box-tailscale-ip>:1234/v1
    roles: [chat, edit]

Cline / Roo Code work too: pick the "OpenAI Compatible" provider, set the base URL to the same address, and enter anything as the API key (LM Studio ignores it).

Skip tab-autocomplete for now — it needs sub-second latency that this box can't deliver over the network. Chat and edit-selection are the usable modes.

9c. Why not Cursor

Cursor routes model calls through its own servers, not your machine, so a Tailscale address is unreachable — you'd have to point it at the public Cloudflare hostname, and Cursor can't send the Access service-token headers, so the only way to make it work is stripping the auth off your tunnel. Don't weaken the lock for it — Cursor's marquee features are restricted to its own models anyway. Use Continue or Cline on this box, and Cursor with its own models.

Expectation check: a Coder-14B is a competent assistant, not a frontier one — good for privacy-sensitive code, offline work, and unlimited usage; not a replacement for Claude Code or Cursor's hosted models on hard problems.

Done when: a Continue chat on your dev machine answers a coding question while the box's GPU graph spikes.

10. TTS: give the box a voice (optional)

Text-to-speech is its own model class — LLMs don't speak, and LM Studio doesn't serve audio — but open TTS models are tiny next to everything else on this box and run faster than real time on CPU alone.

10a. Kokoro — the quick win

Kokoro (~82M params, Apache) is the current open-weight standout: excellent quality at a thousandth the size of your text model. The community server kokoro-fastapi wraps it in an OpenAI-compatible /v1/audio/speech endpoint — the same API shape the rest of this guide already speaks, which is why the backend can treat it as a drop-in for OpenAI TTS rather than a second integration.

Why this one is easy on this hardware. The box's binding constraint everywhere else is memory bandwidth — Qwen3 reads ~2GB per token, which is what caps it at 26-31 tok/s. Kokoro is about a thousandth that size, never approaches the ceiling, and is compute-bound instead, which is the axis with headroom. It runs comfortably faster than real time on CPU; the 890M iGPU is not needed and not worth fighting for.

Install

espeak-ng first — Kokoro phonemises through it, and its absence is the single most common Windows failure (it surfaces as a phonemizer error at first synthesis, not at install):

winget install -e eSpeak-NG.eSpeak-NG

Then the server itself, same venv pattern as ComfyUI in Section 5:

cd $HOME
git clone https://github.com/remsky/Kokoro-FastAPI.git
cd Kokoro-FastAPI
python -m venv venv
.\venv\Scripts\Activate.ps1

# CPU torch EXPLICITLY. The default index pulls a CUDA build — ~2.5GB of
# wheels that cannot run on an AMD iGPU and will silently fall back to
# CPU anyway, if it imports at all.
pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install -r requirements.txt

The repo ships its own start scripts, and their names have changed between releases — check the README for the current CPU entrypoint rather than trusting a name written here. Whatever it is called, it should bind port 8880. First run downloads the model (~350MB).

If the native install fights you, the project also publishes Docker images and that route is more deterministic — at the cost of adding Docker Desktop to a box that does not otherwise need it.

Open the port to Tailscale only

Same rule as everything else here: reachable from your machines, closed to the internet (Section 6).

New-NetFirewallRule -DisplayName "Kokoro TTS (Tailscale only)" `
  -Direction Inbound -Protocol TCP -LocalPort 8880 `
  -RemoteAddress 100.64.0.0/10 -Action Allow -Profile Any

Verify

curl -X POST http://localhost:8880/v1/audio/speech `
  -H "Content-Type: application/json" `
  -d '{"model":"kokoro","input":"RabinAI, online.","voice":"af_heart","response_format":"mp3"}' `
  --output test.mp3

Ask for mp3: it is what the browser gets served and what the backend requests, so testing the format you actually ship avoids discovering a wav/mp3 mismatch later.

Kokoro ships a fixed set of voices (af_*/am_* female/male American, bf_*/bm_* British) — pick one as RabinAI's default.

Wire it to the site

bfoster-services already prefers Kokoro and falls back to OpenAI TTS on any failure, so this is configuration, not code:

KOKORO_URL=http://100.x.y.z:8880/v1     # Tailscale, or the tunnel hostname
KOKORO_VOICE=af_heart

Leave it unset and TTS still works on OpenAI — the box stays genuinely optional, the same way it is for the assistant. For anything reaching it from outside the mesh (Heroku), add a tunnel hostname pointing at localhost:8880 and reuse the same Access token, exactly like ComfyUI in 7a; the backend already sends those headers.

Done when: test.mp3 plays back the sentence, and the Rabin Forest read-aloud button returns X-TTS-Engine: kokoro instead of openai.

Watch for: narration and chat now share the CPU. Chunked read-aloud synthesises while Qwen3 is still generating the answer it is speaking, and this box peaks near four concurrent requests before latency degrades. If answers get slower once TTS is live, that contention is the first thing to measure — not the model.

10b. Mapping your own voice (eventually)

Kokoro cannot clone voices — fixed roster only. Cloning is a different model family, and the current pick is Chatterbox (Resemble AI, MIT-licensed): zero-shot cloning from a short reference clip, no training run needed. (The older alternative, Coqui XTTS-v2, clones well but its license forbids commercial use — Chatterbox's MIT is cleaner.)

The recipe when you get to it:

  1. Record ~10-20 seconds of clean speech. The clip's quality caps the clone's quality, so it's worth five minutes of care:
    • Room beats microphone. A phone mic is fine; echo is not. Record in a soft-furnished room (or facing a closet full of clothes) — never a bathroom or kitchen. If you can hear the room in the recording, so will every clone.
    • Kill the hum. HVAC, fans, fridge, the box itself — background noise gets baked into the voice. Pause the loud stuff.
    • Mic at 6-8 inches, slightly off to the side of your mouth so P's and B's don't pop, and speak at normal volume — no clipping, no whispering.
    • Delivery is what gets cloned, not words. Speak the way you want RabinAI to sound: natural conversational energy, complete sentences, some rise and fall. A monotone read produces a monotone clone. The actual content doesn't matter.
    • One continuous take, no edits, no music, nobody else audible. Save as WAV (or the highest-quality format your recorder offers) — re-record rather than clean up a bad take.
  2. Follow the Chatterbox README to stand it up (Python, same venv habits as ComfyUI) and pass the clip as the reference audio.
  3. Expect slower-than-Kokoro generation on this hardware — cloning models are ~1-2GB and heavier per sentence. Fine for playground use, not for narrating books.

Note: Chatterbox watermarks its audio output (inaudible, detectable) — sensible default for cloned voices, and no obstacle for RabinAI duty.

Done when: the box says something you never recorded, in your voice, and it's only medium-creepy.

10c. ElevenLabs — the honest alternative

Everything above keeps the voice on this hardware. This section does not, and it is here because for one specific job — your voice, good enough that a stranger doesn't wince — the hosted option is still meaningfully ahead of anything self-hostable, and pretending otherwise in a guide you wrote for yourself would be silly.

Read this before spending anything: you cannot buy a portable voice. ElevenLabs (and PlayHT, and Resemble) keep the model on their servers. There is no weights file to download and serve from the box. You are renting a voice, not acquiring an asset — which is the exact opposite of what 10b/10c-local are for. If "runs on my hardware" is the point, stop here and fine-tune Piper instead.

Two clone grades:

  • Instant Voice Clone — a couple of minutes of audio, ready immediately, noticeably better than any local zero-shot option.
  • Professional Voice Clone — closer to 30 minutes to a few hours of audio, trained over hours, and the one that actually sounds like you. Gated behind a paid tier and an identity-verification recording, which is a reasonable thing for them to require and a real step to plan for.

Either way the recording rules in 10b are unchanged and still the highest-leverage part. A hosted model does not fix a bad room.

The shape that matters: do NOT synthesise live per answer

The instinct is to slot ElevenLabs in as another engine ahead of OpenAI in /readaloud. Do the arithmetic first. Billing is per character, and a typical assistant answer is ~1,000 characters — so a 100k-character monthly allowance is roughly 100 answers. One curious visitor clicking through a conversation can eat a day's worth. On a public portfolio that is not a cost risk, it is a certainty.

Two shapes fix it, and they compose:

  1. Pre-render the set pieces. The intro, the project blurbs, the greeting — a fixed list, synthesised once, committed as static MP3s and served from Vercel. Zero runtime cost, instant playback, and you can re-roll a take until it sounds right, which is where a good voice earns its money anyway.

  2. Cache by answer hash. For arbitrary answers, key the audio on a hash of the text and store the MP3. The first person to ask a question pays for it; everyone after gets it free and instantly. The assistant's answers to common questions repeat far more than they vary, so the hit rate is high — this is the same trick as AskGWINnett's first-turn answer cache, applied to audio.

With both in place, ElevenLabs is affordable. With neither, it is a metered API on a public button, which is the one configuration to avoid.

Wiring

The backend already picks an engine and falls back, so this is a third branch rather than a new integration: prefer a cached/pre-rendered clip, then ElevenLabs when configured, then Kokoro, then OpenAI. The client is unchanged — it asks for audio and plays what comes back.

ELEVENLABS_API_KEY=...
ELEVENLABS_VOICE_ID=...        # the cloned voice
ELEVENLABS_MODEL=...           # a low-latency model for live use

Leave them unset and nothing changes.

Done when: the pre-rendered intro plays in your voice with no network call at all, and a fresh answer is spoken once and served from cache the second time.

The honest summary of section 10: Kokoro for live narration because it is fast and local; a fine-tuned Piper voice if owning the asset matters more than polish; ElevenLabs if polish matters more than owning it. Those are three different goals and there is no configuration that serves all three.

11. Quick sanity checklist before first real extraction run

  • BIOS updated, VRAM allocation set (32-48GB)
  • AMD Adrenalin driver installed
  • Toolchain in place: git --version, node --version, py --version, claude --version all answer (Section 3)
  • LM Studio loads the 30B model with GPU offload confirmed in Task Manager (not silently running on CPU)
  • Local server running at http://localhost:1234/v1
  • Tailscale connects from your dev machine to the box
  • Port 1234 locked to Tailscale: LAN-IP curl fails, Tailscale-IP curl works (Section 6a)
  • (Production leg) tunnel hostname returns 403 without the Access token headers and 200 with them — both curls from Section 7c
  • LOCAL_LLM_MODEL matches /v1/models exactly, org prefix and all (qwen/qwen3-30b-a3b) — an unresolvable name doesn't error, it silently runs on whatever model is loaded
  • Timeout is long enough for a cold JIT load (~16-19s) plus generation — 30s is not; 90s is (Section 8)
  • LOCAL_LLM_URL set locally, tryLocalExtraction returns valid JSON on a real meeting-minutes sample
  • Logging records both the path taken and the model the response says served it, so an idle box and a wrong model are both visible (Section 8)
  • Run one extraction both ways (local vs. Gemini) on the same input and spot-check the output quality before trusting it at scale
  • Run that same comparison a third way — local with /no_think — and decide whether the 5x speedup costs you any field quality (Section 4)