nest
A multi-platform AI bot app I wrote — one background process runs three QQ group-chat bots, each with its own persona. Its project Q&A was designed from eval results — a local reranker decides whether the docs cover a question, an LLM picks which sections to read — taking correct answers on real questions from 79% to 98%.
One process, three group-chat bots with three personalities
nest is a multi-platform AI bot app I wrote; for now it connects to QQ. A single background process runs three bots around the clock on a Mac mini. Each bot is a QQ account plus a character, with its own name, persona and model — Big Fat Whale on DeepSeek, Rainbow Pony on Grok, Little Dragon Girl on ChatGPT. They decide when to join a conversation, answer serious questions properly, send the stickers the group has collected, pick up the group’s slang, and watch feeds and web pages, pushing changes rewritten in their own voice. A web console shows every message — what each bot replied, and why it stayed quiet when it didn’t.
10 pages: live messages with the reasoning behind each decision, personas and models, usage, stickers, slang, monitors, project docs, QQ containers and storage; remote access over Tailscale with a password.
Lets a bot answer strictly from a piece of software's help docs, and hand off what it can't answer to the person who maintains them.
OpenAI-compatible Chat Completions, OpenAI Responses, and ChatGPT; five failures in a row pause a model for 60 seconds.
Always answer when addressed, chime in now and then, take turns between bots; first sort small talk from serious questions, questions about past chat and questions about a project, then decide how to answer.
Events from every account flow into one queue, so SQLite has a single writer; disconnects reconnect with exponential backoff and backfill the messages missed in between.
Principles before code — each one names what it guards against
| Principle | What it guards against |
|---|---|
| One process runs every account, each with its own context | Every extra process costs its own memory; coordination such as bots taking turns is only possible inside one process |
| The background service and the UI are separate | The UI costs nothing until you open it; only the service stays resident |
| Event-driven, no polling | CPU ≈ 0 when nobody is talking |
| Reconnect automatically and backfill missed messages | Nothing said during a few minutes offline is lost |
| Every "didn't reply" is logged with a reason, visible in the console | No guessing why a bot went quiet; uncertain sends are neither counted as success nor blindly retried |
| One database, a single writer, versioned migrations, scheduled clean-up | Concurrent writes corrupting data; log tables growing forever |
| Black-box tests from day one | Fixing one thing and breaking another; all tests offline, no real QQ, no real model calls |
Going live never took the bots offline: nest first connected to the same accounts in watch-only mode, running alongside the previous bots so the console could compare what it would have said; then the bots were switched over one at a time. From the first line of code to all three bots live took two days; together they use about 80MB of memory, with idle CPU near zero.
Chiming in like a group member: when to speak, how many messages, how fast
- Messages arrive Wait for 2 quiet seconds, at most 8
- Should it reply? Always when addressed; otherwise only after 5 new messages, 120 seconds, or a question to the whole group
- Whose turn? Bots chat among themselves for at most 4 rounds, then wait for a human; one bot chimes in per group per 30 seconds
- Sort the question No question mark means small talk, at no model cost; the rest is a serious question, past chat, or a project question
- Generate Can search the web, look up past chat or the project docs; may attach a sticker
- Send in pieces Up to 3 messages of at most 80 characters, paced like typing
Everything else follows one rule: whatever a model produces is checked by code. Only stickers are collected, never members’ ordinary photos; they’re de-duplicated by content and described by a vision model, and a reply can pick one from the candidates. Slang is proposed by a model and kept only if code confirms each word really appeared at least twice. When a screenshot is transcribed, long strings that look like keys are masked first. Feed, web-page and price monitors only record a baseline on their first check, without pushing. Each bot and each group can run in dry-run mode first, recording what it would have sent.
Answer only from the docs — and when they don't say, say so and pass it on
Little Dragon Girl looks after the player group for CicaTavern, another of my projects. People ask “is there a feature for this, and where do I turn it on?” or “what does this error mean?”. She used to answer from impression and often invented UI paths that don’t exist — and a wrong path sends people hunting for a button that isn’t there, which is worse than no answer. So she was given CicaTavern’s help docs: 405 sections, about 150K tokens, far too much for the prompt; and the retrieval models had to share a 16GB Mac mini with three QQ clients.
- Read screenshots A vision model transcribes error screenshots first
- Three-way recall BM25 keywords · bge-m3 vectors · exact error text, fused by RRF into 30 sections
- bge rerank Top score below 0.17 = not in the docs; hand off to the author
- Model picks sections The answering model picks up to 6 of the 30; on error, fall back to the reranker's top 6
- Answer from the docs Quote UI paths as written, say "not sure" for the rest; answer every part of a multi-part question
- Hand off Message the docs maintainer privately; their answer is searchable right away
Measure every design choice before changing it
The eval set has 298 questions: 43 real ones (asked in the group within a week about CicaTavern itself, each hand-labelled with the sections that answer it), 22 no-answer questions (about features the docs never mention — checked first that the words appear nowhere in them), and 233 synthetic questions written from the docs in the group’s own voice, in four styles: common phrasing, describing the need, describing the symptom, and pasting the error. Retrieval is scored on whether the right section comes first and on how many no-answer questions get rejected. For answers, Little Dragon Girl replies with her production model and prompts, and a model from a different vendor, Claude Opus 5.5, does the grading so no model grades itself.
| Approach | Real questions: right section first | No-answer questions: judged absent | Runs |
|---|---|---|---|
| A Keywords only (BM25) | 54% | 45% | Local |
| B Vectors only (bge-m3) | 70% | 59% | Local |
| C Keywords + vectors, fused | 67% | 59% | Local |
| D Fused → bge rerank, with context headers | 58% | 73% | Local |
| E Fused → bge rerank (shipped) | 72% | 91% | Local, 0.85 s |
| F LLM picks from the full table of contents | 93% | 55% | ~16K tokens per question |
| H LLM ranks E's candidates | 95% | 59% | ~3.7K tokens per question |
In the first answer eval, 9 real questions weren’t answered well. Checking what she had been given, 5 of them never received the right section — exactly the 2 wrong answers and 3 of the partial ones. The problem wasn’t her answering; it was which sections she got. Before changing anything, five strategies were simulated offline from the rankings already recorded in round 1, without a single model call; that showed no score threshold could rescue right sections ranked 8th and 9th. So every project question now has the answering model pick its sections, while bge still decides whether the docs cover it.
Engineering highlights
Eight days of solo work, from the first line of code to project Q&A backed by evals, came down to these problems.
One process, one writer
One OneBot adapter per QQ account, every event into one queue and one dispatcher, so SQLite has a single writer; reconnects back off exponentially and backfill missed messages, and uncertain sends are never blindly retried. Coordinating bots — taking turns, for instance — is only possible inside one process.
Retrieval stays on the Mac
bge-m3 embeddings and the bge-reranker-v2-m3 run under llama.cpp on the Mac mini (Q8, Metal, ~1.5GB for both): questions never leave the machine and need no API keys; a few hundred sections are compared in memory with no vector database; if a model service goes down, retrieval degrades instead of failing.
Offline tests and layered review
609 tests, all offline — a fake OneBot and fake models, no real QQ, no real model calls. Before the console shipped, three reviewers critiqued the design and the code was reviewed from six angles; every issue was reproduced by an independent verifier before being fixed, and all 21 confirmed issues were fixed and re-checked.
Always on, light on memory
nest and both model services run under launchd and start at boot; QQ runs one NapCat container per account, cutting each account's client from ~800MB to ~260MB, with a staggered restart every Monday morning; storage is cleaned daily and the database compacted.