Architecture case study
Luke, a Family Assistant
Luke is the self-hosted family assistant I built and run for my household. Its knowledge, documents and skills stay useful when the model, chat app or code around them changes.
TypeScript / Node.js / provider-neutral model runtime / SQLite working state / channel adapters / Markdown and versioned JSON / permission-gated capabilities
Overview
I built this because useful family context was scattered across chats, calendars, email and files. Repeating it to a generic assistant did not create durable knowledge or a system the household could inspect and own.
The project started as a conversational tool runner and gradually became a local-first family assistant. One installation serves the household, but each person keeps a stable identity and attributed actions. I wanted approved knowledge, documents and skills to be shareable without pretending that a messaging account is the person or the source of truth.
I treat channels and model providers as adapters around the same runtime. The current implementation supports a hosted LLM through an agent SDK and a generic tool loop for compatible chat-completions APIs, both behind the same permission and session boundaries. I do not silently fall back between providers. If an adapter cannot read a PDF, it refuses the request before sending anything to the model.
Requirements
- Run on hardware I control, with private household data outside the Git repository.
- Serve one household with stable, attributed members and shared approved information.
- Work with hosted and local models through explicit adapters, with no silent provider switch.
- Let code, not the prompt, decide which tools may run, and keep an auditable record of those decisions.
- Accept messages durably across CLI, Telegram and email, and recover unfinished work after model limits, restarts or failures.
- Keep permanent knowledge and original documents human-readable; treat indexes and dashboards as replaceable.
- Let the family propose and improve skills through conversation, with checks, approval and rollback.
Architecture
Every request gets context, crosses a permission boundary and leaves a record.
I only acknowledge work after it has been persisted. From there it can be retried, resumed after a restart and traced back to the member, conversation and source that created it. The interaction contract keeps those facts stable without pushing channel-specific behaviour into the agent runtime.
The model sees only enabled capabilities. Every tool passes through scope and risk checks plus optional human confirmation; unknown tools fail closed. File operations also enforce lexical and real-path confinement. I keep permanent information in Markdown, versioned JSON and original files, while databases hold working state that I can rebuild.
Key components
Interaction contract
Every channel supplies the same member, source, conversation, attachment and capability context to the runtime.
Family identity
External channel identities map to stable household members. Conversations stay separate and actions remain attributable.
Model providers
Provider adapters own the complete tool-using turn. Fallback is limited to configured routes; it does not authorise an arbitrary provider change or expand tool permissions.
Permission engine
src/permissions.ts evaluates tool metadata, asks for human confirmation when required and logs allow/deny decisions.
Safe file layer
src/files.ts blocks traversal and symlink escapes. Dedicated handlers then narrow writes to their own notes, knowledge, document, memory or skill locations.
Durable intake
src/durable-jobs.ts persists Telegram messages, PDFs and email before processing. It deduplicates message IDs, recovers interrupted jobs and applies bounded retry with subscription-reset awareness.
Capability catalogue
Capabilities group tools with the settings, storage, workers and health checks they require. Configuration can disable them; only the permission gate can authorise them.
Versioned skills
Conversational changes create inspectable proposals. Checks and human approval activate an exact immutable version; rollback preserves history.
Document pipeline
Original evidence is retained alongside a validated, versioned JSON record with provenance, ambiguities and correction history. Integrations consume that portable boundary.
Telegram state
src/telegram-state.ts persists sessions and reminders, serialises each conversation and lets confirmation replies unblock an active tool call. Shared scheduled work executes once before delivery to multiple recipients.
Portable storage
Markdown, original files and versioned JSON hold permanent information. Search indexes and dashboard views are replaceable projections. Lifecycle-bearing runtime records require their own verified backups and exports.
Configuration boundary
A validated, secret-free configuration describes household identity, providers, storage and enabled capabilities. Migration and activation are explicit.
Recovery
SQLite stores runtime and lifecycle state. Versioned, checksummed JSON exports provide a portable recovery route, but original documents alone do not replace backups of operational records. Restore must be verified against a new empty database.
Implementation detail
I did not want “provider-neutral” to mean a thin text-completion interface that stopped being useful as soon as the model called a tool. The abstraction covers the complete agent turn, including tool continuation and declared capabilities:
export interface AgentRuntime {
readonly providerId: string;
execute(request: AgentRequest): Promise<AgentResult>;
capabilities(): ModelCapabilities;
}
if (attachments.some(isPdf)) {
throw new Error(`${providerId} cannot read PDF documents.`);
}
The generic tool-loop runtime converts each tool's Zod schema to JSON Schema, validates returned arguments before calling the handler, limits the number of turns, aborts on timeout and refuses unavailable tools. I keep provider sessions behind application-owned IDs, so channel records do not inherit a provider's identity model.
For documents, I use content-addressed extraction revisions. A revision ID hashes the original bytes, extractor identity, extractor version and normalised payload. Approval re-hashes the source, rejects unresolved ambiguity and supersedes the previous approved revision:
revisionId = sha256({
sourceSha256,
extractor,
extractorVersion,
facts,
references,
amounts,
dates,
ambiguities
});
approve only if currentSourceHash === sourceSha256
&& ambiguities.length === 0;
I use the same pattern for skills. Proposed Markdown becomes an immutable semantic version with its content hash, requested capabilities, diff and deterministic include/exclude assertions. Activation revalidates the exact bytes and records who approved them. A skill can request an existing capability, but it cannot grant itself one.
Decisions that mattered
- Safety lives in code. Permissions, identity, limits and replay protection live in code. Early versions used phrase lists to decide what a message meant. They broke on Polish and on ordinary replies, so meaning moved to the model and safety stayed in code.
- Every tool call is checked before it runs. Capability switches narrow the tools, channel policy narrows them again, and risk metadata, confirmation and audit are the final check.
- Fetched content is evidence, never authority. Email and web pages are untrusted input. Every fetch goes through one function that checks the grant again on each redirect.
- Fall back only when the provider fails. Luke switches model provider on quota, authentication, server or network errors. It does not switch on its own timeouts, validation errors or refusals, because another provider would not fix those.
- Acknowledge only what is saved. A message is accepted only after it is stored. Each conversation is processed in order, and recoverable failures retry without asking anyone to resend.
- Knowledge is not workflow state. Readable Markdown holds narrative context; SQLite holds things with a status or a due date. Facts in both carry their source.
Trade-offs
| Decision | Trade-off | Why |
|---|---|---|
| Provider adapters behind one complete-turn contract | More adapter code, consistent tools and permissions across hosted and local models | I keep provider-specific sessions and response formats outside channels and durable household records. |
| Markdown knowledge before vector search | Less fuzzy retrieval, better debuggability | For early usage, grep-able source-of-truth files were more reliable than opaque embeddings. |
| One household, attributed members | More identity and visibility rules, less duplicate context | Approved information can be shared without losing who requested or changed something. |
| Risk-based confirmation rather than confirmation for every write | Routine automation stays usable, sensitive actions still pause | Scope describes where a tool acts; independently assigned risk determines whether a human must approve it. |
| Markdown plus SQLite | Two storage models to maintain, clearer semantics and queries | Knowledge, evidence and operational lifecycle have different access patterns and should not be forced into one format. |
| Authenticated chat for action approval | Monitored email is observational rather than autonomous | Untrusted message content can propose an action but cannot directly modify state or contact an external recipient. |
| Runtime database with portable snapshots | More export and restore machinery, no database becomes the only copy | Operational queries stay fast while checksummed JSON preserves a provider-neutral recovery path. |
Where it runs
Luke runs in its own container on a small home server. Releases are versioned, so a bad deploy can go back to the previous working version.
The house keeps working without the internet. Anything important keeps a physical fallback, because a bad deployment here can turn off a light somebody is using.
Admin access to the server is time-boxed. Coding agents can always inspect it read-only, but root needs a tap from me, and agents cannot change their own access rules.
Local control is the main path. Remote access is optional.
Where it is now
Between June and October 2026 the project went through more than 1,000 commits, with a dated decision log of about 2,900 lines.
- CLI, private Telegram and allow-listed email intake, with durable session resume and request queuing.
- Stable member identities across channels, attachments and scheduled work.
- Permission gate, safe file layer and append-only audit log in code, with security and reliability tests around them.
- Calendar, email drafts, reminders, accounting intake and document tools behind the same permission path.
- Skills that can be proposed, checked, approved, versioned and rolled back without changing code-level permissions.
Lessons
- A prompt is not a security boundary. Permission checks have to live in code.
- Folder structure and readable household context solved more of the memory problem than vector search would have.
- Audit logs and session recovery were needed much earlier than I expected.
- Testing the wording of skills cost more than it caught. Versioned skills, review of the exact diff, checking the next real run and one-command rollback worked better.
What I would build next
The current installation is still the proving environment. The interaction contract, stable member identity, provider adapters, explicit migrations, portable restore path and versioned skill lifecycle are now in place. I still want capability compatibility to fail during configuration rather than relying on adapter-level rejection at execution time.
After that, I would focus on packaging and access: run the same application and durable data through Docker, then validate local models and ARM64 hardware. I do not want any of those delivery formats to create a second data model or weaken the household boundary.