Every morning at 7:45 my phone buzzes. It’s a short message: the weather, the two emails that actually deserve my attention (out of twenty that don’t), what’s on my calendar for the next three days, and a gentle list of things I promised myself I’d do. Nobody typed it. A Mac mini from 2012, humming quietly in my living room, did.
It’s not the smartest computer in the house. It’s not even the second smartest. But it’s the one that knows what I’m up to.
This series is about how I built it, and mostly about the things I got wrong along the way. There are quite a few.
The itch
I was already using AI chat apps every day, and they were great, right up until I closed the tab. Then:
- They forgot me. Every conversation started from scratch, like a very smart goldfish.
- They waited for me. They never pinged me first, and I couldn’t ask them something from my wrist while carrying groceries.
- They couldn’t touch anything. Not my documents, not my calendar, not the TV I was trying to convince to play the next episode.
To be fair, the big labs are working on exactly this. Personal assistants with memory, voice and a friendly face are clearly where things are heading, and products like OpenAI’s Dots and Meta’s Muse look genuinely good. If you want something that just works, they’re a perfectly reasonable answer.
I wanted something slightly different, and slightly more stubborn.
What I actually wanted
Two things, really:
- It had to run on the subscriptions I already pay for. No API keys and no surprise invoices: just my regular Claude and ChatGPT plans. That turned out to be a fun constraint, because it makes every design choice a budget choice too.
- I wanted to be in control. With a setup that lives on my own machine, I decide which tools the assistant can use, which actions need my confirmation, how its memory is organised, and when it gets a fresh start. If something annoys me, I change it that evening instead of waiting for a product update.
So the brain is Claude Code, Anthropic’s coding agent, which turns out to make a surprisingly good personal assistant when you give it the right tools. Codex, OpenAI’s equivalent, does the errands: web searches, reading photos, generating the occasional image.
A word on privacy, because I don’t want to oversell it: this is not a “my data never leaves the house” project. The models run in the cloud, so plenty of what I ask goes to Anthropic and OpenAI anyway. What stays with me is the structure: the memory files, the tools, the rules and the logs. That’s less romantic than “fully private”, but it’s honest.
What it does today
After roughly two weeks of evenings (and a few late nights I won’t admit to), the assistant:
- keeps a memory of my projects, people and preferences in plain text files, and writes itself a little diary every night before its session is reset;
- talks to me on Telegram (text, voice notes, photos), from a button on my watch, and in a hands-free voice chat on my phone;
- searches my documents, plays things on the TV, and knows where a show is streaming;
- runs small scheduled jobs: the morning brief, reminders, and a news monitor for a couple of topics I care about.
What I got wrong first
My first plan was, let’s say, ambitious. A custom voice gateway. A web app. Then a native iPhone and Watch app, with an animated bear as the mascot (yes, really). I spent days drawing architecture diagrams.
Then one afternoon I plugged in an off-the-shelf Telegram channel and a phone shortcut, and they did about 90% of what I wanted. The bear is still waiting for his app.
Lesson learned: build the cheapest interface that lets you use the thing every day, then let real life tell you what’s missing. Almost everything I changed later came from living with the assistant, not from planning it.
What I’d reuse
- Memory as plain files. If you can open it in a text editor, you can trust it, fix it, and back it up.
- Reset the chat, keep the memory. A nightly reset keeps the assistant quick and cheap; the diary carries the continuity.
- Use the cheapest model that does the job. Prices, reminders and weather need no AI at all, just a script. Searches go to a lighter model, and the expensive one only handles what truly needs it.
Next up
In the next post I’ll open the box: one small server, a coding agent with a day job as a personal assistant, and the duct tape that connects it to the rest of my life.