Personal project
Heartbeat
A local-first system that turns personal data scattered across devices and services into a private, queryable record.
Most of my personal data already existed, but in fragments: health records on my phone, reading activity, code across two Macs, location data in photos, energy use and activity from online services.
I built Heartbeat to bring those fragments together without handing the resulting dataset to another platform. It runs continuously on an always-on Mac Mini, collecting and normalising data into a private archive I control.
One system, many sources
Heartbeat handles REST and GraphQL APIs, iOS Shortcuts, iCloud transfers, local databases, git history and photo metadata stored on a NAS. Each source has its own schedule, format and failure modes.
A small Python orchestrator checks YAML task definitions every minute, runs eligible collectors and writes structured logs. Tasks can run at a set time, on selected days, after an interval or when a file appears. New collectors are added through configuration rather than scheduler changes.
The system currently coordinates 19 pipelines across health, location, photos, reading, coding, calendars, energy, weather and media activity, with new sources added regularly.
A narrow security boundary
Some useful inputs sit behind macOS privacy controls. Instead of giving the Python runtime broad file access, I built a small Swift bridge with one job: move approved files from protected iCloud locations into Heartbeat’s input directory.
This keeps the privileged component narrow and auditable while the rest of the system runs with ordinary permissions.
Reliable without managed infrastructure
Heartbeat is stateless between runs. Scheduling history comes from its logs, task definitions are reloaded on every tick, and collectors use idempotent updates where data may arrive more than once.
Failures remain isolated to individual tasks. Scripts have timeouts, long-running jobs use locks, and daily logs make problems traceable. The scheduler and ingestion logic are covered by 250 automated tests.
A useful, portable archive
Validated records are stored as ordinary JSON, organised by source and date. They can be inspected directly or queried through a local DuckDB interface without an import step. Selected read-only views are available over a private Tailscale connection.
The archive includes GPS metadata from more than 78,000 photos alongside continuously updated activity from current sources.
Outcome
Heartbeat has become a quiet piece of personal infrastructure. It documents what ran and why, and provides a durable foundation for exploring patterns across work, health, movement, reading and media.
Individual services still provide the raw signals, but the connections between them now live in a system I designed and control.