# DESIGN.md — diwan Audience: whoever (human or agent) picks up **development of diwan itself** next. If you're an agent dispatched *by* diwan to triage notes or do delegated work, you want `PROTOCOL.md` (given to you directly in your dispatch prompt) and the content repo's own `AGENTS.md` instead — this doc is one level up, about the tool, not about using the tool. ## What this is diwan is an AI layer over a plain org-mode "second brain" (notes, tasks, calendar, projects — normally edited by hand in Emacs/beorg, synced via Nextcloud). It periodically triages a capture inbox, keeps files tidy, and runs delegated research/work when asked, while treating the org files as the only real source of truth. **This repo is diwan the tool, and nothing else.** It contains the CLI and the generic procedure docs (`PROTOCOL.md`, `TICK.md`, `DO.md`) — no personal notes, no content-repo-specific convention docs. It's meant to be installed independently (on `PATH`, eventually via its own nix flake) and pointed at *any* notes repo. Marc's actual notes live in `~/nc/org-agenda` (the real, continuously-synced data) and, for development/testing, in a throwaway sibling repo `diwan-dev-notes` (a git-tracked fixture copy, safe to break). Neither of those repos contains any of diwan's own code or docs — see "Known gaps" for exactly how far that packaging currently goes. Status as of this writing: the core loop (tick → triage/dispatch, do → delegated work, both via real herdr sessions) is built and has been exercised end-to-end against live `herdr`. The tool/content split (this repo vs. `diwan-dev-notes`) happened 2026-11-09, and the harness was narrowed to `pi` exclusively right after that (dropping Claude Code support entirely) — see "Known gaps" for what both of those still owe a fresh live pass on. ## Principles (why things are shaped this way) These were distilled from how Marc already works and what he explicitly asked for; they're the standard to hold any future change to: 1. **Org files are ground truth, not a cache.** If diwan vanished, the notes still work in Emacs/beorg. Nothing diwan needs lives outside `.org` files except the content repo's own `.diwan/state.json`, which is pure bookkeeping, not content. 2. **Act first, review passively.** No per-edit approval gate — every action is reported and trivially revertable instead. 3. **Mechanism over magic.** Small, inspectable, deterministic steps over one opaque "agent did something" blob — Marc wants to be able to understand a failure well enough to jump in personally. 4. **Grow by accretion.** Ship the smallest loop that works; add complexity only once the simpler version has been lived with. (Concretely: v0 has no custom CLI-level org-tag manipulation, no scheduler, no kill-on-timeout logic — these are deliberately not yet built.) 5. **One harness, chosen deliberately: `pi`.** diwan dispatches `pi` and only `pi` — not a pluggable choice. Claude Code was the original harness and worked, but needed `--permission-mode auto` to avoid prompting for approval on every tool call, and blocks on an interactive first-run trust dialog that herdr's real-pane dispatch can't answer (hit live 2026-11-09 against the fresh `diwan-dev-notes` content repo). `pi` has no per-tool-call approval gate and no such trust dialog (only a one-time, non-interactive-safe project-trust decision for protected `.pi/*` resources, handled by `--approve`) — it was built to run unattended from the start, which is exactly diwan's own model. The procedure docs (`TICK.md`/`DO.md`) still don't hardcode anything pi-specific into their wording, as a matter of good practice, but there's no expectation of swapping harnesses again. 6. **Two symmetric channels, one substrate.** Talk to diwan via the notes (`:chat:`) or via the live herdr session — both write to the same git-tracked truth. 7. **Mobile stays dumb.** Nothing new to install on the phone. 8. **Tool and content are separate repos.** diwan the tool (this repo) ships independent of any one person's notes; a content repo (e.g. `diwan-dev-notes`, or Marc's real `~/nc/org-agenda`) only ever contains notes, `AGENTS.md`, `diwan.org`, and diwan's own tiny `.diwan/` runtime-bookkeeping directory (`state.json`, `.lock`) — never the CLI or the procedure docs. This is what makes the content repo distributable (synced via Nextcloud to a server, edited from beorg) without dragging a whole dev tool along with it, and what keeps diwan's own generic contract reviewable/versionable independent of any one person's data. ## Architecture at a glance Three layers, each independently simple: - **Git**, in the *content* repo, no remote (for Marc's prototype content repo — a real deployment may add one, e.g. for Nextcloud sync, but that's the content repo's business, not this one's). Every automated write to the content repo is a commit. This *is* the revert mechanism (principle #2) — not a bespoke undo log. Commits are authored as `diwan ` when diwan itself is the actor, or the user's own git identity when `diwan tick` snapshots whatever manual edits were lying around at the start of a run (so no tick ever operates on a dirty/mixed workspace, and "who changed this" is answerable from `git log --author` alone). This repo (the tool) has its own, unrelated git history — just ordinary source control for diwan's own development. - **herdr**, a terminal-multiplexer-for-agents with a JSON-over-socket CLI (`herdr workspace/tab/pane/agent ...`). diwan doesn't invent a session store: one herdr workspace labeled `diwan` holds every tick run and every open `do` session as a separate, independently addressable, persistent tab. See "herdr integration" below for the exact calls. - **The `diwan` CLI** (`./diwan` at this repo's root), a single dependency-free Python script (shebang runs it through `nix shell nixpkgs#python3`, so no system Python install needed on NixOS; a real flake package would wrap this properly instead). It does only the mechanical, deterministic parts: git bookkeeping, enough org parsing to read/write a `DIWAN_ID` property and insert a `Close diwan session` checkbox item, and herdr dispatch. It never triages, writes prose, or makes judgment calls — that's the dispatched agent's job, guided by `TICK.md`/`DO.md`. It distinguishes two roots explicitly (see the script's own docstring): `DIWAN_DIR` (where it and its docs are installed — `Path(__file__).resolve().parent`, overridable via the `DIWAN_DIR` env var for whatever a future flake's install layout needs) and `DIWAN_NOTES_DIR` (whatever notes repo it's invoked against, found via `git rev-parse --show-toplevel` from the current directory by default, overridable via the `DIWAN_NOTES_DIR` env var for e.g. a cron/systemd timer that doesn't cd into it first). `diwan` must be run from inside (or below) a content repo, or have `DIWAN_NOTES_DIR` set — it has no content of its own to act on. Everything else (what counts as "trivial", where an inbox item belongs, how to answer a chat question) is intentionally **not** code — it's prose in `TICK.md`/`DO.md` that the dispatched `pi` agent has to interpret. The CLI's job ends at "here's a prompt, here's a terminal"; everything judgment-shaped is left to the model. ## Doc map | File | Lives in | Audience | Contents | |---|---|---|---| | `DESIGN.md` (this file) | tool repo | diwan developers | why, architecture, current implementation status, known gaps | | `PROTOCOL.md` | tool repo | any dispatched diwan agent (tick or do) | diwan's generic contract: tags, `diwan.org`, concurrency contract, revert model, herdr topology — deployment-independent | | `TICK.md` | tool repo | a `diwan tick` agent specifically | the 5-step tick procedure | | `DO.md` | tool repo | a `diwan do` agent specifically | scope, chat protocol, commit discipline, how to stop | | `AGENTS.md` | **content repo** | any dispatched diwan agent (tick or do) | that deployment's specifics: layout, org conventions already in use — read on top of `PROTOCOL.md`, never restates it | | `herdr --skill` (external) | — | anyone scripting herdr | authoritative herdr CLI reference — re-check this if herdr's version changes, since `diwan`'s calls were verified against 0.9.5 specifically | If you change a convention, update it in exactly one of these (whichever owns it) and make sure the others still only *point* at it rather than re-stating it — the duplication between `PROTOCOL.md` and any content repo's `AGENTS.md` is deliberately kept to zero; don't let that drift. Anything true of diwan regardless of which notes repo it's pointed at belongs in `PROTOCOL.md` (this repo); anything specific to one person's notes belongs in that content repo's own `AGENTS.md`, never here. ## What `diwan tick` actually does Two halves: a deterministic CLI half (`cmd_tick` in `diwan`) and a judgment-driven agent half (`TICK.md`). **CLI half**, in order: 1. If the content repo's working tree is dirty, commit everything under the user's own git identity (`chore: snapshot manual edits before tick`) — this is what makes "no manual commit requirement" (PROTOCOL.md) actually true. 2. Load the content repo's `.diwan/state.json`'s `last_tick_commit`. Diff that commit against current `HEAD` (in the content repo), excluding `.diwan/` itself (so diwan's own bookkeeping commits never count as "something changed"). No prior tick → treat as "always changed." 3. If nothing changed and `--force` wasn't passed: print and exit. This is the only gate on whether a tick does anything at all. 4. Otherwise, pick a tab name `tick-` (lowercase + digits + hyphens only — see "herdr integration" for why that's not negotiable) and dispatch a fresh agent there (`herdr_dispatch_fresh`), in a pane whose working directory is the *content* repo (`DIWAN_NOTES_DIR`), with a prompt that says: read `/PROTOCOL.md` and `AGENTS.md` (the latter resolved relative to the agent's own cwd, i.e. the content repo), follow `/TICK.md`, here's the commit range. **This dispatch is fire-and-forget** — the CLI call returns as soon as the prompt is submitted, it does not wait for the tick to finish. 5. Record the new tab name in the content repo's `state.json`'s `tick_history` (kept to the last `MAX_TICK_TABS`, default 3, override via `DIWAN_MAX_TICK_TABS`), commit that under the diwan identity, and — if this pushed any older tick tabs out of the retention window — close those tabs for real via `herdr tab close`. Note what's *not* here: `last_tick_commit` is deliberately **not** updated by the CLI at dispatch time. It's updated by the dispatched agent itself, as the literal last step of `TICK.md`, only if everything else succeeded. If a tick agent crashes or is killed partway, the next tick sees the same "changed" range again rather than silently skipping work — correctness over tidiness. **Agent half** (`TICK.md`), five steps, each its own commit: 1. Triage `inbox.org` only (other files changing is someone else's edit): events → `esdeveniments.org`, tasks → wherever they actually belong, `:diwan:` items get moved but not acted on yet, new topics only get a new file when nothing existing fits. 2. Tag consistency: any `:session:`/`:chat:` heading without a `:diwan:` tag on itself or an ancestor gets `:diwan:` added directly to it. Then resolve `:chat:` ownership: chats under a `:session:` ancestor are left for that heading's own session; orphan chats get answered inline if trivial, or escalated (see step 3's three-way decision) if not. 3. Handle every `:diwan:`-tagged heading without an open `:session:`: trivial ones get answered inline, no tag beyond `:diwan:` ever assigned. Non-trivial ones split two ways — self-contained within the heading's own subtree gets a `:session:` tag and `diwan do : --diwan-id ` run *by the tick agent itself* (the tick agent is the only thing that ever originates a fresh `DIWAN_ID`); inherently cross-cutting work (spans multiple files/headings, can't be parallelized — archiving, bulk cleanup, moving things between files) is done by the tick agent *directly*, right there, no session at all, possibly spread across several tick runs if it needs Marc's input first. The same handling applies to any open item under `diwan.org`'s `Tasks` heading — diwan's own self-maintenance backlog, no tag needed there since the location already signals it (see PROTOCOL.md's `diwan.org` section). Already-open sessions (`:session:` tag + `DIWAN_ID` present, `Close diwan session` still `[ ]`) get their subtree's content left alone as always, but are checked for a nudge: if any commit in this tick's change range *not* authored by the diwan git identity touched a line inside that subtree (a hand-authored heading edit, a new unread `:chat:` entry, anything Marc did), the tick agent re-runs `diwan do : --diwan-id ` on the existing ID — a continuation dispatch into the session's own tab/agent, not a fresh one — purely to wake it up; no nudge at all if nothing hand-authored touched the subtree. This is the only mechanism that gets a change Marc makes to an active heading in front of its own idle agent — nothing watches the file otherwise. A session whose `Close diwan session` is already `[X]` gets retired for real: the tick agent looks up its `DIWAN_ID`'s herdr tab and actually closes it (`herdr tab close`), then removes the `:session:` tag from the heading — the `DIWAN_ID` property itself is left in place as a historical record. None of dispatch, nudge, or closure get a Reports entry on their own; see step 4. 4. Delete any already-`[X]` Reports subheadings first, then — only for things that actually clear the "real information" bar, not routine mechanical steps like a bare dispatch or a session closure Marc asked for — write one short, titled, checkbox subheading per reportable thing this run did into `diwan.org`'s Reports log (not one paragraph for the whole run), each with the title itself linking to the one heading it's about (plain text if there isn't one) and a 1-2 line signed body. Zero new entries is a normal outcome. 5. Only if 1–4 all succeeded: update `state.json.last_tick_commit` to current `HEAD` and commit under the diwan identity. **Verified live** (2026-11-08, herdr 0.9.5 + Claude Code, before both the 2026-11-09 tool/content split and the pi-only harness switch): dispatched for real, ran to completion with zero permission prompts, correctly left two already-open `:session:`-tagged sessions untouched (back when `:session:` was still spelled `:diwan:`, before the tag rename — see "Known gaps" for what's unverified post-rename), correctly found nothing to triage, wrote a proper signed report, and updated `state.json` correctly. A second immediate `diwan tick` correctly no-op'd. **Not yet re-verified with a real dispatched agent** against the current state of things (post-split two-repo layout, `pi` as the sole harness) — path resolution and `pi`'s own startup have each been tested separately (see below and "Known gaps"), but not yet together through a full live tick run. Do that before trusting it for real. ## What `diwan do` actually does Same split: `cmd_do` (CLI) decides identity/dispatch mechanics; `DO.md` governs what the agent does once running. **CLI half** (`cmd_do`): 1. Parse `target`: if it matches `:` (resolved against the content repo), it's heading mode; anything else is treated as an ad-hoc free-text prompt with no org heading involved (useful for manual testing, or tasks that genuinely don't belong in the notes). 2. **Heading mode:** read the headline at that line, check its `DIWAN_ID` property. - Already has one → **continuation**. (If `--diwan-id` was also passed and disagrees with the stored one, that's an error, not a silent overwrite — a mismatch means something's confused and a human should look.) The existing ID is validated against herdr's naming rule before anything else happens, so a corrupted/hand-edited property fails with a clear message instead of a confusing herdr-side error later. - No `DIWAN_ID` yet → **fresh**. Compute one: `--diwan-id` if given, else a slug of the heading text (`slugify`, capped and shaped to satisfy herdr's agent-name rule — see below), de-duplicated against every other `DIWAN_ID` already in the content repo's tree (`unique_id`). Write the property and insert `** [ ] Close diwan session` as the first child (`assign_diwan_id`), in one commit under the diwan identity. This is the *only* place a `DIWAN_ID` is minted or a Close-session item is inserted — mechanical, not left to agent judgment, so it's always consistent. 3. **Ad-hoc mode:** `diwan_id` is `--diwan-id` or a generated `adhoc-<8 hex chars>`; always goes through the continuation dispatch path (see below) since there's no heading state to distinguish fresh from resumed. 4. Dispatch: fresh headings use `herdr_dispatch_fresh`; everything else (continuation headings, ad-hoc) uses `herdr_dispatch_continuation`. Prints the `diwan_id` either way — this is the command's only stdout contract, which is what lets `TICK.md` step 3 just run `diwan do ...` as an ordinary shell command and know what session it created. **Continuation dispatch, concretely** (`herdr_dispatch_continuation`): check `herdr agent get ` — alive → just `agent prompt ` (fire-and-forget, no `--wait`). Not alive (the normal case between ticks; CLI agent processes exit, they don't idle forever) → look up the tab by label in the `diwan` workspace, start a fresh agent in its existing root pane, prompt it. Tab itself gone too (e.g. closed by hand) → falls through to the same fresh-tab path `herdr_dispatch_fresh` uses. **Continuity across resumptions is carried by the heading's own content, not by agent conversation memory** — a resumed `do` agent has no idea what it said last time except what's written in the org file. This is a real, current limitation, not an oversight to "fix" without thinking about it — it's a direct consequence of principle #1 (org is ground truth) and keeps the system's state in exactly one place. **Agent half** (`DO.md`): stay inside your own heading's subtree; if something needs doing elsewhere, ask in `:chat:` rather than reaching out of scope (cross-cutting work like this is now `tick`'s job to do directly, per the `:diwan:` tag section — don't try to talk another `do` session into doing it either); use the chat protocol for questions/FYIs; commit progress in small units under the diwan identity through the flock-wrapped recipe; never touch your own `Close diwan session` item — that's Marc's call, acted on by the next tick. One explicit exception to subtree-only: a `do` agent may append its own entry to `diwan.org`'s Reports log once it finishes something worth telling Marc about, since that's its only channel to him outside of a tick run — it still never edits or deletes an existing entry there, only `tick` does that. A tick that merely nudges an idle session (see `cmd_tick` step 3 above) does not report the nudge itself — it has no idea yet whether anything came of it. **Verified live with Claude Code** (pre tool/content split, pre pi-only switch): dispatched for real against the "gather info about Greece trip" heading. Did real web research, wrote a structured result into the heading, correctly asked clarifying questions via a signed `:chat:` entry instead of guessing, left `Close diwan session` untouched, committed as `diwan ` with a descriptive message. (Historical — Claude Code is no longer a supported harness; see "One harness, chosen deliberately" above.) **`pi`, verified live, ad-hoc dispatch only** (2026-11-08, herdr 0.9.5 + pi 1.5.0, pre-split; re-confirmed 2026-11-09 post-split, post-switch against the fresh `diwan-dev-notes` repo): `diwan do "" --diwan-id ` correctly starts a herdr `pi` agent (detected as `agent: "pi"`), with no trust/approval prompt blocking it even in a content repo `pi` had never seen before (confirming it really doesn't share Claude Code's first-run trust-dialog problem — see "Known gaps"), loads `AGENTS.md` as context, and replies exactly as instructed. A second `diwan do` call against the same `--diwan-id` reuses the same pane/tab and agent (continuation dispatch, not a fresh one) and replies correctly again. **Not yet exercised**: a real `tick` or `do` run against actual heading content with `pi` as the agent (only ad-hoc free-text prompts so far) — worth a full pass before relying on it for real delegated work. ## `DIWAN_ID` ⇄ herdr, precisely A `DIWAN_ID` *is* a herdr tab label in the `diwan` workspace, and (while its agent process is alive) also that agent's name. This is why it must satisfy herdr's agent-name rule: `^[a-z][a-z0-9_-]{0,31}$`. `slugify()` enforces this at mint time (lowercase, non-alnum → hyphen, truncated to 24 chars, forced to start with a letter) and `cmd_do` additionally validates *every* `DIWAN_ID` it's about to dispatch — including ones already stored in a heading — against that same regex before touching herdr, erroring out with a clear message rather than letting herdr reject it after the CLI has already created a tab (which used to leave an orphan tab behind; fixed by closing the just-created tab on any `agent start` failure). Lifecycle: **fresh** (no property) → `diwan do` mints an ID, creates the tab+agent, heading now has `DIWAN_ID` + open (`[ ]`) `Close diwan session` → **active** (any further `tick`/`do` on this heading is a continuation; `tick` explicitly leaves active sessions alone) → Marc checks off `Close diwan session` (`[X]`) → next `tick` actually retires it (per `TICK.md` step 3): closes the herdr tab for real, removes the `:session:` tag, leaves the `DIWAN_ID` property in place as a historical record — **retired** (the heading can still be found later by that `DIWAN_ID`, but there's no live terminal and no tag pointing at one). ## Known gaps / explicit non-goals (don't "fix" these without discussing) - **Tool/content repo split just happened (2026-11-09), not yet re-verified live.** This repo used to be both diwan's own dev copy *and* a throwaway test-content repo at once; it's now tool-only, with content split out to a sibling `diwan-dev-notes` repo. The CLI was updated to resolve `DIWAN_DIR` (its own install dir, `Path(__file__).resolve().parent`, overridable via the `DIWAN_DIR` env var) separately from `DIWAN_NOTES_DIR` (`git rev-parse --show-toplevel` from the invoking directory by default, overridable via the `DIWAN_NOTES_DIR` env var), and the dispatch prompts now pass `DIWAN_DIR`-absolute paths for `PROTOCOL.md`/`TICK.md`/`DO.md`. This has been syntax-checked and smoke-tested for path resolution (including both env var overrides, from outside any repo), but **not yet run through a full live tick/do pass with a real dispatched agent** against the new two-repo layout — do that before trusting it for real. - **No flake yet.** The split prepared the ground (tool repo is self-contained, no content-repo assumptions baked into its own files) but there's no `flake.nix` in this repo yet, and the shebang still runs via a bare `nix shell nixpkgs#python3` rather than a proper package. Until that lands, "installed on `PATH`" means manually symlinking/aliasing `diwan` somewhere on `PATH` yourself. `DIWAN_DIR` exists specifically so a future flake's install layout (e.g. docs copied into `$out/share/diwan/` alongside a wrapped binary) can override the default `Path(__file__).resolve().parent` guess if it doesn't hold. - **Retired-session teardown is designed but not yet verified live.** `TICK.md` step 3 now has the tick agent actually close the herdr tab (`herdr tab close`) and strip the `:session:` tag once `Close diwan session` is checked off, rather than just reporting that it happened — but no real tick run has exercised this path yet. Worth a close look the first few times it fires: does the tab lookup by `DIWAN_ID` label reliably find the right tab, does tag-stripping leave `:diwan:`/other tags intact, does it behave sanely if the tab was already closed by hand. - **No real conversation continuity across resumption.** Covered above — intentional for now, but worth revisiting if resumed `do` sessions turn out to need more context than the heading text carries. - **`pi` is smoke-tested for ad-hoc dispatch, but not yet exercised against real heading content.** `--approve` has been confirmed live for ad-hoc `diwan do` dispatch and continuation, including against a content repo `pi` had never run in before (no trust-dialog block, unlike Claude Code — see "One harness, chosen deliberately" above). Still untested: a real `diwan tick` run, and `diwan do` against a real `:` heading (DIWAN_ID minting, `:chat:` protocol, scope discipline) rather than a free-text ad-hoc prompt. - **`diwan.org`'s `Config` heading is a placeholder, not wired up.** PROTOCOL.md documents an intended home for configuration values governing diwan's own behavior (tick cadence, tab retention, that kind of thing), but the CLI reads none of it — the env vars (`DIWAN_DIR`, `DIWAN_NOTES_DIR`, `DIWAN_PI_APPROVE`, `DIWAN_MAX_TICK_TABS`) remain the only real knobs. Don't assume anything in that heading does something until the CLI actually reads it. - **The `:diwan:`/`:session:`/`:chat:` tag split, and tick doing cross-cutting maintenance directly, is new and unverified live.** Before this split, `:diwan:` alone meant what `:session:` means now, and anything non-trivial got a dedicated `do` session regardless of whether the work was subtree-shaped or genuinely repo-wide — which is exactly what motivated the split (a real `do` session hit the subtree wall trying to do an archive/cleanup/folder-move task). No tick run has yet exercised the new three-way decision in `TICK.md` step 3 (inline / `:session:` dispatch / tick-direct), the tag-consistency backfill in step 2, or tick actually doing a multi-file maintenance job itself across one or more runs — worth a close look at the first few real runs that hit this path. - **Concurrency is scope discipline, not real locking**, beyond the git-commit flock. Two `do` agents editing truly disjoint subtrees should be fine; nothing stops two agents editing the *same* subtree if the scope rule is violated. There's no enforcement, only convention (PROTOCOL.md). - **No automated tests.** Everything described as "verified live" above was exercised by hand against a real herdr install. There is no regression suite — a future session changing `diwan` should re-run at least the manual smoke tests (tick no-op gate, tick dispatch, do fresh dispatch, do continuation, a deliberately-invalid `DIWAN_ID`) before trusting a change.