DIWAN_DIR: where diwan itself is installed (was TOOL_ROOT / DIWAN_TOOL_ROOT). DIWAN_NOTES_DIR: the content repo diwan is invoked against (was CONTENT_ROOT); now also has its own env var override, not just git rev-parse --show-toplevel discovery, for cron/systemd use where cwd may not be the content repo. The content repo's own runtime-bookkeeping directory (state.json, .lock) is renamed STATE_DIR internally to avoid colliding with the new DIWAN_DIR name; its on-disk path is unchanged (still a .diwan/ subdirectory of the content repo).
109 lines
25 KiB
Markdown
109 lines
25 KiB
Markdown
# DESIGN.md — diwan
|
||
|
||
Audience: whoever (human or agent) picks up **development of diwan itself** next. If you're an agent dispatched *by* diwan to triage notes or do delegated work, you want `PROTOCOL.md` (given to you directly in your dispatch prompt) and the content repo's own `AGENTS.md` instead — this doc is one level up, about the tool, not about using the tool.
|
||
|
||
## What this is
|
||
|
||
diwan is an AI layer over a plain org-mode "second brain" (notes, tasks, calendar, projects — normally edited by hand in Emacs/beorg, synced via Nextcloud). It periodically triages a capture inbox, keeps files tidy, and runs delegated research/work when asked, while treating the org files as the only real source of truth.
|
||
|
||
**This repo is diwan the tool, and nothing else.** It contains the CLI and the generic procedure docs (`PROTOCOL.md`, `TICK.md`, `DO.md`) — no personal notes, no content-repo-specific convention docs. It's meant to be installed independently (on `PATH`, eventually via its own nix flake) and pointed at *any* notes repo. Marc's actual notes live in `~/nc/org-agenda` (the real, continuously-synced data) and, for development/testing, in a throwaway sibling repo `diwan-dev-notes` (a git-tracked fixture copy, safe to break). Neither of those repos contains any of diwan's own code or docs — see "Known gaps" for exactly how far that packaging currently goes.
|
||
|
||
Status as of this writing: the core loop (tick → triage/dispatch, do → delegated work, both via real herdr sessions) is built and has been exercised end-to-end against live `herdr` and `claude`. The tool/content split (this repo vs. `diwan-dev-notes`) happened 2026-11-09, right after that verification — see "Known gaps" for what that split still owes a fresh live pass on.
|
||
|
||
## Principles (why things are shaped this way)
|
||
|
||
These were distilled from how Marc already works and what he explicitly asked for; they're the standard to hold any future change to:
|
||
|
||
1. **Org files are ground truth, not a cache.** If diwan vanished, the notes still work in Emacs/beorg. Nothing diwan needs lives outside `.org` files except the content repo's own `.diwan/state.json`, which is pure bookkeeping, not content.
|
||
2. **Act first, review passively.** No per-edit approval gate — every action is reported and trivially revertable instead.
|
||
3. **Mechanism over magic.** Small, inspectable, deterministic steps over one opaque "agent did something" blob — Marc wants to be able to understand a failure well enough to jump in personally.
|
||
4. **Grow by accretion.** Ship the smallest loop that works; add complexity only once the simpler version has been lived with. (Concretely: v0 has no custom CLI-level org-tag manipulation, no scheduler, no kill-on-timeout logic — these are deliberately not yet built.)
|
||
5. **Harness as a thin, swappable shell.** The procedure (what tick/do should do) and the primitives (how edits actually happen) are decoupled from which LLM product is driving. Claude Code today; `pi` is a supported herdr integration kind and should work by just changing `DIWAN_HARNESS`.
|
||
6. **Two symmetric channels, one substrate.** Talk to diwan via the notes (`:chat:`) or via the live herdr session — both write to the same git-tracked truth.
|
||
7. **Mobile stays dumb.** Nothing new to install on the phone.
|
||
8. **Tool and content are separate repos.** diwan the tool (this repo) ships independent of any one person's notes; a content repo (e.g. `diwan-dev-notes`, or Marc's real `~/nc/org-agenda`) only ever contains notes, `AGENTS.md`, `diwan.org`, and diwan's own tiny `.diwan/` runtime-bookkeeping directory (`state.json`, `.lock`) — never the CLI or the procedure docs. This is what makes the content repo distributable (synced via Nextcloud to a server, edited from beorg) without dragging a whole dev tool along with it, and what keeps diwan's own generic contract reviewable/versionable independent of any one person's data.
|
||
|
||
## Architecture at a glance
|
||
|
||
Three layers, each independently simple:
|
||
|
||
- **Git**, in the *content* repo, no remote (for Marc's prototype content repo — a real deployment may add one, e.g. for Nextcloud sync, but that's the content repo's business, not this one's). Every automated write to the content repo is a commit. This *is* the revert mechanism (principle #2) — not a bespoke undo log. Commits are authored as `diwan <diwan@local>` when diwan itself is the actor, or the user's own git identity when `diwan tick` snapshots whatever manual edits were lying around at the start of a run (so no tick ever operates on a dirty/mixed workspace, and "who changed this" is answerable from `git log --author` alone). This repo (the tool) has its own, unrelated git history — just ordinary source control for diwan's own development.
|
||
- **herdr**, a terminal-multiplexer-for-agents with a JSON-over-socket CLI (`herdr workspace/tab/pane/agent ...`). diwan doesn't invent a session store: one herdr workspace labeled `diwan` holds every tick run and every open `do` session as a separate, independently addressable, persistent tab. See "herdr integration" below for the exact calls.
|
||
- **The `diwan` CLI** (`./diwan` at this repo's root), a single dependency-free Python script (shebang runs it through `nix shell nixpkgs#python3`, so no system Python install needed on NixOS; a real flake package would wrap this properly instead). It does only the mechanical, deterministic parts: git bookkeeping, enough org parsing to read/write a `DIWAN_ID` property and insert a `Close diwan session` checkbox item, and herdr dispatch. It never triages, writes prose, or makes judgment calls — that's the dispatched agent's job, guided by `TICK.md`/`DO.md`. It distinguishes two roots explicitly (see the script's own docstring): `DIWAN_DIR` (where it and its docs are installed — `Path(__file__).resolve().parent`, overridable via the `DIWAN_DIR` env var for whatever a future flake's install layout needs) and `DIWAN_NOTES_DIR` (whatever notes repo it's invoked against, found via `git rev-parse --show-toplevel` from the current directory by default, overridable via the `DIWAN_NOTES_DIR` env var for e.g. a cron/systemd timer that doesn't cd into it first). `diwan` must be run from inside (or below) a content repo, or have `DIWAN_NOTES_DIR` set — it has no content of its own to act on.
|
||
|
||
Everything else (what counts as "trivial", where an inbox item belongs, how to answer a chat question) is intentionally **not** code — it's prose in `TICK.md`/`DO.md` that whatever LLM is dispatched has to interpret. This is the harness-agnosticism principle made concrete: the CLI's job ends at "here's a prompt, here's a terminal," everything judgment-shaped is left to the model.
|
||
|
||
## Doc map
|
||
|
||
| File | Lives in | Audience | Contents |
|
||
|---|---|---|---|
|
||
| `DESIGN.md` (this file) | tool repo | diwan developers | why, architecture, current implementation status, known gaps |
|
||
| `PROTOCOL.md` | tool repo | any dispatched diwan agent (tick or do) | diwan's generic contract: tags, `diwan.org`, concurrency contract, revert model, herdr topology — deployment-independent |
|
||
| `TICK.md` | tool repo | a `diwan tick` agent specifically | the 5-step tick procedure |
|
||
| `DO.md` | tool repo | a `diwan do` agent specifically | scope, chat protocol, commit discipline, how to stop |
|
||
| `AGENTS.md` | **content repo** | any dispatched diwan agent (tick or do) | that deployment's specifics: layout, org conventions already in use — read on top of `PROTOCOL.md`, never restates it |
|
||
| `herdr --skill` (external) | — | anyone scripting herdr | authoritative herdr CLI reference — re-check this if herdr's version changes, since `diwan`'s calls were verified against 0.9.5 specifically |
|
||
|
||
If you change a convention, update it in exactly one of these (whichever owns it) and make sure the others still only *point* at it rather than re-stating it — the duplication between `PROTOCOL.md` and any content repo's `AGENTS.md` is deliberately kept to zero; don't let that drift. Anything true of diwan regardless of which notes repo it's pointed at belongs in `PROTOCOL.md` (this repo); anything specific to one person's notes belongs in that content repo's own `AGENTS.md`, never here.
|
||
|
||
## What `diwan tick` actually does
|
||
|
||
Two halves: a deterministic CLI half (`cmd_tick` in `diwan`) and a judgment-driven agent half (`TICK.md`).
|
||
|
||
**CLI half**, in order:
|
||
1. If the content repo's working tree is dirty, commit everything under the user's own git identity (`chore: snapshot manual edits before tick`) — this is what makes "no manual commit requirement" (PROTOCOL.md) actually true.
|
||
2. Load the content repo's `.diwan/state.json`'s `last_tick_commit`. Diff that commit against current `HEAD` (in the content repo), excluding `.diwan/` itself (so diwan's own bookkeeping commits never count as "something changed"). No prior tick → treat as "always changed."
|
||
3. If nothing changed and `--force` wasn't passed: print and exit. This is the only gate on whether a tick does anything at all.
|
||
4. Otherwise, pick a tab name `tick-<UTC YYYYMMDD-HHMMSS>` (lowercase + digits + hyphens only — see "herdr integration" for why that's not negotiable) and dispatch a fresh agent there (`herdr_dispatch_fresh`), in a pane whose working directory is the *content* repo (`DIWAN_NOTES_DIR`), with a prompt that says: read `<DIWAN_DIR>/PROTOCOL.md` and `AGENTS.md` (the latter resolved relative to the agent's own cwd, i.e. the content repo), follow `<DIWAN_DIR>/TICK.md`, here's the commit range. **This dispatch is fire-and-forget** — the CLI call returns as soon as the prompt is submitted, it does not wait for the tick to finish.
|
||
5. Record the new tab name in the content repo's `state.json`'s `tick_history` (kept to the last `MAX_TICK_TABS`, default 3, override via `DIWAN_MAX_TICK_TABS`), commit that under the diwan identity, and — if this pushed any older tick tabs out of the retention window — close those tabs for real via `herdr tab close`.
|
||
|
||
Note what's *not* here: `last_tick_commit` is deliberately **not** updated by the CLI at dispatch time. It's updated by the dispatched agent itself, as the literal last step of `TICK.md`, only if everything else succeeded. If a tick agent crashes or is killed partway, the next tick sees the same "changed" range again rather than silently skipping work — correctness over tidiness.
|
||
|
||
**Agent half** (`TICK.md`), five steps, each its own commit:
|
||
1. Triage `inbox.org` only (other files changing is someone else's edit): events → `esdeveniments.org`, tasks → wherever they actually belong, `:diwan:` items get moved but not acted on yet, new topics only get a new file when nothing existing fits.
|
||
2. Tag consistency: any `:session:`/`:chat:` heading without a `:diwan:` tag on itself or an ancestor gets `:diwan:` added directly to it. Then resolve `:chat:` ownership: chats under a `:session:` ancestor are left for that heading's own session; orphan chats get answered inline if trivial, or escalated (see step 3's three-way decision) if not.
|
||
3. Handle every `:diwan:`-tagged heading without an open `:session:`: trivial ones get answered inline, no tag beyond `:diwan:` ever assigned. Non-trivial ones split two ways — self-contained within the heading's own subtree gets a `:session:` tag and `diwan do <file>:<line> --diwan-id <slug>` run *by the tick agent itself* (the tick agent is the only thing that ever originates a fresh `DIWAN_ID`); inherently cross-cutting work (spans multiple files/headings, can't be parallelized — archiving, bulk cleanup, moving things between files) is done by the tick agent *directly*, right there, no session at all, possibly spread across several tick runs if it needs Marc's input first. The same handling applies to any open item under `diwan.org`'s `Tasks` heading — diwan's own self-maintenance backlog, no tag needed there since the location already signals it (see PROTOCOL.md's `diwan.org` section). Already-open sessions (`:session:` tag + `DIWAN_ID` present, `Close diwan session` still `[ ]`) get their subtree's content left alone as always, but are checked for a nudge: if any commit in this tick's change range *not* authored by the diwan git identity touched a line inside that subtree (a hand-authored heading edit, a new unread `:chat:` entry, anything Marc did), the tick agent re-runs `diwan do <file>:<line> --diwan-id <id>` on the existing ID — a continuation dispatch into the session's own tab/agent, not a fresh one — purely to wake it up; no nudge at all if nothing hand-authored touched the subtree. This is the only mechanism that gets a change Marc makes to an active heading in front of its own idle agent — nothing watches the file otherwise.
|
||
4. Write one short, titled, checkbox subheading per notable thing this run did into `diwan.org`'s Reports log (not one paragraph for the whole run), each with an optional `:LINK:` property and a 1-2 line signed body; delete any already-`[X]` subheadings first.
|
||
5. Only if 1–4 all succeeded: update `state.json.last_tick_commit` to current `HEAD` and commit under the diwan identity.
|
||
|
||
**Verified live** (2026-11-08, herdr 0.9.5 + Claude Code, `claude` kind, against the old mixed tool+content repo, before the 2026-11-09 split): dispatched for real, ran to completion with zero permission prompts (see "Known gaps" for why that almost didn't work), correctly left two already-open `:session:`-tagged sessions untouched (back when `:session:` was still spelled `:diwan:`, before the tag rename — see "Known gaps" for what's unverified post-rename), correctly found nothing to triage, wrote a proper signed report, and updated `state.json` correctly. A second immediate `diwan tick` correctly no-op'd. **Not yet re-verified with a real dispatched agent** against the post-split layout (tool repo + separate `diwan-dev-notes` content repo, `DIWAN_DIR`/`DIWAN_NOTES_DIR` as two different directories) — path resolution itself has been dry-run tested (see below), but no agent has actually followed the resulting prompt yet. See "Known gaps".
|
||
|
||
## What `diwan do` actually does
|
||
|
||
Same split: `cmd_do` (CLI) decides identity/dispatch mechanics; `DO.md` governs what the agent does once running.
|
||
|
||
**CLI half** (`cmd_do`):
|
||
1. Parse `target`: if it matches `<existing file>:<line number>` (resolved against the content repo), it's heading mode; anything else is treated as an ad-hoc free-text prompt with no org heading involved (useful for manual testing, or tasks that genuinely don't belong in the notes).
|
||
2. **Heading mode:** read the headline at that line, check its `DIWAN_ID` property.
|
||
- Already has one → **continuation**. (If `--diwan-id` was also passed and disagrees with the stored one, that's an error, not a silent overwrite — a mismatch means something's confused and a human should look.) The existing ID is validated against herdr's naming rule before anything else happens, so a corrupted/hand-edited property fails with a clear message instead of a confusing herdr-side error later.
|
||
- No `DIWAN_ID` yet → **fresh**. Compute one: `--diwan-id` if given, else a slug of the heading text (`slugify`, capped and shaped to satisfy herdr's agent-name rule — see below), de-duplicated against every other `DIWAN_ID` already in the content repo's tree (`unique_id`). Write the property and insert `** [ ] Close diwan session` as the first child (`assign_diwan_id`), in one commit under the diwan identity. This is the *only* place a `DIWAN_ID` is minted or a Close-session item is inserted — mechanical, not left to agent judgment, so it's always consistent.
|
||
3. **Ad-hoc mode:** `diwan_id` is `--diwan-id` or a generated `adhoc-<8 hex chars>`; always goes through the continuation dispatch path (see below) since there's no heading state to distinguish fresh from resumed.
|
||
4. Dispatch: fresh headings use `herdr_dispatch_fresh`; everything else (continuation headings, ad-hoc) uses `herdr_dispatch_continuation`. Prints the `diwan_id` either way — this is the command's only stdout contract, which is what lets `TICK.md` step 3 just run `diwan do ...` as an ordinary shell command and know what session it created.
|
||
|
||
**Continuation dispatch, concretely** (`herdr_dispatch_continuation`): check `herdr agent get <id>` — alive → just `agent prompt <id> <text>` (fire-and-forget, no `--wait`). Not alive (the normal case between ticks; CLI agent processes exit, they don't idle forever) → look up the tab by label in the `diwan` workspace, start a fresh agent in its existing root pane, prompt it. Tab itself gone too (e.g. closed by hand) → falls through to the same fresh-tab path `herdr_dispatch_fresh` uses. **Continuity across resumptions is carried by the heading's own content, not by agent conversation memory** — a resumed `do` agent has no idea what it said last time except what's written in the org file. This is a real, current limitation, not an oversight to "fix" without thinking about it — it's a direct consequence of principle #1 (org is ground truth) and keeps the system's state in exactly one place.
|
||
|
||
**Agent half** (`DO.md`): stay inside your own heading's subtree; if something needs doing elsewhere, ask in `:chat:` rather than reaching out of scope (cross-cutting work like this is now `tick`'s job to do directly, per the `:diwan:` tag section — don't try to talk another `do` session into doing it either); use the chat protocol for questions/FYIs; commit progress in small units under the diwan identity through the flock-wrapped recipe; never touch your own `Close diwan session` item — that's Marc's call, acted on by the next tick. One explicit exception to subtree-only: a `do` agent may append its own entry to `diwan.org`'s Reports log once it finishes something worth telling Marc about, since that's its only channel to him outside of a tick run — it still never edits or deletes an existing entry there, only `tick` does that. A tick that merely nudges an idle session (see `cmd_tick` step 3 above) does not report the nudge itself — it has no idea yet whether anything came of it.
|
||
|
||
**Verified live** (against the old mixed repo, pre-split): dispatched for real against the "gather info about Greece trip" heading. Did real web research, wrote a structured result into the heading, correctly asked clarifying questions via a signed `:chat:` entry instead of guessing, left `Close diwan session` untouched, committed as `diwan <diwan@local>` with a descriptive message.
|
||
|
||
**`pi` kind, verified live** (2026-11-08, herdr 0.9.5 + pi 1.5.0, pre-split): ad-hoc `diwan do "<prompt>" --diwan-id pi-smoketest` with `DIWAN_HARNESS=pi` correctly started a herdr `pi` agent (detected as `agent: "pi"`, no trust/approval prompt blocked it, confirming `--approve` is doing its job with no protected `.pi/*` resources in that repo to actually gate), loaded `AGENTS.md` as context the way `claude` does, and replied exactly as instructed. A second `diwan do` call against the same `--diwan-id` reused the same pane/tab and agent (continuation dispatch, not a fresh one) and replied correctly again. Not yet exercised: a real `tick` or `do` run against actual heading content with `DIWAN_HARNESS=pi` (only ad-hoc free-text prompts so far) — same caveat as any first integration, worth a full pass before relying on it for real delegated work.
|
||
|
||
## `DIWAN_ID` ⇄ herdr, precisely
|
||
|
||
A `DIWAN_ID` *is* a herdr tab label in the `diwan` workspace, and (while its agent process is alive) also that agent's name. This is why it must satisfy herdr's agent-name rule: `^[a-z][a-z0-9_-]{0,31}$`. `slugify()` enforces this at mint time (lowercase, non-alnum → hyphen, truncated to 24 chars, forced to start with a letter) and `cmd_do` additionally validates *every* `DIWAN_ID` it's about to dispatch — including ones already stored in a heading — against that same regex before touching herdr, erroring out with a clear message rather than letting herdr reject it after the CLI has already created a tab (which used to leave an orphan tab behind; fixed by closing the just-created tab on any `agent start` failure).
|
||
|
||
Lifecycle: **fresh** (no property) → `diwan do` mints an ID, creates the tab+agent, heading now has `DIWAN_ID` + open (`[ ]`) `Close diwan session` → **active** (any further `tick`/`do` on this heading is a continuation; `tick` explicitly leaves active sessions alone) → Marc checks off `Close diwan session` (`[X]`) → next `tick` notices and retires it (per `TICK.md` step 3 — note: as of this writing the tick procedure only *reports* that a session closed; it does not yet actually call `herdr tab close` on it — see "Known gaps").
|
||
|
||
## Known gaps / explicit non-goals (don't "fix" these without discussing)
|
||
|
||
- **Tool/content repo split just happened (2026-11-09), not yet re-verified live.** This repo used to be both diwan's own dev copy *and* a throwaway test-content repo at once; it's now tool-only, with content split out to a sibling `diwan-dev-notes` repo. The CLI was updated to resolve `DIWAN_DIR` (its own install dir, `Path(__file__).resolve().parent`, overridable via the `DIWAN_DIR` env var) separately from `DIWAN_NOTES_DIR` (`git rev-parse --show-toplevel` from the invoking directory by default, overridable via the `DIWAN_NOTES_DIR` env var), and the dispatch prompts now pass `DIWAN_DIR`-absolute paths for `PROTOCOL.md`/`TICK.md`/`DO.md`. This has been syntax-checked and smoke-tested for path resolution (including both env var overrides, from outside any repo), but **not yet run through a full live tick/do pass with a real dispatched agent** against the new two-repo layout — do that before trusting it for real.
|
||
- **No flake yet.** The split prepared the ground (tool repo is self-contained, no content-repo assumptions baked into its own files) but there's no `flake.nix` in this repo yet, and the shebang still runs via a bare `nix shell nixpkgs#python3` rather than a proper package. Until that lands, "installed on `PATH`" means manually symlinking/aliasing `diwan` somewhere on `PATH` yourself. `DIWAN_DIR` exists specifically so a future flake's install layout (e.g. docs copied into `$out/share/diwan/` alongside a wrapped binary) can override the default `Path(__file__).resolve().parent` guess if it doesn't hold.
|
||
- **Retired-session teardown.** `TICK.md` step 3 says a closed session has "nothing for you to do to the heading's content" and that herdr-side teardown "is not your job" — but nothing else currently closes that tab either. Right now a `Close diwan session` checked off (`[X]`) just... sits there, tab still open. This is a real gap, not yet decided whether the CLI should gain a path for it or whether `TICK.md` should just say to run `herdr tab close` directly.
|
||
- **No real conversation continuity across resumption.** Covered above — intentional for now, but worth revisiting if resumed `do` sessions turn out to need more context than the heading text carries.
|
||
- **`pi` kind is wired up and smoke-tested, but not yet exercised against real heading content.** `DIWAN_HARNESS=pi` gets its own equivalent of the Claude autonomy flag (`--approve`, see "`pi` kind, verified live" above) and has been confirmed live for ad-hoc `diwan do` dispatch and continuation. Still untested: `diwan tick` with `DIWAN_HARNESS=pi`, and `diwan do` against a real `<file>:<line>` heading (DIWAN_ID minting, `:chat:` protocol, scope discipline) rather than a free-text ad-hoc prompt.
|
||
- **`diwan.org`'s `Config` heading is a placeholder, not wired up.** PROTOCOL.md documents an intended home for configuration values governing diwan's own behavior (tick cadence, tab retention, harness choice), but the CLI reads none of it — the env vars (`DIWAN_HARNESS`, `DIWAN_PERMISSION_MODE`, `DIWAN_PI_APPROVE`, `DIWAN_MAX_TICK_TABS`) remain the only real knobs. Don't assume anything in that heading does something until the CLI actually reads it.
|
||
- **The `:diwan:`/`:session:`/`:chat:` tag split, and tick doing cross-cutting maintenance directly, is new and unverified live.** Before this split, `:diwan:` alone meant what `:session:` means now, and anything non-trivial got a dedicated `do` session regardless of whether the work was subtree-shaped or genuinely repo-wide — which is exactly what motivated the split (a real `do` session hit the subtree wall trying to do an archive/cleanup/folder-move task). No tick run has yet exercised the new three-way decision in `TICK.md` step 3 (inline / `:session:` dispatch / tick-direct), the tag-consistency backfill in step 2, or tick actually doing a multi-file maintenance job itself across one or more runs — worth a close look at the first few real runs that hit this path.
|
||
- **`--permission-mode auto` is a real autonomy decision, not a default to take for granted.** Dispatched agents run with Claude Code's own judgment-based auto mode (chosen over a full `--dangerously-skip-permissions` bypass, which Claude's docs scope to no-internet sandboxes — diwan agents do real web research). Override via `DIWAN_PERMISSION_MODE` env var. If this is ever revisited, it should be a deliberate conversation, not a silent change.
|
||
- **Concurrency is scope discipline, not real locking**, beyond the git-commit flock. Two `do` agents editing truly disjoint subtrees should be fine; nothing stops two agents editing the *same* subtree if the scope rule is violated. There's no enforcement, only convention (PROTOCOL.md).
|
||
- **No automated tests.** Everything described as "verified live" above was exercised by hand against a real herdr+claude install. There is no regression suite — a future session changing `diwan` should re-run at least the manual smoke tests (tick no-op gate, tick dispatch, do fresh dispatch, do continuation, a deliberately-invalid `DIWAN_ID`) before trusting a change.
|