Files
diwan/DESIGN.md
T
marc 34f1f4396e Close diwan session: TODO/DONE instead of checkbox; purge evolutionary residue from all docs
1. 'Close diwan session' now uses TODO/DONE (assign_diwan_id inserts
'** TODO Close diwan session'), matching Reports. No headline anywhere
in diwan's own docs uses a [ ]/[X] checkbox now -- only :chat: list
items do, which is correct org usage (checklist items, not headline
states) and unaffected.

2. Rewrote PROTOCOL.md/TICK.md/DO.md/DESIGN.md to read as a single
coherent design rather than a changelog of how they got here. Removed:
contrastive 'not because X, but because Y' phrasing that only makes
sense if you know the prior rule, references to old tag names/formats
that no longer exist, a Claude-Code-vs-pi rationale that belongs in
DESIGN.md (if anywhere) not the agent-facing PROTOCOL.md, and 'no
:LINK: property' (never describe a convention by naming something it
doesn't have).

3. DESIGN.md got the heaviest pass since it had drifted from reality,
not just tone: it still said 'no flake.nix yet' (there is one, in
production use via samfelag), claimed several things 'not yet
re-verified live' that have since been verified multiple times
(tick's no-op gate, triage, the tag-consistency/escalation decision,
pi against real heading content via tick), and didn't mention
'diwan remote' or the actual flake.nix packaging shape anywhere.
Rewrote the verified-live claims and Known Gaps list to match current
reality, added flake.nix to the doc map and architecture section.

Live content migrated to match: diwan-dev-notes' decorate-home heading
('Close diwan session' checkbox -> TODO).
2026-11-09 23:17:17 +01:00

107 lines
22 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# DESIGN.md — diwan
Audience: whoever (human or agent) picks up **development of diwan itself** next. If you're an agent dispatched *by* diwan to triage notes or do delegated work, you want `PROTOCOL.md` (given to you directly in your dispatch prompt) and the content repo's own `AGENTS.md` instead — this doc is one level up, about the tool, not about using the tool.
## What this is
diwan is an AI layer over a plain org-mode "second brain" (notes, tasks, calendar, projects — normally edited by hand in Emacs/beorg, synced via Nextcloud). It periodically triages a capture inbox, keeps files tidy, and runs delegated research/work when asked, while treating the org files as the only real source of truth.
**This repo is diwan the tool, and nothing else.** It contains the CLI, a nix flake packaging it, and the generic procedure docs (`PROTOCOL.md`, `TICK.md`, `DO.md`) — no personal notes, no content-repo-specific convention docs. It's installed independently (on `PATH`, via this repo's own flake) and pointed at *any* notes repo. Marc's actual notes live in `~/nc/org-agenda` (the real, continuously-synced data) and, for development/testing, in a throwaway sibling repo `diwan-dev-notes` (a git-tracked fixture copy, safe to break). Neither of those repos contains any of diwan's own code or docs.
The core loop (tick → triage/dispatch, do → delegated work, both via real herdr sessions, `pi` as the sole harness) is built, packaged as a flake, and deployed: `samfelag` (Marc's NixOS fleet config) has a module that installs the package and runs `diwan tick` on a systemd timer, already live on `thingvellir`. See "Known gaps" for what's still open.
## Principles (why things are shaped this way)
These were distilled from how Marc already works and what he explicitly asked for; they're the standard to hold any future change to:
1. **Org files are ground truth, not a cache.** If diwan vanished, the notes still work in Emacs/beorg. Nothing diwan needs lives outside `.org` files except the content repo's own `.diwan/state.json`, which is pure bookkeeping, not content.
2. **Act first, review passively.** No per-edit approval gate — every action is reported and trivially revertable instead.
3. **Mechanism over magic.** Small, inspectable, deterministic steps over one opaque "agent did something" blob — Marc wants to be able to understand a failure well enough to jump in personally.
4. **Grow by accretion.** Ship the smallest loop that works; add complexity only once the simpler version has been lived with. (Concretely: v0 has no custom CLI-level org-tag manipulation, no scheduler beyond a plain systemd timer, no kill-on-timeout logic — these are deliberately not yet built.)
5. **One harness, chosen deliberately: `pi`.** diwan dispatches `pi` exclusively, not a pluggable choice. It has no per-tool-call approval gate and no interactive first-run trust dialog — it runs unattended from the start, which is exactly diwan's own model. The procedure docs (`TICK.md`/`DO.md`) don't hardcode anything pi-specific into their wording anyway, as a matter of good practice.
6. **Two symmetric channels, one substrate.** Talk to diwan via the notes (`:chat:`) or via the live herdr session — both write to the same git-tracked truth.
7. **Mobile stays dumb.** Nothing new to install on the phone.
8. **Tool and content are separate repos.** diwan the tool (this repo) ships independent of any one person's notes; a content repo (e.g. `diwan-dev-notes`, or Marc's real `~/nc/org-agenda`) only ever contains notes, `AGENTS.md`, `diwan.org`, and diwan's own tiny `.diwan/` runtime-bookkeeping directory (`state.json`, `.lock`) — never the CLI or the procedure docs. This is what makes the content repo distributable (synced via Nextcloud to a server, edited from beorg) without dragging a whole dev tool along with it, and what keeps diwan's own generic contract reviewable/versionable independent of any one person's data.
## Architecture at a glance
Three layers, each independently simple:
- **Git**, in the *content* repo. Every automated write to the content repo is a commit. This *is* the revert mechanism (principle #2) — not a bespoke undo log. Commits are authored as `diwan <diwan@local>` when diwan itself is the actor, or the user's own git identity when `diwan tick` snapshots whatever manual edits were lying around at the start of a run (so no tick ever operates on a dirty/mixed workspace, and "who changed this" is answerable from `git log --author` alone). This repo (the tool) has its own, unrelated git history — just ordinary source control for diwan's own development.
- **herdr**, a terminal-multiplexer-for-agents with a JSON-over-socket CLI (`herdr workspace/tab/pane/agent ...`). diwan doesn't invent a session store: one herdr workspace labeled `diwan` holds every tick run and every open `do` session as a separate, independently addressable, persistent tab. See PROTOCOL.md's herdr section for the exact calls.
- **The `diwan` CLI**, a single dependency-free Python script, packaged by this repo's `flake.nix` (`pname = "diwan"`): the script and `PROTOCOL.md`/`TICK.md`/`DO.md` get installed together under `$out/share/diwan/`, and `$out/bin/diwan` is a `makeWrapper`-generated wrapper around `python3` that sets `DIWAN_DIR` explicitly to that `share/diwan` directory and puts `git` on `PATH`. It does only the mechanical, deterministic parts: git bookkeeping, enough org parsing to read/write a `DIWAN_ID` property and insert a `Close diwan session` TODO item, and herdr dispatch. It never triages, writes prose, or makes judgment calls — that's the dispatched agent's job, guided by `TICK.md`/`DO.md`. It distinguishes two roots explicitly (see the script's own docstring): `DIWAN_DIR` (where it and its docs are installed, overridable via the `DIWAN_DIR` env var) and `DIWAN_NOTES_DIR` (whatever notes repo it's invoked against, found via `git rev-parse --show-toplevel` from the current directory by default, overridable via the `DIWAN_NOTES_DIR` env var — this is how a systemd service that doesn't `cd` anywhere points it at the right repo). `diwan` must be run from inside (or below) a content repo, or have `DIWAN_NOTES_DIR` set — it has no content of its own to act on.
Everything else (what counts as "trivial", where an inbox item belongs, how to answer a chat question) is intentionally **not** code — it's prose in `TICK.md`/`DO.md` that the dispatched `pi` agent has to interpret. The CLI's job ends at "here's a prompt, here's a terminal"; everything judgment-shaped is left to the model.
## Doc map
| File | Lives in | Audience | Contents |
|---|---|---|---|
| `DESIGN.md` (this file) | tool repo | diwan developers | why, architecture, current implementation status, known gaps |
| `flake.nix` | tool repo | nix, and anything consuming this flake (e.g. `samfelag`) | packages the CLI + docs together; see "Architecture at a glance" |
| `PROTOCOL.md` | tool repo | any dispatched diwan agent (tick or do) | diwan's generic contract: tags, `diwan.org`, concurrency contract, revert model, herdr topology — deployment-independent |
| `TICK.md` | tool repo | a `diwan tick` agent specifically | the 5-step tick procedure |
| `DO.md` | tool repo | a `diwan do` agent specifically | scope, chat protocol, commit discipline, how to stop |
| `AGENTS.md` | **content repo** | any dispatched diwan agent (tick or do) | that deployment's specifics: layout, org conventions already in use — read on top of `PROTOCOL.md`, never restates it |
| `herdr --skill` (external) | — | anyone scripting herdr | authoritative herdr CLI reference — re-check this if herdr's version changes, since `diwan`'s calls were verified against 0.9.5 specifically |
If you change a convention, update it in exactly one of these (whichever owns it) and make sure the others still only *point* at it rather than re-stating it — the duplication between `PROTOCOL.md` and any content repo's `AGENTS.md` is deliberately kept to zero; don't let that drift. Anything true of diwan regardless of which notes repo it's pointed at belongs in `PROTOCOL.md` (this repo); anything specific to one person's notes belongs in that content repo's own `AGENTS.md`, never here.
## What `diwan tick` actually does
Two halves: a deterministic CLI half (`cmd_tick` in `diwan`) and a judgment-driven agent half (`TICK.md`).
**CLI half**, in order:
1. If the content repo's working tree is dirty, commit everything under the user's own git identity (`chore: snapshot manual edits before tick`) — this is what makes "no manual commit requirement" (PROTOCOL.md) actually true.
2. Load the content repo's `.diwan/state.json`'s `last_tick_commit`. Diff that commit against current `HEAD` (in the content repo), excluding `.diwan/` itself (so diwan's own bookkeeping commits never count as "something changed"). No prior tick → treat as "always changed."
3. If nothing changed and `--force` wasn't passed: print and exit. This is the only gate on whether a tick does anything at all.
4. Otherwise, pick a tab name `tick-<UTC YYYYMMDD-HHMMSS>` (lowercase + digits + hyphens only — see PROTOCOL.md's herdr section for why that's not negotiable) and dispatch a fresh agent there, in a pane whose working directory is the *content* repo (`DIWAN_NOTES_DIR`), with a prompt that says: read `<DIWAN_DIR>/PROTOCOL.md` and `AGENTS.md` (the latter resolved relative to the agent's own cwd, i.e. the content repo), follow `<DIWAN_DIR>/TICK.md`, here's the commit range. **This dispatch is fire-and-forget** — the CLI call returns as soon as the prompt is submitted, it does not wait for the tick to finish.
5. Record the new tab name in the content repo's `state.json`'s `tick_history` (kept to the last `MAX_TICK_TABS`, default 3, override via `DIWAN_MAX_TICK_TABS`), commit that under the diwan identity, and — if this pushed any older tick tabs out of the retention window — close those tabs for real via `herdr tab close`.
Note what's *not* here: `last_tick_commit` is deliberately **not** updated by the CLI at dispatch time. It's updated by the dispatched agent itself, as the literal last step of `TICK.md`, only if everything else succeeded. If a tick agent crashes or is killed partway, the next tick sees the same "changed" range again rather than silently skipping work — correctness over tidiness.
**Agent half** (`TICK.md`), five steps, each its own commit:
1. Triage `inbox.org` only (other files changing is someone else's edit): events → `esdeveniments.org`, tasks → wherever they actually belong, `:diwan:` items get moved but not acted on yet, new topics only get a new file when nothing existing fits.
2. Tag consistency: any `:session:`/`:chat:` heading without a `:diwan:` tag on itself or an ancestor gets `:diwan:` added directly to it. Then resolve `:chat:` ownership: chats under a `:session:` ancestor are left for that heading's own session; orphan chats get answered inline if trivial, or escalated (see step 3's three-way decision) if not.
3. Handle every `:diwan:`-tagged heading without an open `:session:`: trivial ones get answered inline, no tag beyond `:diwan:` ever assigned. Non-trivial ones split two ways — self-contained within the heading's own subtree gets a `:session:` tag and `diwan do <file>:<line> --diwan-id <slug>` run *by the tick agent itself* (the tick agent is the only thing that ever originates a fresh `DIWAN_ID`); inherently cross-cutting work (spans multiple files/headings, can't be parallelized — archiving, bulk cleanup, moving things between files) is done by the tick agent *directly*, right there, no session at all, possibly spread across several tick runs if it needs Marc's input first. The same handling applies to any open item under `diwan.org`'s `Tasks` heading — diwan's own self-maintenance backlog, no tag needed there since the location already signals it. Already-open sessions (`:session:` tag + `DIWAN_ID` present, `Close diwan session` still `TODO`) get their subtree's content left alone as always, but are checked for a nudge: if any commit in this tick's change range *not* authored by the diwan git identity touched a line inside that subtree, the tick agent re-runs `diwan do <file>:<line> --diwan-id <id>` on the existing ID to wake it up; no nudge at all if nothing hand-authored touched the subtree. A session whose `Close diwan session` is marked `DONE` gets retired: the tick agent closes its `DIWAN_ID`'s herdr tab (`herdr tab close`) and removes the `:session:` tag from the heading, leaving the `DIWAN_ID` property in place as a historical record.
4. Delete any already-`DONE` Reports subheadings first, then write one short, titled `TODO` subheading per reportable thing this run did into `diwan.org`'s Reports log — the bar is deliberately broad (items moved, dispatches, closures, inline answers, findings, notes left for Marc to read), excluding only a run where nothing happened at all and a bare nudge (no information yet on whether it did anything) — each with the title itself linking to the one heading it's about, and a 1-2 line signed body.
5. Only if 1–4 all succeeded: update `state.json.last_tick_commit` to current `HEAD` and commit under the diwan identity.
**Verified live**, repeatedly, against the packaged flake build and `diwan-dev-notes`: the no-op gate (an immediate second `tick` after one that found nothing new correctly does nothing), triage, the tag-consistency/escalation decision in step 3 (confirmed doing the right thing for a cross-cutting archive task and for a stale leftover agent it correctly declined to nudge), and Reports writing. **Not yet verified live**: the current (broadened) Reports bar and `TODO`/`DONE` format, and the session-closure teardown path (nobody has checked off a `Close diwan session` and had a tick actually process it yet) — both designed and committed, but no real run has exercised either.
## What `diwan do` actually does
Same split: `cmd_do` (CLI) decides identity/dispatch mechanics; `DO.md` governs what the agent does once running.
**CLI half** (`cmd_do`):
1. Parse `target`: if it matches `<existing file>:<line number>` (resolved against the content repo), it's heading mode; anything else is treated as an ad-hoc free-text prompt with no org heading involved (useful for manual testing, or tasks that genuinely don't belong in the notes).
2. **Heading mode:** read the headline at that line, check its `DIWAN_ID` property.
- Already has one → **continuation**. (If `--diwan-id` was also passed and disagrees with the stored one, that's an error, not a silent overwrite — a mismatch means something's confused and a human should look.) The existing ID is validated against herdr's naming rule before anything else happens, so a corrupted/hand-edited property fails with a clear message instead of a confusing herdr-side error later.
- No `DIWAN_ID` yet → **fresh**. Compute one: `--diwan-id` if given, else a slug of the heading text (`slugify`, capped and shaped to satisfy herdr's agent-name rule — see below), de-duplicated against every other `DIWAN_ID` already in the content repo's tree (`unique_id`). Write the property and insert `** TODO Close diwan session` as the first child (`assign_diwan_id`), in one commit under the diwan identity. This is the *only* place a `DIWAN_ID` is minted or a Close-session item is inserted — mechanical, not left to agent judgment, so it's always consistent.
3. **Ad-hoc mode:** `diwan_id` is `--diwan-id` or a generated `adhoc-<8 hex chars>`; always goes through the continuation dispatch path (see below) since there's no heading state to distinguish fresh from resumed.
4. Dispatch: fresh headings use `herdr_dispatch_fresh`; everything else (continuation headings, ad-hoc) uses `herdr_dispatch_continuation`. Prints the `diwan_id` either way — this is the command's only stdout contract, which is what lets `TICK.md` step 3 just run `diwan do ...` as an ordinary shell command and know what session it created.
**Continuation dispatch, concretely** (`herdr_dispatch_continuation`): check `herdr agent get <id>` — alive → just `agent prompt <id> <text>` (fire-and-forget, no `--wait`). Not alive (the normal case between ticks; CLI agent processes exit, they don't idle forever) → look up the tab by label in the `diwan` workspace, start a fresh agent in its existing root pane, prompt it. Tab itself gone too (e.g. closed by hand) → falls through to the same fresh-tab path `herdr_dispatch_fresh` uses. **Continuity across resumptions is carried by the heading's own content, not by agent conversation memory** — a resumed `do` agent has no idea what it said last time except what's written in the org file. This is a real, current limitation, not an oversight to "fix" without thinking about it — it's a direct consequence of principle #1 (org is ground truth) and keeps the system's state in exactly one place.
**Agent half** (`DO.md`): stay inside your own heading's subtree; if something needs doing elsewhere, ask in `:chat:` rather than reaching out of scope; use the chat protocol for questions/FYIs; commit progress in small units under the diwan identity through the flock-wrapped recipe; never touch your own `Close diwan session` item — that's Marc's call, acted on by the next tick. One explicit exception to subtree-only: a `do` agent may append its own entry to `diwan.org`'s Reports log once it finishes something worth telling Marc about, since that's its only channel to him outside of a tick run — it still never edits or deletes an existing entry there, only `tick` does that.
**Verified live**: real web research dispatched against a heading, writing a structured result into it and asking clarifying questions via a signed `:chat:` entry instead of guessing, committed under the diwan identity. Ad-hoc dispatch and continuation (reusing the same pane/tab/agent on a second call rather than creating a new one) are both confirmed working, including cold-starting in a content repo `pi` had never run in before (no trust-dialog block — unlike some other harnesses, `pi` doesn't have one). **Not yet verified live**: `diwan do` in heading mode (fresh `DIWAN_ID` minting) specifically with `pi` as the dispatched agent — the heading-mode mechanics have only been exercised end-to-end via a tick's own self-contained-work dispatch, not tested in isolation since the pi-only switch.
## `DIWAN_ID` ⇄ herdr, precisely
A `DIWAN_ID` *is* a herdr tab label in the `diwan` workspace, and (while its agent process is alive) also that agent's name. This is why it must satisfy herdr's agent-name rule: `^[a-z][a-z0-9_-]{0,31}$`. `slugify()` enforces this at mint time (lowercase, non-alnum → hyphen, truncated to 24 chars, forced to start with a letter) and `cmd_do` additionally validates *every* `DIWAN_ID` it's about to dispatch — including ones already stored in a heading — against that same regex before touching herdr, erroring out with a clear message rather than letting herdr reject it after the CLI has already created a tab (which used to leave an orphan tab behind; fixed by closing the just-created tab on any `agent start` failure).
Lifecycle: **fresh** (no property) → `diwan do` mints an ID, creates the tab+agent, heading now has `DIWAN_ID` + open (`TODO`) `Close diwan session` → **active** (any further `tick`/`do` on this heading is a continuation; `tick` explicitly leaves active sessions alone) → Marc marks `Close diwan session` `DONE` → next `tick` retires it: closes the herdr tab, removes the `:session:` tag, leaves the `DIWAN_ID` property in place as a historical record — **retired** (the heading can still be found later by that `DIWAN_ID`, but there's no live terminal and no tag pointing at one).
## Known gaps / explicit non-goals (don't "fix" these without discussing)
- **Tick scheduling beyond a plain systemd timer.** `samfelag`'s module runs `diwan tick` on a configurable `OnCalendar` cadence — that's it, no smarter scheduling, no overlap detection between runs. Fine for a personal single-host setup; revisit only if that stops being true.
- **Retired-session teardown is designed but not yet verified live.** `TICK.md` step 3 has the tick agent close the herdr tab (`herdr tab close`) and strip the `:session:` tag once `Close diwan session` is marked `DONE` — but no real tick run has exercised this path yet. Worth a close look the first few times it fires: does the tab lookup by `DIWAN_ID` label reliably find the right tab, does tag-stripping leave `:diwan:`/other tags intact, does it behave sanely if the tab was already closed by hand.
- **The new (broadened) Reports bar and `TODO`/`DONE` format are unverified live.** Designed and committed, but no real tick run has written a Reports entry under the current rules yet.
- **No real conversation continuity across resumption.** Covered above — intentional for now, but worth revisiting if resumed `do` sessions turn out to need more context than the heading text carries.
- **`diwan do` in heading mode hasn't been exercised in isolation since the pi-only switch.** Fresh `DIWAN_ID` minting works as part of a tick's own dispatch decision, but hasn't been separately smoke-tested the way ad-hoc dispatch and continuation have.
- **`diwan remote <machine>` is implemented but not verified against a real remote herdr server.** Argument parsing, the no-herdr-on-`PATH` fallback, and a fast failure on an unknown `--machine` label are all confirmed; an actual `herdr machine add` + `diwan remote` round trip against a live remote server is not.
- **`diwan.org`'s `Config` heading is a placeholder, not wired up.** PROTOCOL.md documents an intended home for configuration values governing diwan's own behavior (tick cadence, tab retention, that kind of thing), but the CLI reads none of it — the env vars (`DIWAN_DIR`, `DIWAN_NOTES_DIR`, `DIWAN_PI_APPROVE`, `DIWAN_MAX_TICK_TABS`) remain the only real knobs. Don't assume anything in that heading does something until the CLI actually reads it.
- **Concurrency is scope discipline, not real locking**, beyond the git-commit flock. Two `do` agents editing truly disjoint subtrees should be fine; nothing stops two agents editing the *same* subtree if the scope rule is violated. There's no enforcement, only convention (PROTOCOL.md).
- **No automated tests.** Everything described as "verified live" above was exercised by hand against a real herdr + `pi` install. There is no regression suite — a future session changing `diwan` should re-run at least the manual smoke tests (tick no-op gate, tick dispatch, do fresh dispatch, do continuation, a deliberately-invalid `DIWAN_ID`) before trusting a change.