Files
diwan/DESIGN.md
T
marc 19e6ae3a59 Reports: TODO/DONE format instead of checkboxes, much broader reportability bar
Format: '** TODO <title>' unread, '** DONE <title>' read -- tick deletes
DONE entries at the start of its next run (previously [X]). This only
changes the Reports heading's own keyword; :chat: entries and Close
diwan session stay checkbox-based, unaffected.

Reportability: reversed last turn's narrow 'only real findings' bar.
The report is now explicitly meant to be the one place Marc checks for
everything new or needing attention -- so triage moves, new dispatches,
session closures, and inline answers are reportable again, alongside
findings/ambiguities/problems and anywhere a note was left for Marc to
read (even one the agent already marked [X] itself -- that's not the
same as Marc having seen it). Only two exclusions remain: a run where
nothing happened at all, and a bare nudge (no information yet on
whether anything came of it).

No code changes -- this is PROTOCOL.md/TICK.md/DO.md/DESIGN.md only.
Also cleaned up one stale 'did nothing' entry in diwan-dev-notes'
diwan.org that violated the rule even under the old wording.
2026-11-09 23:08:42 +01:00

26 KiB
Raw Blame History

DESIGN.md — diwan

Audience: whoever (human or agent) picks up development of diwan itself next. If you're an agent dispatched by diwan to triage notes or do delegated work, you want PROTOCOL.md (given to you directly in your dispatch prompt) and the content repo's own AGENTS.md instead — this doc is one level up, about the tool, not about using the tool.

What this is

diwan is an AI layer over a plain org-mode "second brain" (notes, tasks, calendar, projects — normally edited by hand in Emacs/beorg, synced via Nextcloud). It periodically triages a capture inbox, keeps files tidy, and runs delegated research/work when asked, while treating the org files as the only real source of truth.

This repo is diwan the tool, and nothing else. It contains the CLI and the generic procedure docs (PROTOCOL.md, TICK.md, DO.md) — no personal notes, no content-repo-specific convention docs. It's meant to be installed independently (on PATH, eventually via its own nix flake) and pointed at any notes repo. Marc's actual notes live in ~/nc/org-agenda (the real, continuously-synced data) and, for development/testing, in a throwaway sibling repo diwan-dev-notes (a git-tracked fixture copy, safe to break). Neither of those repos contains any of diwan's own code or docs — see "Known gaps" for exactly how far that packaging currently goes.

Status as of this writing: the core loop (tick → triage/dispatch, do → delegated work, both via real herdr sessions) is built and has been exercised end-to-end against live herdr. The tool/content split (this repo vs. diwan-dev-notes) happened 2026-11-09, and the harness was narrowed to pi exclusively right after that (dropping Claude Code support entirely) — see "Known gaps" for what both of those still owe a fresh live pass on.

Principles (why things are shaped this way)

These were distilled from how Marc already works and what he explicitly asked for; they're the standard to hold any future change to:

  1. Org files are ground truth, not a cache. If diwan vanished, the notes still work in Emacs/beorg. Nothing diwan needs lives outside .org files except the content repo's own .diwan/state.json, which is pure bookkeeping, not content.
  2. Act first, review passively. No per-edit approval gate — every action is reported and trivially revertable instead.
  3. Mechanism over magic. Small, inspectable, deterministic steps over one opaque "agent did something" blob — Marc wants to be able to understand a failure well enough to jump in personally.
  4. Grow by accretion. Ship the smallest loop that works; add complexity only once the simpler version has been lived with. (Concretely: v0 has no custom CLI-level org-tag manipulation, no scheduler, no kill-on-timeout logic — these are deliberately not yet built.)
  5. One harness, chosen deliberately: pi. diwan dispatches pi and only pi — not a pluggable choice. Claude Code was the original harness and worked, but needed --permission-mode auto to avoid prompting for approval on every tool call, and blocks on an interactive first-run trust dialog that herdr's real-pane dispatch can't answer (hit live 2026-11-09 against the fresh diwan-dev-notes content repo). pi has no per-tool-call approval gate and no such trust dialog (only a one-time, non-interactive-safe project-trust decision for protected .pi/* resources, handled by --approve) — it was built to run unattended from the start, which is exactly diwan's own model. The procedure docs (TICK.md/DO.md) still don't hardcode anything pi-specific into their wording, as a matter of good practice, but there's no expectation of swapping harnesses again.
  6. Two symmetric channels, one substrate. Talk to diwan via the notes (:chat:) or via the live herdr session — both write to the same git-tracked truth.
  7. Mobile stays dumb. Nothing new to install on the phone.
  8. Tool and content are separate repos. diwan the tool (this repo) ships independent of any one person's notes; a content repo (e.g. diwan-dev-notes, or Marc's real ~/nc/org-agenda) only ever contains notes, AGENTS.md, diwan.org, and diwan's own tiny .diwan/ runtime-bookkeeping directory (state.json, .lock) — never the CLI or the procedure docs. This is what makes the content repo distributable (synced via Nextcloud to a server, edited from beorg) without dragging a whole dev tool along with it, and what keeps diwan's own generic contract reviewable/versionable independent of any one person's data.

Architecture at a glance

Three layers, each independently simple:

  • Git, in the content repo, no remote (for Marc's prototype content repo — a real deployment may add one, e.g. for Nextcloud sync, but that's the content repo's business, not this one's). Every automated write to the content repo is a commit. This is the revert mechanism (principle #2) — not a bespoke undo log. Commits are authored as diwan <diwan@local> when diwan itself is the actor, or the user's own git identity when diwan tick snapshots whatever manual edits were lying around at the start of a run (so no tick ever operates on a dirty/mixed workspace, and "who changed this" is answerable from git log --author alone). This repo (the tool) has its own, unrelated git history — just ordinary source control for diwan's own development.
  • herdr, a terminal-multiplexer-for-agents with a JSON-over-socket CLI (herdr workspace/tab/pane/agent ...). diwan doesn't invent a session store: one herdr workspace labeled diwan holds every tick run and every open do session as a separate, independently addressable, persistent tab. See "herdr integration" below for the exact calls.
  • The diwan CLI (./diwan at this repo's root), a single dependency-free Python script (shebang runs it through nix shell nixpkgs#python3, so no system Python install needed on NixOS; a real flake package would wrap this properly instead). It does only the mechanical, deterministic parts: git bookkeeping, enough org parsing to read/write a DIWAN_ID property and insert a Close diwan session checkbox item, and herdr dispatch. It never triages, writes prose, or makes judgment calls — that's the dispatched agent's job, guided by TICK.md/DO.md. It distinguishes two roots explicitly (see the script's own docstring): DIWAN_DIR (where it and its docs are installed — Path(__file__).resolve().parent, overridable via the DIWAN_DIR env var for whatever a future flake's install layout needs) and DIWAN_NOTES_DIR (whatever notes repo it's invoked against, found via git rev-parse --show-toplevel from the current directory by default, overridable via the DIWAN_NOTES_DIR env var for e.g. a cron/systemd timer that doesn't cd into it first). diwan must be run from inside (or below) a content repo, or have DIWAN_NOTES_DIR set — it has no content of its own to act on.

Everything else (what counts as "trivial", where an inbox item belongs, how to answer a chat question) is intentionally not code — it's prose in TICK.md/DO.md that the dispatched pi agent has to interpret. The CLI's job ends at "here's a prompt, here's a terminal"; everything judgment-shaped is left to the model.

Doc map

File Lives in Audience Contents
DESIGN.md (this file) tool repo diwan developers why, architecture, current implementation status, known gaps
PROTOCOL.md tool repo any dispatched diwan agent (tick or do) diwan's generic contract: tags, diwan.org, concurrency contract, revert model, herdr topology — deployment-independent
TICK.md tool repo a diwan tick agent specifically the 5-step tick procedure
DO.md tool repo a diwan do agent specifically scope, chat protocol, commit discipline, how to stop
AGENTS.md content repo any dispatched diwan agent (tick or do) that deployment's specifics: layout, org conventions already in use — read on top of PROTOCOL.md, never restates it
herdr --skill (external) — anyone scripting herdr authoritative herdr CLI reference — re-check this if herdr's version changes, since diwan's calls were verified against 0.9.5 specifically

If you change a convention, update it in exactly one of these (whichever owns it) and make sure the others still only point at it rather than re-stating it — the duplication between PROTOCOL.md and any content repo's AGENTS.md is deliberately kept to zero; don't let that drift. Anything true of diwan regardless of which notes repo it's pointed at belongs in PROTOCOL.md (this repo); anything specific to one person's notes belongs in that content repo's own AGENTS.md, never here.

What diwan tick actually does

Two halves: a deterministic CLI half (cmd_tick in diwan) and a judgment-driven agent half (TICK.md).

CLI half, in order:

  1. If the content repo's working tree is dirty, commit everything under the user's own git identity (chore: snapshot manual edits before tick) — this is what makes "no manual commit requirement" (PROTOCOL.md) actually true.
  2. Load the content repo's .diwan/state.json's last_tick_commit. Diff that commit against current HEAD (in the content repo), excluding .diwan/ itself (so diwan's own bookkeeping commits never count as "something changed"). No prior tick → treat as "always changed."
  3. If nothing changed and --force wasn't passed: print and exit. This is the only gate on whether a tick does anything at all.
  4. Otherwise, pick a tab name tick-<UTC YYYYMMDD-HHMMSS> (lowercase + digits + hyphens only — see "herdr integration" for why that's not negotiable) and dispatch a fresh agent there (herdr_dispatch_fresh), in a pane whose working directory is the content repo (DIWAN_NOTES_DIR), with a prompt that says: read <DIWAN_DIR>/PROTOCOL.md and AGENTS.md (the latter resolved relative to the agent's own cwd, i.e. the content repo), follow <DIWAN_DIR>/TICK.md, here's the commit range. This dispatch is fire-and-forget — the CLI call returns as soon as the prompt is submitted, it does not wait for the tick to finish.
  5. Record the new tab name in the content repo's state.json's tick_history (kept to the last MAX_TICK_TABS, default 3, override via DIWAN_MAX_TICK_TABS), commit that under the diwan identity, and — if this pushed any older tick tabs out of the retention window — close those tabs for real via herdr tab close.

Note what's not here: last_tick_commit is deliberately not updated by the CLI at dispatch time. It's updated by the dispatched agent itself, as the literal last step of TICK.md, only if everything else succeeded. If a tick agent crashes or is killed partway, the next tick sees the same "changed" range again rather than silently skipping work — correctness over tidiness.

Agent half (TICK.md), five steps, each its own commit:

  1. Triage inbox.org only (other files changing is someone else's edit): events → esdeveniments.org, tasks → wherever they actually belong, :diwan: items get moved but not acted on yet, new topics only get a new file when nothing existing fits.
  2. Tag consistency: any :session:/:chat: heading without a :diwan: tag on itself or an ancestor gets :diwan: added directly to it. Then resolve :chat: ownership: chats under a :session: ancestor are left for that heading's own session; orphan chats get answered inline if trivial, or escalated (see step 3's three-way decision) if not.
  3. Handle every :diwan:-tagged heading without an open :session:: trivial ones get answered inline, no tag beyond :diwan: ever assigned. Non-trivial ones split two ways — self-contained within the heading's own subtree gets a :session: tag and diwan do <file>:<line> --diwan-id <slug> run by the tick agent itself (the tick agent is the only thing that ever originates a fresh DIWAN_ID); inherently cross-cutting work (spans multiple files/headings, can't be parallelized — archiving, bulk cleanup, moving things between files) is done by the tick agent directly, right there, no session at all, possibly spread across several tick runs if it needs Marc's input first. The same handling applies to any open item under diwan.org's Tasks heading — diwan's own self-maintenance backlog, no tag needed there since the location already signals it (see PROTOCOL.md's diwan.org section). Already-open sessions (:session: tag + DIWAN_ID present, Close diwan session still [ ]) get their subtree's content left alone as always, but are checked for a nudge: if any commit in this tick's change range not authored by the diwan git identity touched a line inside that subtree (a hand-authored heading edit, a new unread :chat: entry, anything Marc did), the tick agent re-runs diwan do <file>:<line> --diwan-id <id> on the existing ID — a continuation dispatch into the session's own tab/agent, not a fresh one — purely to wake it up; no nudge at all if nothing hand-authored touched the subtree. This is the only mechanism that gets a change Marc makes to an active heading in front of its own idle agent — nothing watches the file otherwise. A session whose Close diwan session is already [X] gets retired for real: the tick agent looks up its DIWAN_ID's herdr tab and actually closes it (herdr tab close), then removes the :session: tag from the heading — the DIWAN_ID property itself is left in place as a historical record. Dispatch and closure both get a Reports entry (new/changed state Marc hasn't seen); a bare nudge does not, not because it's unimportant but because there's no information yet on whether it did anything — see step 4.
  4. Delete any already-DONE Reports subheadings first, then write one short, titled TODO subheading per reportable thing this run did into diwan.org's Reports log (not one paragraph for the whole run) — the bar is deliberately broad (items moved, dispatches, closures, inline answers, findings, notes left for Marc to read), excluding only a run where nothing happened at all and a bare nudge (no information yet on whether it did anything) — each with the title itself linking to the one heading it's about (plain text if there isn't one) and a 1-2 line signed body.
  5. Only if 1–4 all succeeded: update state.json.last_tick_commit to current HEAD and commit under the diwan identity.

Verified live (2026-11-08, herdr 0.9.5 + Claude Code, before both the 2026-11-09 tool/content split and the pi-only harness switch): dispatched for real, ran to completion with zero permission prompts, correctly left two already-open :session:-tagged sessions untouched (back when :session: was still spelled :diwan:, before the tag rename — see "Known gaps" for what's unverified post-rename), correctly found nothing to triage, wrote a proper signed report, and updated state.json correctly. A second immediate diwan tick correctly no-op'd. Not yet re-verified with a real dispatched agent against the current state of things (post-split two-repo layout, pi as the sole harness) — path resolution and pi's own startup have each been tested separately (see below and "Known gaps"), but not yet together through a full live tick run. Do that before trusting it for real.

What diwan do actually does

Same split: cmd_do (CLI) decides identity/dispatch mechanics; DO.md governs what the agent does once running.

CLI half (cmd_do):

  1. Parse target: if it matches <existing file>:<line number> (resolved against the content repo), it's heading mode; anything else is treated as an ad-hoc free-text prompt with no org heading involved (useful for manual testing, or tasks that genuinely don't belong in the notes).
  2. Heading mode: read the headline at that line, check its DIWAN_ID property.
    • Already has one → continuation. (If --diwan-id was also passed and disagrees with the stored one, that's an error, not a silent overwrite — a mismatch means something's confused and a human should look.) The existing ID is validated against herdr's naming rule before anything else happens, so a corrupted/hand-edited property fails with a clear message instead of a confusing herdr-side error later.
    • No DIWAN_ID yet → fresh. Compute one: --diwan-id if given, else a slug of the heading text (slugify, capped and shaped to satisfy herdr's agent-name rule — see below), de-duplicated against every other DIWAN_ID already in the content repo's tree (unique_id). Write the property and insert ** [ ] Close diwan session as the first child (assign_diwan_id), in one commit under the diwan identity. This is the only place a DIWAN_ID is minted or a Close-session item is inserted — mechanical, not left to agent judgment, so it's always consistent.
  3. Ad-hoc mode: diwan_id is --diwan-id or a generated adhoc-<8 hex chars>; always goes through the continuation dispatch path (see below) since there's no heading state to distinguish fresh from resumed.
  4. Dispatch: fresh headings use herdr_dispatch_fresh; everything else (continuation headings, ad-hoc) uses herdr_dispatch_continuation. Prints the diwan_id either way — this is the command's only stdout contract, which is what lets TICK.md step 3 just run diwan do ... as an ordinary shell command and know what session it created.

Continuation dispatch, concretely (herdr_dispatch_continuation): check herdr agent get <id> — alive → just agent prompt <id> <text> (fire-and-forget, no --wait). Not alive (the normal case between ticks; CLI agent processes exit, they don't idle forever) → look up the tab by label in the diwan workspace, start a fresh agent in its existing root pane, prompt it. Tab itself gone too (e.g. closed by hand) → falls through to the same fresh-tab path herdr_dispatch_fresh uses. Continuity across resumptions is carried by the heading's own content, not by agent conversation memory — a resumed do agent has no idea what it said last time except what's written in the org file. This is a real, current limitation, not an oversight to "fix" without thinking about it — it's a direct consequence of principle #1 (org is ground truth) and keeps the system's state in exactly one place.

Agent half (DO.md): stay inside your own heading's subtree; if something needs doing elsewhere, ask in :chat: rather than reaching out of scope (cross-cutting work like this is now tick's job to do directly, per the :diwan: tag section — don't try to talk another do session into doing it either); use the chat protocol for questions/FYIs; commit progress in small units under the diwan identity through the flock-wrapped recipe; never touch your own Close diwan session item — that's Marc's call, acted on by the next tick. One explicit exception to subtree-only: a do agent may append its own entry to diwan.org's Reports log once it finishes something worth telling Marc about, since that's its only channel to him outside of a tick run — it still never edits or deletes an existing entry there, only tick does that. A tick that merely nudges an idle session (see cmd_tick step 3 above) does not report the nudge itself — it has no idea yet whether anything came of it.

Verified live with Claude Code (pre tool/content split, pre pi-only switch): dispatched for real against the "gather info about Greece trip" heading. Did real web research, wrote a structured result into the heading, correctly asked clarifying questions via a signed :chat: entry instead of guessing, left Close diwan session untouched, committed as diwan <diwan@local> with a descriptive message. (Historical — Claude Code is no longer a supported harness; see "One harness, chosen deliberately" above.)

pi, verified live, ad-hoc dispatch only (2026-11-08, herdr 0.9.5 + pi 1.5.0, pre-split; re-confirmed 2026-11-09 post-split, post-switch against the fresh diwan-dev-notes repo): diwan do "<prompt>" --diwan-id <id> correctly starts a herdr pi agent (detected as agent: "pi"), with no trust/approval prompt blocking it even in a content repo pi had never seen before (confirming it really doesn't share Claude Code's first-run trust-dialog problem — see "Known gaps"), loads AGENTS.md as context, and replies exactly as instructed. A second diwan do call against the same --diwan-id reuses the same pane/tab and agent (continuation dispatch, not a fresh one) and replies correctly again. Not yet exercised: a real tick or do run against actual heading content with pi as the agent (only ad-hoc free-text prompts so far) — worth a full pass before relying on it for real delegated work.

DIWAN_ID ⇄ herdr, precisely

A DIWAN_ID is a herdr tab label in the diwan workspace, and (while its agent process is alive) also that agent's name. This is why it must satisfy herdr's agent-name rule: ^[a-z][a-z0-9_-]{0,31}$. slugify() enforces this at mint time (lowercase, non-alnum → hyphen, truncated to 24 chars, forced to start with a letter) and cmd_do additionally validates every DIWAN_ID it's about to dispatch — including ones already stored in a heading — against that same regex before touching herdr, erroring out with a clear message rather than letting herdr reject it after the CLI has already created a tab (which used to leave an orphan tab behind; fixed by closing the just-created tab on any agent start failure).

Lifecycle: fresh (no property) → diwan do mints an ID, creates the tab+agent, heading now has DIWAN_ID + open ([ ]) Close diwan session → active (any further tick/do on this heading is a continuation; tick explicitly leaves active sessions alone) → Marc checks off Close diwan session ([X]) → next tick actually retires it (per TICK.md step 3): closes the herdr tab for real, removes the :session: tag, leaves the DIWAN_ID property in place as a historical record — retired (the heading can still be found later by that DIWAN_ID, but there's no live terminal and no tag pointing at one).

Known gaps / explicit non-goals (don't "fix" these without discussing)

  • Tool/content repo split just happened (2026-11-09), not yet re-verified live. This repo used to be both diwan's own dev copy and a throwaway test-content repo at once; it's now tool-only, with content split out to a sibling diwan-dev-notes repo. The CLI was updated to resolve DIWAN_DIR (its own install dir, Path(__file__).resolve().parent, overridable via the DIWAN_DIR env var) separately from DIWAN_NOTES_DIR (git rev-parse --show-toplevel from the invoking directory by default, overridable via the DIWAN_NOTES_DIR env var), and the dispatch prompts now pass DIWAN_DIR-absolute paths for PROTOCOL.md/TICK.md/DO.md. This has been syntax-checked and smoke-tested for path resolution (including both env var overrides, from outside any repo), but not yet run through a full live tick/do pass with a real dispatched agent against the new two-repo layout — do that before trusting it for real.
  • No flake yet. The split prepared the ground (tool repo is self-contained, no content-repo assumptions baked into its own files) but there's no flake.nix in this repo yet, and the shebang still runs via a bare nix shell nixpkgs#python3 rather than a proper package. Until that lands, "installed on PATH" means manually symlinking/aliasing diwan somewhere on PATH yourself. DIWAN_DIR exists specifically so a future flake's install layout (e.g. docs copied into $out/share/diwan/ alongside a wrapped binary) can override the default Path(__file__).resolve().parent guess if it doesn't hold.
  • Retired-session teardown is designed but not yet verified live. TICK.md step 3 now has the tick agent actually close the herdr tab (herdr tab close) and strip the :session: tag once Close diwan session is checked off, rather than just reporting that it happened — but no real tick run has exercised this path yet. Worth a close look the first few times it fires: does the tab lookup by DIWAN_ID label reliably find the right tab, does tag-stripping leave :diwan:/other tags intact, does it behave sanely if the tab was already closed by hand.
  • No real conversation continuity across resumption. Covered above — intentional for now, but worth revisiting if resumed do sessions turn out to need more context than the heading text carries.
  • pi is smoke-tested for ad-hoc dispatch, but not yet exercised against real heading content. --approve has been confirmed live for ad-hoc diwan do dispatch and continuation, including against a content repo pi had never run in before (no trust-dialog block, unlike Claude Code — see "One harness, chosen deliberately" above). Still untested: a real diwan tick run, and diwan do against a real <file>:<line> heading (DIWAN_ID minting, :chat: protocol, scope discipline) rather than a free-text ad-hoc prompt.
  • diwan.org's Config heading is a placeholder, not wired up. PROTOCOL.md documents an intended home for configuration values governing diwan's own behavior (tick cadence, tab retention, that kind of thing), but the CLI reads none of it — the env vars (DIWAN_DIR, DIWAN_NOTES_DIR, DIWAN_PI_APPROVE, DIWAN_MAX_TICK_TABS) remain the only real knobs. Don't assume anything in that heading does something until the CLI actually reads it.
  • The :diwan:/:session:/:chat: tag split, and tick doing cross-cutting maintenance directly, is new and unverified live. Before this split, :diwan: alone meant what :session: means now, and anything non-trivial got a dedicated do session regardless of whether the work was subtree-shaped or genuinely repo-wide — which is exactly what motivated the split (a real do session hit the subtree wall trying to do an archive/cleanup/folder-move task). No tick run has yet exercised the new three-way decision in TICK.md step 3 (inline / :session: dispatch / tick-direct), the tag-consistency backfill in step 2, or tick actually doing a multi-file maintenance job itself across one or more runs — worth a close look at the first few real runs that hit this path.
  • Concurrency is scope discipline, not real locking, beyond the git-commit flock. Two do agents editing truly disjoint subtrees should be fine; nothing stops two agents editing the same subtree if the scope rule is violated. There's no enforcement, only convention (PROTOCOL.md).
  • No automated tests. Everything described as "verified live" above was exercised by hand against a real herdr install. There is no regression suite — a future session changing diwan should re-run at least the manual smoke tests (tick no-op gate, tick dispatch, do fresh dispatch, do continuation, a deliberately-invalid DIWAN_ID) before trusting a change.