From fc0e556b4235c9199ca409e7843400a919ddb76d Mon Sep 17 00:00:00 2001 From: marc Date: Mon, 9 Nov 2026 10:13:24 +0100 Subject: [PATCH] diwan: the tool, split out from personal notes content Contains only the generic diwan CLI and procedure docs (diwan, PROTOCOL.md, TICK.md, DO.md, DESIGN.md). No personal notes, no content-repo-specific AGENTS.md -- those now live in a separate content repo (e.g. diwan-dev-notes for testing, ~/nc/org-agenda for the real thing), discovered at runtime via git rev-parse --show-toplevel from wherever diwan is invoked. History reset: this repo previously doubled as both diwan's own dev copy and a throwaway test-content repo, with 48 commits of the two tangled together. Starting fresh now that the split is done -- still v0/prototyping. --- .gitignore | 1 + DESIGN.md | 108 ++++++++++ DO.md | 31 +++ PROTOCOL.md | 130 +++++++++++++ TICK.md | 61 ++++++ diwan | 552 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 6 files changed, 883 insertions(+) create mode 100644 .gitignore create mode 100644 DESIGN.md create mode 100644 DO.md create mode 100644 PROTOCOL.md create mode 100644 TICK.md create mode 100755 diwan diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..d6b130c --- /dev/null +++ b/.gitignore @@ -0,0 +1 @@ +.claude/settings.local.json diff --git a/DESIGN.md b/DESIGN.md new file mode 100644 index 0000000..ceeca22 --- /dev/null +++ b/DESIGN.md @@ -0,0 +1,108 @@ +# DESIGN.md — diwan + +Audience: whoever (human or agent) picks up **development of diwan itself** next. If you're an agent dispatched *by* diwan to triage notes or do delegated work, you want `PROTOCOL.md` (given to you directly in your dispatch prompt) and the content repo's own `AGENTS.md` instead — this doc is one level up, about the tool, not about using the tool. + +## What this is + +diwan is an AI layer over a plain org-mode "second brain" (notes, tasks, calendar, projects — normally edited by hand in Emacs/beorg, synced via Nextcloud). It periodically triages a capture inbox, keeps files tidy, and runs delegated research/work when asked, while treating the org files as the only real source of truth. + +**This repo is diwan the tool, and nothing else.** It contains the CLI and the generic procedure docs (`PROTOCOL.md`, `TICK.md`, `DO.md`) — no personal notes, no content-repo-specific convention docs. It's meant to be installed independently (on `PATH`, eventually via its own nix flake) and pointed at *any* notes repo. Marc's actual notes live in `~/nc/org-agenda` (the real, continuously-synced data) and, for development/testing, in a throwaway sibling repo `diwan-dev-notes` (a git-tracked fixture copy, safe to break). Neither of those repos contains any of diwan's own code or docs — see "Known gaps" for exactly how far that packaging currently goes. + +Status as of this writing: the core loop (tick → triage/dispatch, do → delegated work, both via real herdr sessions) is built and has been exercised end-to-end against live `herdr` and `claude`. The tool/content split (this repo vs. `diwan-dev-notes`) happened 2026-11-09, right after that verification — see "Known gaps" for what that split still owes a fresh live pass on. + +## Principles (why things are shaped this way) + +These were distilled from how Marc already works and what he explicitly asked for; they're the standard to hold any future change to: + +1. **Org files are ground truth, not a cache.** If diwan vanished, the notes still work in Emacs/beorg. Nothing diwan needs lives outside `.org` files except the content repo's own `.diwan/state.json`, which is pure bookkeeping, not content. +2. **Act first, review passively.** No per-edit approval gate — every action is reported and trivially revertable instead. +3. **Mechanism over magic.** Small, inspectable, deterministic steps over one opaque "agent did something" blob — Marc wants to be able to understand a failure well enough to jump in personally. +4. **Grow by accretion.** Ship the smallest loop that works; add complexity only once the simpler version has been lived with. (Concretely: v0 has no custom CLI-level org-tag manipulation, no scheduler, no kill-on-timeout logic — these are deliberately not yet built.) +5. **Harness as a thin, swappable shell.** The procedure (what tick/do should do) and the primitives (how edits actually happen) are decoupled from which LLM product is driving. Claude Code today; `pi` is a supported herdr integration kind and should work by just changing `DIWAN_HARNESS`. +6. **Two symmetric channels, one substrate.** Talk to diwan via the notes (`:chat:`) or via the live herdr session — both write to the same git-tracked truth. +7. **Mobile stays dumb.** Nothing new to install on the phone. +8. **Tool and content are separate repos.** diwan the tool (this repo) ships independent of any one person's notes; a content repo (e.g. `diwan-dev-notes`, or Marc's real `~/nc/org-agenda`) only ever contains notes, `AGENTS.md`, `diwan.org`, and diwan's own tiny `.diwan/` runtime-bookkeeping directory (`state.json`, `.lock`) — never the CLI or the procedure docs. This is what makes the content repo distributable (synced via Nextcloud to a server, edited from beorg) without dragging a whole dev tool along with it, and what keeps diwan's own generic contract reviewable/versionable independent of any one person's data. + +## Architecture at a glance + +Three layers, each independently simple: + +- **Git**, in the *content* repo, no remote (for Marc's prototype content repo — a real deployment may add one, e.g. for Nextcloud sync, but that's the content repo's business, not this one's). Every automated write to the content repo is a commit. This *is* the revert mechanism (principle #2) — not a bespoke undo log. Commits are authored as `diwan ` when diwan itself is the actor, or the user's own git identity when `diwan tick` snapshots whatever manual edits were lying around at the start of a run (so no tick ever operates on a dirty/mixed workspace, and "who changed this" is answerable from `git log --author` alone). This repo (the tool) has its own, unrelated git history — just ordinary source control for diwan's own development. +- **herdr**, a terminal-multiplexer-for-agents with a JSON-over-socket CLI (`herdr workspace/tab/pane/agent ...`). diwan doesn't invent a session store: one herdr workspace labeled `diwan` holds every tick run and every open `do` session as a separate, independently addressable, persistent tab. See "herdr integration" below for the exact calls. +- **The `diwan` CLI** (`./diwan` at this repo's root), a single dependency-free Python script (shebang runs it through `nix shell nixpkgs#python3`, so no system Python install needed on NixOS; a real flake package would wrap this properly instead). It does only the mechanical, deterministic parts: git bookkeeping, enough org parsing to read/write a `DIWAN_ID` property and insert a `Close diwan session` checkbox item, and herdr dispatch. It never triages, writes prose, or makes judgment calls — that's the dispatched agent's job, guided by `TICK.md`/`DO.md`. It distinguishes two roots explicitly (see the script's own docstring): `TOOL_ROOT` (where it and its docs are installed — `Path(__file__).resolve().parent`, overridable via `DIWAN_TOOL_ROOT` for whatever a future flake's install layout needs) and `CONTENT_ROOT` (whatever notes repo it's invoked against, found via `git rev-parse --show-toplevel` from the current directory). `diwan` must be run from inside (or below) a content repo — it has no content of its own to act on. + +Everything else (what counts as "trivial", where an inbox item belongs, how to answer a chat question) is intentionally **not** code — it's prose in `TICK.md`/`DO.md` that whatever LLM is dispatched has to interpret. This is the harness-agnosticism principle made concrete: the CLI's job ends at "here's a prompt, here's a terminal," everything judgment-shaped is left to the model. + +## Doc map + +| File | Lives in | Audience | Contents | +|---|---|---|---| +| `DESIGN.md` (this file) | tool repo | diwan developers | why, architecture, current implementation status, known gaps | +| `PROTOCOL.md` | tool repo | any dispatched diwan agent (tick or do) | diwan's generic contract: tags, `diwan.org`, concurrency contract, revert model, herdr topology — deployment-independent | +| `TICK.md` | tool repo | a `diwan tick` agent specifically | the 5-step tick procedure | +| `DO.md` | tool repo | a `diwan do` agent specifically | scope, chat protocol, commit discipline, how to stop | +| `AGENTS.md` | **content repo** | any dispatched diwan agent (tick or do) | that deployment's specifics: layout, org conventions already in use — read on top of `PROTOCOL.md`, never restates it | +| `herdr --skill` (external) | — | anyone scripting herdr | authoritative herdr CLI reference — re-check this if herdr's version changes, since `diwan`'s calls were verified against 0.9.5 specifically | + +If you change a convention, update it in exactly one of these (whichever owns it) and make sure the others still only *point* at it rather than re-stating it — the duplication between `PROTOCOL.md` and any content repo's `AGENTS.md` is deliberately kept to zero; don't let that drift. Anything true of diwan regardless of which notes repo it's pointed at belongs in `PROTOCOL.md` (this repo); anything specific to one person's notes belongs in that content repo's own `AGENTS.md`, never here. + +## What `diwan tick` actually does + +Two halves: a deterministic CLI half (`cmd_tick` in `diwan`) and a judgment-driven agent half (`TICK.md`). + +**CLI half**, in order: +1. If the content repo's working tree is dirty, commit everything under the user's own git identity (`chore: snapshot manual edits before tick`) — this is what makes "no manual commit requirement" (PROTOCOL.md) actually true. +2. Load the content repo's `.diwan/state.json`'s `last_tick_commit`. Diff that commit against current `HEAD` (in the content repo), excluding `.diwan/` itself (so diwan's own bookkeeping commits never count as "something changed"). No prior tick → treat as "always changed." +3. If nothing changed and `--force` wasn't passed: print and exit. This is the only gate on whether a tick does anything at all. +4. Otherwise, pick a tab name `tick-` (lowercase + digits + hyphens only — see "herdr integration" for why that's not negotiable) and dispatch a fresh agent there (`herdr_dispatch_fresh`), in a pane whose working directory is the *content* repo, with a prompt that says: read `/PROTOCOL.md` and `AGENTS.md` (the latter resolved relative to the agent's own cwd, i.e. the content repo), follow `/TICK.md`, here's the commit range. **This dispatch is fire-and-forget** — the CLI call returns as soon as the prompt is submitted, it does not wait for the tick to finish. +5. Record the new tab name in the content repo's `state.json`'s `tick_history` (kept to the last `MAX_TICK_TABS`, default 3, override via `DIWAN_MAX_TICK_TABS`), commit that under the diwan identity, and — if this pushed any older tick tabs out of the retention window — close those tabs for real via `herdr tab close`. + +Note what's *not* here: `last_tick_commit` is deliberately **not** updated by the CLI at dispatch time. It's updated by the dispatched agent itself, as the literal last step of `TICK.md`, only if everything else succeeded. If a tick agent crashes or is killed partway, the next tick sees the same "changed" range again rather than silently skipping work — correctness over tidiness. + +**Agent half** (`TICK.md`), five steps, each its own commit: +1. Triage `inbox.org` only (other files changing is someone else's edit): events → `esdeveniments.org`, tasks → wherever they actually belong, `:diwan:` items get moved but not acted on yet, new topics only get a new file when nothing existing fits. +2. Tag consistency: any `:session:`/`:chat:` heading without a `:diwan:` tag on itself or an ancestor gets `:diwan:` added directly to it. Then resolve `:chat:` ownership: chats under a `:session:` ancestor are left for that heading's own session; orphan chats get answered inline if trivial, or escalated (see step 3's three-way decision) if not. +3. Handle every `:diwan:`-tagged heading without an open `:session:`: trivial ones get answered inline, no tag beyond `:diwan:` ever assigned. Non-trivial ones split two ways — self-contained within the heading's own subtree gets a `:session:` tag and `diwan do : --diwan-id ` run *by the tick agent itself* (the tick agent is the only thing that ever originates a fresh `DIWAN_ID`); inherently cross-cutting work (spans multiple files/headings, can't be parallelized — archiving, bulk cleanup, moving things between files) is done by the tick agent *directly*, right there, no session at all, possibly spread across several tick runs if it needs Marc's input first. The same handling applies to any open item under `diwan.org`'s `Tasks` heading — diwan's own self-maintenance backlog, no tag needed there since the location already signals it (see PROTOCOL.md's `diwan.org` section). Already-open sessions (`:session:` tag + `DIWAN_ID` present, `Close diwan session` still `[ ]`) get their subtree's content left alone as always, but are checked for a nudge: if any commit in this tick's change range *not* authored by the diwan git identity touched a line inside that subtree (a hand-authored heading edit, a new unread `:chat:` entry, anything Marc did), the tick agent re-runs `diwan do : --diwan-id ` on the existing ID — a continuation dispatch into the session's own tab/agent, not a fresh one — purely to wake it up; no nudge at all if nothing hand-authored touched the subtree. This is the only mechanism that gets a change Marc makes to an active heading in front of its own idle agent — nothing watches the file otherwise. +4. Write one short, titled, checkbox subheading per notable thing this run did into `diwan.org`'s Reports log (not one paragraph for the whole run), each with an optional `:LINK:` property and a 1-2 line signed body; delete any already-`[X]` subheadings first. +5. Only if 1–4 all succeeded: update `state.json.last_tick_commit` to current `HEAD` and commit under the diwan identity. + +**Verified live** (2026-11-08, herdr 0.9.5 + Claude Code, `claude` kind, against the old mixed tool+content repo, before the 2026-11-09 split): dispatched for real, ran to completion with zero permission prompts (see "Known gaps" for why that almost didn't work), correctly left two already-open `:session:`-tagged sessions untouched (back when `:session:` was still spelled `:diwan:`, before the tag rename — see "Known gaps" for what's unverified post-rename), correctly found nothing to triage, wrote a proper signed report, and updated `state.json` correctly. A second immediate `diwan tick` correctly no-op'd. **Not yet re-verified** against the post-split layout (tool repo + separate `diwan-dev-notes` content repo, `TOOL_ROOT`/`CONTENT_ROOT` as two different directories) — see "Known gaps". + +## What `diwan do` actually does + +Same split: `cmd_do` (CLI) decides identity/dispatch mechanics; `DO.md` governs what the agent does once running. + +**CLI half** (`cmd_do`): +1. Parse `target`: if it matches `:` (resolved against the content repo), it's heading mode; anything else is treated as an ad-hoc free-text prompt with no org heading involved (useful for manual testing, or tasks that genuinely don't belong in the notes). +2. **Heading mode:** read the headline at that line, check its `DIWAN_ID` property. + - Already has one → **continuation**. (If `--diwan-id` was also passed and disagrees with the stored one, that's an error, not a silent overwrite — a mismatch means something's confused and a human should look.) The existing ID is validated against herdr's naming rule before anything else happens, so a corrupted/hand-edited property fails with a clear message instead of a confusing herdr-side error later. + - No `DIWAN_ID` yet → **fresh**. Compute one: `--diwan-id` if given, else a slug of the heading text (`slugify`, capped and shaped to satisfy herdr's agent-name rule — see below), de-duplicated against every other `DIWAN_ID` already in the content repo's tree (`unique_id`). Write the property and insert `** [ ] Close diwan session` as the first child (`assign_diwan_id`), in one commit under the diwan identity. This is the *only* place a `DIWAN_ID` is minted or a Close-session item is inserted — mechanical, not left to agent judgment, so it's always consistent. +3. **Ad-hoc mode:** `diwan_id` is `--diwan-id` or a generated `adhoc-<8 hex chars>`; always goes through the continuation dispatch path (see below) since there's no heading state to distinguish fresh from resumed. +4. Dispatch: fresh headings use `herdr_dispatch_fresh`; everything else (continuation headings, ad-hoc) uses `herdr_dispatch_continuation`. Prints the `diwan_id` either way — this is the command's only stdout contract, which is what lets `TICK.md` step 3 just run `diwan do ...` as an ordinary shell command and know what session it created. + +**Continuation dispatch, concretely** (`herdr_dispatch_continuation`): check `herdr agent get ` — alive → just `agent prompt ` (fire-and-forget, no `--wait`). Not alive (the normal case between ticks; CLI agent processes exit, they don't idle forever) → look up the tab by label in the `diwan` workspace, start a fresh agent in its existing root pane, prompt it. Tab itself gone too (e.g. closed by hand) → falls through to the same fresh-tab path `herdr_dispatch_fresh` uses. **Continuity across resumptions is carried by the heading's own content, not by agent conversation memory** — a resumed `do` agent has no idea what it said last time except what's written in the org file. This is a real, current limitation, not an oversight to "fix" without thinking about it — it's a direct consequence of principle #1 (org is ground truth) and keeps the system's state in exactly one place. + +**Agent half** (`DO.md`): stay inside your own heading's subtree; if something needs doing elsewhere, ask in `:chat:` rather than reaching out of scope (cross-cutting work like this is now `tick`'s job to do directly, per the `:diwan:` tag section — don't try to talk another `do` session into doing it either); use the chat protocol for questions/FYIs; commit progress in small units under the diwan identity through the flock-wrapped recipe; never touch your own `Close diwan session` item — that's Marc's call, acted on by the next tick. One explicit exception to subtree-only: a `do` agent may append its own entry to `diwan.org`'s Reports log once it finishes something worth telling Marc about, since that's its only channel to him outside of a tick run — it still never edits or deletes an existing entry there, only `tick` does that. A tick that merely nudges an idle session (see `cmd_tick` step 3 above) does not report the nudge itself — it has no idea yet whether anything came of it. + +**Verified live** (against the old mixed repo, pre-split): dispatched for real against the "gather info about Greece trip" heading. Did real web research, wrote a structured result into the heading, correctly asked clarifying questions via a signed `:chat:` entry instead of guessing, left `Close diwan session` untouched, committed as `diwan ` with a descriptive message. + +**`pi` kind, verified live** (2026-11-08, herdr 0.9.5 + pi 1.5.0, pre-split): ad-hoc `diwan do "" --diwan-id pi-smoketest` with `DIWAN_HARNESS=pi` correctly started a herdr `pi` agent (detected as `agent: "pi"`, no trust/approval prompt blocked it, confirming `--approve` is doing its job with no protected `.pi/*` resources in that repo to actually gate), loaded `AGENTS.md` as context the way `claude` does, and replied exactly as instructed. A second `diwan do` call against the same `--diwan-id` reused the same pane/tab and agent (continuation dispatch, not a fresh one) and replied correctly again. Not yet exercised: a real `tick` or `do` run against actual heading content with `DIWAN_HARNESS=pi` (only ad-hoc free-text prompts so far) — same caveat as any first integration, worth a full pass before relying on it for real delegated work. + +## `DIWAN_ID` ⇄ herdr, precisely + +A `DIWAN_ID` *is* a herdr tab label in the `diwan` workspace, and (while its agent process is alive) also that agent's name. This is why it must satisfy herdr's agent-name rule: `^[a-z][a-z0-9_-]{0,31}$`. `slugify()` enforces this at mint time (lowercase, non-alnum → hyphen, truncated to 24 chars, forced to start with a letter) and `cmd_do` additionally validates *every* `DIWAN_ID` it's about to dispatch — including ones already stored in a heading — against that same regex before touching herdr, erroring out with a clear message rather than letting herdr reject it after the CLI has already created a tab (which used to leave an orphan tab behind; fixed by closing the just-created tab on any `agent start` failure). + +Lifecycle: **fresh** (no property) → `diwan do` mints an ID, creates the tab+agent, heading now has `DIWAN_ID` + open (`[ ]`) `Close diwan session` → **active** (any further `tick`/`do` on this heading is a continuation; `tick` explicitly leaves active sessions alone) → Marc checks off `Close diwan session` (`[X]`) → next `tick` notices and retires it (per `TICK.md` step 3 — note: as of this writing the tick procedure only *reports* that a session closed; it does not yet actually call `herdr tab close` on it — see "Known gaps"). + +## Known gaps / explicit non-goals (don't "fix" these without discussing) + +- **Tool/content repo split just happened (2026-11-09), not yet re-verified live.** This repo used to be both diwan's own dev copy *and* a throwaway test-content repo at once; it's now tool-only, with content split out to a sibling `diwan-dev-notes` repo. The CLI was updated to resolve `TOOL_ROOT` (its own install dir, `Path(__file__).resolve().parent`, overridable via `DIWAN_TOOL_ROOT`) separately from `CONTENT_ROOT` (`git rev-parse --show-toplevel` from the invoking directory), and the dispatch prompts now pass `TOOL_ROOT`-absolute paths for `PROTOCOL.md`/`TICK.md`/`DO.md`. This has been syntax-checked and smoke-tested for path resolution, but **not yet run through a full live tick/do pass** against the new two-repo layout — do that before trusting it for real. +- **No flake yet.** The split prepared the ground (tool repo is self-contained, no content-repo assumptions baked into its own files) but there's no `flake.nix` in this repo yet, and the shebang still runs via a bare `nix shell nixpkgs#python3` rather than a proper package. Until that lands, "installed on `PATH`" means manually symlinking/aliasing `diwan` somewhere on `PATH` yourself. `DIWAN_TOOL_ROOT` exists specifically so a future flake's install layout (e.g. docs copied into `$out/share/diwan/` alongside a wrapped binary) can override the default `Path(__file__).resolve().parent` guess if it doesn't hold. +- **Retired-session teardown.** `TICK.md` step 3 says a closed session has "nothing for you to do to the heading's content" and that herdr-side teardown "is not your job" — but nothing else currently closes that tab either. Right now a `Close diwan session` checked off (`[X]`) just... sits there, tab still open. This is a real gap, not yet decided whether the CLI should gain a path for it or whether `TICK.md` should just say to run `herdr tab close` directly. +- **No real conversation continuity across resumption.** Covered above — intentional for now, but worth revisiting if resumed `do` sessions turn out to need more context than the heading text carries. +- **`pi` kind is wired up and smoke-tested, but not yet exercised against real heading content.** `DIWAN_HARNESS=pi` gets its own equivalent of the Claude autonomy flag (`--approve`, see "`pi` kind, verified live" above) and has been confirmed live for ad-hoc `diwan do` dispatch and continuation. Still untested: `diwan tick` with `DIWAN_HARNESS=pi`, and `diwan do` against a real `:` heading (DIWAN_ID minting, `:chat:` protocol, scope discipline) rather than a free-text ad-hoc prompt. +- **`diwan.org`'s `Config` heading is a placeholder, not wired up.** PROTOCOL.md documents an intended home for configuration values governing diwan's own behavior (tick cadence, tab retention, harness choice), but the CLI reads none of it — the env vars (`DIWAN_HARNESS`, `DIWAN_PERMISSION_MODE`, `DIWAN_PI_APPROVE`, `DIWAN_MAX_TICK_TABS`) remain the only real knobs. Don't assume anything in that heading does something until the CLI actually reads it. +- **The `:diwan:`/`:session:`/`:chat:` tag split, and tick doing cross-cutting maintenance directly, is new and unverified live.** Before this split, `:diwan:` alone meant what `:session:` means now, and anything non-trivial got a dedicated `do` session regardless of whether the work was subtree-shaped or genuinely repo-wide — which is exactly what motivated the split (a real `do` session hit the subtree wall trying to do an archive/cleanup/folder-move task). No tick run has yet exercised the new three-way decision in `TICK.md` step 3 (inline / `:session:` dispatch / tick-direct), the tag-consistency backfill in step 2, or tick actually doing a multi-file maintenance job itself across one or more runs — worth a close look at the first few real runs that hit this path. +- **`--permission-mode auto` is a real autonomy decision, not a default to take for granted.** Dispatched agents run with Claude Code's own judgment-based auto mode (chosen over a full `--dangerously-skip-permissions` bypass, which Claude's docs scope to no-internet sandboxes — diwan agents do real web research). Override via `DIWAN_PERMISSION_MODE` env var. If this is ever revisited, it should be a deliberate conversation, not a silent change. +- **Concurrency is scope discipline, not real locking**, beyond the git-commit flock. Two `do` agents editing truly disjoint subtrees should be fine; nothing stops two agents editing the *same* subtree if the scope rule is violated. There's no enforcement, only convention (PROTOCOL.md). +- **No automated tests.** Everything described as "verified live" above was exercised by hand against a real herdr+claude install. There is no regression suite — a future session changing `diwan` should re-run at least the manual smoke tests (tick no-op gate, tick dispatch, do fresh dispatch, do continuation, a deliberately-invalid `DIWAN_ID`) before trusting a change. diff --git a/DO.md b/DO.md new file mode 100644 index 0000000..e0d3d4f --- /dev/null +++ b/DO.md @@ -0,0 +1,31 @@ +# diwan do procedure + +You are a `diwan do` agent. Read `PROTOCOL.md` and the content repo's `AGENTS.md` first if you haven't — this doc assumes both (you were given `PROTOCOL.md`'s path directly in your dispatch prompt). You were dispatched (or resumed) against exactly one heading, identified by a `DIWAN_ID` that is also your herdr terminal name. You may run across many ticks; the same `DIWAN_ID`/terminal persists until Marc closes you. + +## Scope + +You only edit inside the subtree of your heading (from your heading line down to, but not past, the next heading at the same or shallower level). Everything else in the repo is read-only to you. This is what lets you run concurrently with `diwan tick` and other `diwan do` agents without stepping on them — don't break the contract even if it'd be convenient to. + +If doing your task well genuinely requires writing somewhere else (e.g. "add a calendar entry for the flight you just found"), don't do it yourself — leave a clear note in your chat subheading asking for it (or, if it's truly unambiguous busywork, you may say so explicitly and still just ask — the next `diwan tick` will pick it up). When in doubt, ask rather than reach outside your subtree. + +## Communicating with Marc + +Use (or create, if missing) a `:chat:` child heading under your heading, same format as any other diwan chat (see PROTOCOL.md): `[ ]` unread, `[X]` read/acted, your entries signed `[diwan YYYY-MM-DD]`. Use it for: + +- Questions you need answered before you can make progress. +- FYIs about something you decided on Marc's behalf (so he can override it). +- Your actual findings/results, if they don't belong directly in the heading's body — otherwise just write results into the heading itself and use chat only for back-and-forth. + +Don't mark your own entries `[X]` — that's Marc acknowledging them. You may mark *his* `[ ]` entries `[X]` once you've read and acted on them. + +## Reporting + +When you finish a notable piece of work — answered a chat question with real content, completed a chunk of delegated research, did something worth telling Marc about after being resumed/nudged — append one short entry of your own to `diwan.org`'s Reports log, in the format PROTOCOL.md's Reports section describes (a `[ ]` subheading, a short title, an optional `:LINK:` back to your own heading, a signed one-or-two-line body). This is the one explicit exception to staying inside your own subtree: you may append a new entry there, but never edit or delete an existing one — that's `tick`'s job only. Don't report every small step, just the things Marc would actually want to know happened without reading your heading in full. Nothing about resuming or being nudged is itself reportable on its own — only report once you've actually done something as a result. + +## Committing + +Commit your own progress as soon as it's a coherent unit, through the locked helper from PROTOCOL.md's concurrency contract, under the diwan git identity. Small, frequent commits — don't sit on a large uncommitted diff. + +## Finishing a pass + +You don't close your own session. When you've made the progress you can for now (done, blocked on a question, or waiting on external info), just leave the heading and chat in a clear state and stop — you may be resumed later with new instructions when something changes. Never add, remove, or check off the `Close diwan session` item yourself; that's Marc's call, acted on by the next `diwan tick`, not yours. diff --git a/PROTOCOL.md b/PROTOCOL.md new file mode 100644 index 0000000..f989193 --- /dev/null +++ b/PROTOCOL.md @@ -0,0 +1,130 @@ +# PROTOCOL.md — the diwan protocol + +This is diwan's generic contract: the tag system, the `diwan.org` layout, the concurrency/revert model, and the herdr topology. It describes how diwan works, independent of which notes repo it's pointed at — it ships with diwan itself (this repo), not with any one person's content. If you are a dispatched diwan agent (`tick` or `do`), read this file first (you were given its path directly in your dispatch prompt), then the *content* repo's own `AGENTS.md` — a different repo from this one, containing that deployment's specific notes schema and conventions. The two together are the complete contract; this doc never restates anything from a content repo's `AGENTS.md` (Layout, org conventions already in use) and that file never restates anything from here — if you find yourself unsure which owns something, it's a doc bug worth flagging, not something to resolve by guessing. + +## `:diwan:` tag + +A generic way to call diwan's attention to a heading: tagging something `:diwan:` means "I want diwan to take care of this" (or, when diwan adds it itself, "this needs diwan's attention"), without saying anything about *how*. An ordinary task that's normally just for the user becomes a delegated request the moment the tag is added: + +```org +** TODO Archive all DONE entries in the inbox :diwan: +``` + +Can be added directly by the user, or by diwan itself when it decides some other heading (a chat reply, a plain task found during triage) needs this kind of attention — see the tag-consistency rule below. What actually happens to a `:diwan:`-tagged heading depends on what the request needs, decided by whichever diwan agent processes it (ordinarily `tick`, per `TICK.md`): + +- Trivial (a quick lookup, a one-line answer): handled inline immediately, no further tag, no session. +- Non-trivial but self-contained in this heading's own subtree (independent delegated work, nothing that needs writes elsewhere): escalates to a `:session:` tag and a dedicated `diwan do` session. +- Non-trivial *and* inherently cross-cutting — spans multiple files or headings, can't be parallelized, e.g. archiving, bulk cleanup, moving things between files or folders: handled by `diwan tick` itself, directly, possibly across several tick runs — not escalated to a session, since a subtree-scoped `do` agent would just hit the same wall a human doing it alone would (see the Concurrency contract's scope rule). No `:session:` tag for this case. + +## `:session:` tag + +A heading tagged `:session:` has (or is about to get) a dedicated, persistent agent session handling it — a `DIWAN_ID` property and a `Close diwan session` checkbox item. Note the org tag syntax when it's stacked alongside `:diwan:`: one colon-delimited run, no space (`:diwan:session:`, not `:diwan: :session:`) — the latter isn't valid org tag syntax and won't parse as two tags. + +```org +* Gather information about what I could do on my next trip to greece :diwan:session: + :PROPERTIES: + :DIWAN_ID: do-greece-trip + :END: +** [ ] Close diwan session +``` + +Can be added by the user directly, when they already know a heading needs its own ongoing session, or by diwan (ordinarily `tick`) when it decides a `:diwan:`-tagged request is non-trivial and self-contained enough to delegate (see above). `DIWAN_ID` is a herdr terminal name (see "herdr", below) — not an arbitrary id. It points directly at the live/persisted agent session handling this heading, so "jump in personally" is just attaching to that terminal, no separate lookup needed. A heading keeps the same `DIWAN_ID`/`:session:` tag across multiple tick passes until its `Close diwan session` item is checked off (`[X]`) by the user, at which point the *next* tick retires the session and leaves the property and tag as a historical record (does not delete them). + +## `:chat:` tag + +A `:chat:` heading is a conversation thread between the user and diwan, always as a child of the heading it's about. Entries are checklist-style, unread/unacted = `[ ]`, read/acted = `[X]`. diwan signs and timestamps its own entries; the user is not required to: + +```org +** Chat :chat: +- [ ] Please investigate some ways I could decorate my kitchen +- [X] [diwan 2026-10-13] I have added various proposals in the Jams header +- [ ] [diwan 2026-10-21] I found the "Invite cellist" item in the inbox and think it belongs here. Please confirm. +``` + +**Ownership rule:** a `:chat:` heading that lives under a `:session:`-tagged ancestor is handled by that ancestor's `DIWAN_ID` session — don't spin up a separate session just for the chat. + +**Promotion rule:** a `:chat:` heading with *no* `:session:`-tagged ancestor is answered directly by whichever diwan agent is processing it (ordinarily `tick`). For each unread (`[ ]`) entry: trivial — answer/act inline, mark `[X]`. Non-trivial — the same escalation decision as any other `:diwan:`-tagged heading above: a `:session:` tag if the work is self-contained, or handled by `tick` directly if it's cross-cutting. + +**Tag-consistency rule:** `:session:` and `:chat:` both imply diwan's attention, so each should normally appear together with (or have an ancestor carrying) a `:diwan:` tag. This is fine: + +```org +* Heading A :diwan: +** Heading B :session: +``` + +This is not explicit enough — the next `diwan tick` adds `:diwan:` directly to Heading B (not to Heading A) to fix it: + +```org +* Heading A +** Heading B :session: +``` + +Same rule for `:chat:`. + +## Signing format + +`[diwan YYYY-MM-DD]` prefix on a chat line, as shown above. Optional on headings/body text diwan writes elsewhere; required on chat entries so the user can tell diwan's messages from their own at a glance. + +## `diwan.org` + +diwan's own operational file, at the root of the content repo — not a place for the user's personal notes, those belong wherever this deployment's own `AGENTS.md` says. Four top-level headings, each with a distinct job: + +### Reports + +Every notable thing a diwan agent does — a tick dispatching something new, a tick triaging a batch of inbox items, a `do` session finishing a real piece of work — gets its own short subheading under the top-level "Reports" heading in `diwan.org`. One entry per notable thing, not one giant paragraph per run: this is what keeps each entry short and the whole log skimmable with everything collapsed. **Whoever actually did the thing writes the entry** — `diwan tick` reports its own actions (triage moves, inline answers, promotions, new dispatches), and a `diwan do` session reports its *own* progress once it has something worth telling the user, not before. In particular: a tick merely nudging an idle active session (see the `:session:` section above) is not itself reportable — tick has no idea yet whether anything came of it, so it stays silent; if the nudge leads the session to actually do something, that session writes the entry once it has. + +Format: a `[ ]`/`[X]` checkbox right in the subheading title (same pattern as `Close diwan session`), a short title, an optional `:LINK:` property pointing at the one heading the entry is most about (fuzzy heading link, or `id:` link for a note file with an `:ID:` drawer — skip it if there's no single heading to point at), and a signed one-or-two-line body, no more: + +```org +* Reports +** [ ] Dispatched kitchen redecoration + :PROPERTIES: + :LINK: [[file:personal.org::*Plan the kitchen redecoration][Plan the kitchen redecoration]] + :END: + [diwan 2026-11-08] Non-trivial, dispatched as do-kitchen. +** [X] Triaged 2 inbox items + [diwan 2026-11-07] Moved into esdeveniments.org and personal.org. +``` + +Unread = `[ ]`, read = `[X]`, same semantics as `:chat:` — the user checks an entry off once they've seen it. Once an entry is `[X]`, the *next* `diwan tick` deletes the whole subheading, not just the checkbox — this is a disposable log, not a record, and only `tick` ever deletes from it (a `do` session only ever appends its own new entries, never touches an existing one). Questions for the user still go in the relevant heading's own `:chat:`, not here — a report entry may link there, but the report itself is one-way, no back-and-forth expected. + +### Tasks + +Plain `TODO`/`DONE` items about maintaining diwan's own domain — the notes themselves, not any one topic in them: archiving old entries, bulk cleanup, moving files or folders around, repo-wide renames. Anyone (the user or diwan) can add one directly as a plain heading here; living under `diwan.org`'s Tasks heading is itself the signal, no `:diwan:` tag needed on the item (the heading itself may carry one, that's fine, just not required). `diwan tick` treats every open item here as inherently cross-cutting work — see the `:diwan:` tag section above — and handles it directly in its own small commits, across however many tick runs it takes, marking the item `DONE` once actually finished, or asking via a `:chat:` child first if a real judgment call needs the user's input. + +### Config + +Reserved for configuration values governing diwan's own behavior — tick cadence, tab retention, which harness to dispatch, that kind of thing — as one legible place instead of scattered environment variables. **Not implemented yet.** The `diwan` CLI doesn't read this heading for configuration as of this writing; it's a placeholder for a future change, not a convention in force today. Env vars (`DIWAN_HARNESS`, `DIWAN_PERMISSION_MODE`, `DIWAN_PI_APPROVE`, `DIWAN_MAX_TICK_TABS`) remain the only actual knobs until that lands. + +### Meta + +Free-form notes about diwan itself — ideas, friction points, things worth revisiting as it evolves — written by the user or by diwan as they come up, independent of any one tick run. This feeds into developing diwan itself (see `DESIGN.md`); nothing currently reads or acts on this heading automatically, it's input for whoever picks up that work next, not a `tick`/`do` responsibility. + +## Concurrency contract + +Multiple agents (one `tick`, several `do`) may be running against this working tree at once. To keep that manageable without a real merge system: + +1. **Scope discipline.** A `diwan do` agent only ever edits inside the subtree of the heading it owns (its `DIWAN_ID` heading), plus it may read anything. The one exception is `diwan.org`'s Reports log: a `do` agent may *append* a new entry of its own there (never edit or delete an existing one — that's `tick`'s job only), since that's the only channel it has for telling the user about finished work without a tick run in between. A `diwan tick` agent may move/create headings anywhere (that's its job), and is the one that handles non-trivial `:diwan:` requests that are inherently cross-cutting rather than dispatching them to a `do` session (see the `:diwan:` tag section) — but should keep each such move or piece of cross-cutting work a small, separate commit. This makes line-level conflicts rare by construction — don't go outside your lane. +2. **No manual commit requirement.** The user is never expected to commit. `diwan tick` commits any dirty state it finds at the *start* of a run, under the user's own git identity, before computing anything — so no tick ever operates on an uncommitted/mixed workspace. Any agent (tick or do) commits its *own* changes as soon as they're a coherent unit, using the diwan identity: + ```sh + flock .diwan/.lock git -c user.name=diwan -c user.email=diwan@local commit -am "" + ``` + The `flock` is required — it's how concurrent agents avoid racing each other's `git add -A && commit`. Always go through it, never a bare `git commit`. +3. **Small commits over big ones.** Prefer several small, clearly-labeled commits to one large one — this is what makes `git log` a usable report of "what did diwan actually do" and what makes `git revert` safe to use on a single change without undoing unrelated work. + +## Revert model + +Every automated change is a git commit, full stop. There is no separate undo mechanism to learn: `git log`, `git diff`, `git revert ` are *the* way the user inspects or undoes anything diwan did. `.diwan/state.json` is the only piece of diwan state that isn't a note — it just remembers the last commit a completed `tick` left behind, so the next tick knows what's new. + +## herdr + +Sessions are herdr (https://herdr.dev) terminals, not a diwan-invented session store. All diwan activity lives in one herdr workspace labeled `diwan` (created on first use if missing). Within it: + +- a `diwan tick` run is a tab labeled `tick-`, with an agent of kind `$DIWAN_HARNESS` (default `claude`) started in its root pane and prompted with the tick instructions; +- a `diwan do` session for a given `DIWAN_ID` is a tab labeled exactly that `DIWAN_ID`, same pattern — this is *why* `DIWAN_ID` must be a valid herdr agent name (`^[a-z][a-z0-9_-]{0,31}$`), not an arbitrary string. + +To resume a `DIWAN_ID`, `diwan` first checks whether its agent is still alive (`herdr agent get `) and just prompts it if so; if the process has exited (normal between ticks), it finds the tab by label and starts a fresh agent in the same pane, in the same working directory — the heading's own content is what carries continuity, not agent conversation history. `diwan` shells out to the real `herdr` CLI (the `diwan` CLI has the exact calls, verified live against herdr 0.9.5; run `herdr --skill` to check this hasn't drifted on a newer version). When `herdr` isn't on PATH, every call prints what it would have run instead of failing, so the pipeline stays testable without herdr installed. + +## Harness-agnosticism + +Nothing here should assume a specific LLM product. `TICK.md` and `DO.md` are plain-language procedures; the `diwan` CLI is the only "hands" they need (plus ordinary file edits and git). Pointing a different harness at the same procedures should just work. diff --git a/TICK.md b/TICK.md new file mode 100644 index 0000000..bc405a6 --- /dev/null +++ b/TICK.md @@ -0,0 +1,61 @@ +# diwan tick procedure + +You are a `diwan tick` agent. Read `PROTOCOL.md` and the content repo's `AGENTS.md` first if you haven't — this doc assumes both (you were given `PROTOCOL.md`'s path directly in your dispatch prompt). You were dispatched by `diwan tick` with a commit range (`..`) representing what changed since the last completed tick. Your job is one pass of housekeeping over that change, not an open-ended audit of the whole repo. + +Work through these steps in order. Make a separate, small, lock-protected commit (see PROTOCOL.md's concurrency contract) after each coherent chunk of work rather than one giant commit at the end — that's what makes your run legible in `git log` afterwards. + +## 1. Triage the inbox + +Look at what changed in `inbox.org` (and only there — other files changing is someone/something else's edit, not raw capture). The goal is to leave `inbox.org` empty, or as close to it as possible, once this step is done — it's a capture buffer, not a holding pen; don't let items linger there just because placement is a little unclear. For each new item: + +- An event (something with a date/time, or clearly calendar-shaped) → move it into `esdeveniments.org` under "Propers", in roughly chronological position. If it's missing information you'd need to do this well (e.g. a flight with no time yet), move it anyway with what you have, and flag the gap in the report (step 4) rather than leaving it in the inbox. +- A plain task → move it to wherever it actually belongs: an existing project file/folder if it's clearly about one (e.g. `folkugat/`), the relevant domain file (`personal.org`, etc.) if general, or a sensible top-level spot in `personal.org` if nothing fits. Don't create a new file for something that fits an existing one. If you're genuinely unsure which existing spot is right, still move it to your best guess rather than leaving it in the inbox — but add a `:chat:` child heading under it there with a question asking Marc where it should actually live, so he can redirect it next pass instead of it silently sitting in a wrong-but-plausible place. +- A `:diwan:`-tagged item (see PROTOCOL.md) → leave it where it landed for now; it gets handled in step 3, not here. (You may still move it out of the inbox into a sensible home first, same as any other item — the tag travels with it.) +- Something that's clearly the seed of a new project/topic with no existing home → create a new file for it, but only when it genuinely doesn't fit anywhere existing. Default to not creating new files. + +Commit this as its own step: `tick: triage inbox`. + +## 2. Ensure tag consistency, then resolve `:chat:` ownership + +First, tag consistency (see PROTOCOL.md's tag-consistency rule): for every heading tagged `:session:` or `:chat:` anywhere in the tree, check whether it or an ancestor already carries `:diwan:`. If not, add `:diwan:` directly to that heading itself (not to the ancestor) so the implied attention is made explicit. + +Then, chat resolution: find every `:chat:` heading anywhere in the tree with at least one unread (`[ ]`) entry. + +- If it has a `:session:`-tagged ancestor: skip it. That heading's `DIWAN_ID` session owns it. +- If it does not: for each unread entry, decide trivial vs non-trivial. + - Trivial (you can just answer or act right now, no multi-step delegated work): do it, write your reply as a new signed entry (`[diwan YYYY-MM-DD] ...`), mark the original `[X]`. + - Non-trivial: this is the same three-way escalation decision step 3 makes for any `:diwan:`-tagged heading — self-contained work gets a `:session:` tag and a dispatch, cross-cutting work gets handled by you directly, both as part of step 3. Leave the chat entry unread until the work is actually done and its own agent (or you, on a later tick) answers it. + +Commit: `tick: resolve chat ownership and tag consistency`. + +## 3. Handle `:diwan:` headings + +Find every `:diwan:`-tagged heading in the tree (including ones just tagged in step 2). + +- No `:session:` tag yet, and the request is trivial (quick lookup, one-line answer you can produce yourself right now): do it inline, write the result into the heading, and leave it — no session, nothing further to track. This is the common case for small asks. +- No `:session:` tag yet, non-trivial, and self-contained within this heading's own subtree (independent delegated work, nothing that needs writes to other files or headings): add the `:session:` tag and run `diwan do : --diwan-id ` yourself (pick a short, readable slug from the heading text — `diwan do` will write the `DIWAN_ID` property and the `Close diwan session` checkbox item for you; don't hand-edit those). Note the dispatch in your report. +- No `:session:` tag yet, non-trivial, and inherently cross-cutting — touches multiple files or headings at once, or can't be meaningfully parallelized (archiving, bulk cleanup, moving files/folders between locations, repo-wide renames): do this yourself, directly, right now, in small separate commits per coherent chunk — the same way you'd do any other tick step. Don't add a `:session:` tag for this; a subtree-scoped `do` agent would just hit the same wall a human doing it alone would. If there's a real judgment call to make first (not just mechanical execution), ask in the heading's own `:chat:` and leave the rest for a future tick once Marc answers, rather than guessing or acting unsupervised at scale. It's fine for this kind of task to span several tick runs — pick up wherever the heading's own content (and its sub-TODOs, if any) says you left off. +- Already has a `:session:` tag and its `Close diwan session` item is still `[ ]`: it's an active session, already being worked by its own agent — you still have no business editing inside its subtree, but check whether it needs waking up. Look at this tick's change range for any commit *not* authored by the diwan git identity (`diwan `) that touched a line inside that heading's subtree — a direct edit by Marc to the heading body, a new unread `:chat:` entry, anything hand-authored. The tick's own start-of-run manual-edit snapshot commit is the usual source of these. If you find one, nudge the session: run `diwan do : --diwan-id ` again (same existing `DIWAN_ID` — this reuses the session's own tab/agent, it does not create a new one) so it re-reads the heading and notices on its own. If nothing hand-authored touched the subtree, leave it alone entirely, no dispatch at all. A nudge is not itself something to report in step 4 — you have no idea yet whether anything comes of it; if it leads the session to do real work, that session reports it, not you. +- Already has a `:session:` tag and its `Close diwan session` item is `[X]`: this session is being retired. There's nothing for you to do to the heading's content; the CLI/herdr-side teardown of the actual terminal is not your job — just note in your report that it closed. + +Also check `diwan.org`'s `Tasks` heading for any open (not-yet-`DONE`) item — these are the same kind of cross-cutting work as above, already homed in the right spot with no tag needed (see PROTOCOL.md's `diwan.org` section). Pick one up and make progress on it (or finish it) directly, in its own commit(s), the same way as any other cross-cutting `:diwan:` request; mark it `DONE` once actually finished, or ask via a `:chat:` child first if it needs a real judgment call from Marc. + +Commit after each dispatch, cross-cutting chunk of work, or batch of inline answers: `tick: dispatch ` / `tick: ` / `tick: answer `. A nudge doesn't change any file content on its own, so there's nothing to commit for it either. + +## 4. Write the report + +In `diwan.org` at the repo root, under a top-level "Reports" heading (create the file and heading if this is the very first tick), first delete any existing report subheadings already marked `[X]` in their title — Marc has read those, they're disposable, don't let them pile up (this applies to every entry regardless of who wrote it, including ones a `do` session appended). Then add one new subheading per notable thing *this tick itself* did — a new dispatch, an inline answer, a batch of triage moves, not one giant entry covering the whole run, and not anything that happened inside an active `do` session's own subtree (that session reports its own work, see PROTOCOL.md's Reports section — a bare nudge you sent this run is explicitly not reportable by you). Keep titles short enough that Marc can skim the whole log collapsed and know what happened without opening anything. Each entry: + + ** [ ] + :PROPERTIES: + :LINK: + :END: + [diwan YYYY-MM-DD] + +Omit the `:LINK:` property when there's no single heading the entry is about (e.g. "nothing to triage this run"). Anything needing Marc's actual input or a reply (as opposed to a passive FYI) belongs as a normal unread entry in the relevant heading's own `:chat:`, not in the report — the report entry may link there, but don't try to hold a conversation in it. + +Commit: `tick: report`. + +## 5. Record completion + +Last step, only if everything above succeeded: update `.diwan/state.json`'s `last_tick_commit` to the current `HEAD` sha, and commit that under the diwan identity: `tick: mark complete`. If anything above failed partway, leave `state.json` alone — the next tick (or Marc, by hand) should see the same change range again rather than silently skipping it. diff --git a/diwan b/diwan new file mode 100755 index 0000000..6633758 --- /dev/null +++ b/diwan @@ -0,0 +1,552 @@ +#!/usr/bin/env -S nix shell nixpkgs#python3 --command python3 +"""diwan CLI — see PROTOCOL.md, TICK.md, DO.md (installed alongside this +script) for what this orchestrates, and the content repo's own AGENTS.md +for what it's orchestrating against. + +This script only does the mechanical, deterministic parts (git bookkeeping, +DIWAN_ID property + "Close diwan session" checkbox insertion, herdr +dispatch). +All actual triage/research reasoning happens in the dispatched agent, per the +procedure docs. + +herdr integration is against the real `herdr` CLI (workspace/tab/pane/agent +over a JSON socket API — see `herdr --skill`), verified live against herdr +0.9.5. When `herdr` isn't on PATH, every herdr_* call prints what it would +have run instead of failing, so the rest of the pipeline stays testable +without herdr installed. + +Two separate roots: TOOL_ROOT is where diwan itself (this script, plus +PROTOCOL.md/TICK.md/DO.md) is installed — `diwan` is meant to live on PATH +via a nix flake, independent of any one person's notes. CONTENT_ROOT is +whatever notes repo it's being invoked against, discovered from the current +directory via `git rev-parse --show-toplevel` — diwan must be run from +inside (or below) that repo. +""" +import argparse +import contextlib +import fcntl +import json +import os +import re +import shutil +import subprocess +import sys +import uuid +from datetime import datetime, timezone +from pathlib import Path + +TOOL_ROOT = Path(os.environ.get("DIWAN_TOOL_ROOT", str(Path(__file__).resolve().parent))) + + +def find_content_root(): + res = subprocess.run( + ["git", "rev-parse", "--show-toplevel"], capture_output=True, text=True + ) + if res.returncode != 0: + print( + "error: not inside a git repo — run diwan from inside the notes repo " + "you want it to act on.", + file=sys.stderr, + ) + sys.exit(1) + return Path(res.stdout.strip()) + + +CONTENT_ROOT = find_content_root() +DIWAN_DIR = CONTENT_ROOT / ".diwan" +STATE_FILE = DIWAN_DIR / "state.json" +LOCK_FILE = DIWAN_DIR / ".lock" +MAX_TICK_TABS = int(os.environ.get("DIWAN_MAX_TICK_TABS", "3")) +AGENT_KIND = os.environ.get("DIWAN_HARNESS", "claude") +DIWAN_WORKSPACE_LABEL = "diwan" +# claude defaults to prompting for approval on every edit/bash call, which +# defeats principle #2 (no per-edit approval, just reported+revertable). +# --permission-mode auto is Claude Code's own judgment-based autonomous mode +# (acts freely, still pauses on genuinely risky things) — not a blanket +# bypass like --dangerously-skip-permissions, which Claude's own docs say is +# only meant for sandboxes with no internet access (diwan agents may do real +# web research). Override with DIWAN_PERMISSION_MODE if this isn't right. +CLAUDE_PERMISSION_MODE = os.environ.get("DIWAN_PERMISSION_MODE", "auto") +# pi has no per-tool-call approval gate at all (see its own security docs); +# the only thing that can still block a non-interactive dispatch is the +# one-time project-trust prompt for protected .pi/* resources, if any ever +# show up in this repo. --approve makes that a one-shot yes instead of a +# hang, same spirit as CLAUDE_PERMISSION_MODE above. +PI_APPROVE_FLAG = os.environ.get("DIWAN_PI_APPROVE", "--approve") + +HEADLINE_RE = re.compile(r"^(\*+)\s+(.*)$") +HERDR_NAME_RE = re.compile(r"^[a-z][a-z0-9_-]{0,31}$") +TAGS_RE = re.compile(r"\s+:[\w:]+:\s*$") +TODO_KEYWORDS = {"TODO", "IN-PROGRESS", "WAITING", "DONE", "CANCELLED"} + + +# --- git ----------------------------------------------------------------- + +def run_git(args, check=True, capture=False): + return subprocess.run( + ["git", *args], cwd=CONTENT_ROOT, check=check, capture_output=capture, text=True + ) + + +def git_is_dirty(): + out = run_git(["status", "--porcelain"], capture=True).stdout.strip() + return bool(out) + + +def git_head(): + return run_git(["rev-parse", "HEAD"], capture=True).stdout.strip() + + +def git_diff_has_changes(since, until, excludes=()): + out = run_git(["diff", "--name-only", since, until], capture=True).stdout + files = [f for f in out.splitlines() if f] + files = [ + f + for f in files + if not any(f == ex or f.startswith(ex.rstrip("/") + "/") for ex in excludes) + ] + return bool(files) + + +@contextlib.contextmanager +def locked(): + DIWAN_DIR.mkdir(exist_ok=True) + fd = open(LOCK_FILE, "w") + try: + fcntl.flock(fd, fcntl.LOCK_EX) + yield + finally: + fcntl.flock(fd, fcntl.LOCK_UN) + fd.close() + + +def git_commit_all(message, diwan_identity=False): + """Stage and commit everything currently dirty, under a lock. Returns + False if there was nothing to commit.""" + with locked(): + run_git(["add", "-A"]) + if not run_git(["status", "--porcelain"], capture=True).stdout.strip(): + return False + args = [] + if diwan_identity: + args += ["-c", "user.name=diwan", "-c", "user.email=diwan@local"] + args += ["commit", "-m", message] + run_git(args) + return True + + +# --- state ----------------------------------------------------------------- + +def load_state(): + if STATE_FILE.exists(): + return json.loads(STATE_FILE.read_text()) + return {} + + +def save_state(state): + DIWAN_DIR.mkdir(exist_ok=True) + STATE_FILE.write_text(json.dumps(state, indent=2) + "\n") + + +# --- org parsing (just enough for DIWAN_ID + Close-session bookkeeping) --- + +def read_lines(path): + return Path(path).read_text(encoding="utf-8").splitlines() + + +def write_lines(path, lines): + Path(path).write_text("\n".join(lines) + "\n", encoding="utf-8") + + +def parse_headline(line): + m = HEADLINE_RE.match(line) + if not m: + return None + return len(m.group(1)), m.group(2) + + +def heading_title_for_slug(rest): + body = TAGS_RE.sub("", rest) + words = body.split() + if words and words[0] in TODO_KEYWORDS: + words = words[1:] + return " ".join(words) + + +def slugify(text): + """Produce a herdr agent-name-safe slug: ^[a-z][a-z0-9_-]{0,31}$.""" + text = re.sub(r"\[\[.*?\]\[(.*?)\]\]", r"\1", text) # org link -> description + text = text.lower() + text = re.sub(r"[^a-z0-9]+", "-", text).strip("-") + text = text[:24] or "task" + if not re.match(r"^[a-z]", text): + text = "d-" + text + return text[:24] + + +def find_drawer_insert_point(lines, idx): + """idx is a headline's line index. Returns (insert_idx, drawer) where + drawer is (start, end) of an existing :PROPERTIES:...:END: block (end = + index of the :END: line) right after the headline/planning line, or + None if there isn't one yet — in which case insert_idx is where one + should be created.""" + j = idx + 1 + if j < len(lines) and re.match(r"^\s*(SCHEDULED|DEADLINE|CLOSED)\b", lines[j]): + j += 1 + if j < len(lines) and lines[j].strip() == ":PROPERTIES:": + start = j + k = j + 1 + while k < len(lines) and lines[k].strip() != ":END:": + k += 1 + return j, (start, k) + return j, None + + +def get_property(lines, idx, key): + _, drawer = find_drawer_insert_point(lines, idx) + if not drawer: + return None + start, end = drawer + for line in lines[start + 1 : end]: + m = re.match(r"^\s*:([\w_]+):\s*(.*)$", line) + if m and m.group(1).upper() == key.upper(): + return m.group(2).strip() + return None + + +def set_property(lines, idx, key, value): + insert_idx, drawer = find_drawer_insert_point(lines, idx) + if drawer: + start, end = drawer + for i in range(start + 1, end): + m = re.match(r"^\s*:([\w_]+):\s*(.*)$", lines[i]) + if m and m.group(1).upper() == key.upper(): + lines[i] = f" :{key}: {value}" + return + lines.insert(end, f" :{key}: {value}") + else: + lines[insert_idx:insert_idx] = [" :PROPERTIES:", f" :{key}: {value}", " :END:"] + + +def assign_diwan_id(lines, idx, level, diwan_id, child_text="Close diwan session"): + set_property(lines, idx, "DIWAN_ID", diwan_id) + _, drawer = find_drawer_insert_point(lines, idx) # recompute: now guaranteed present + insert_at = drawer[1] + 1 + stars = "*" * (level + 1) + lines.insert(insert_at, f"{stars} [ ] {child_text}") + + +def id_in_use(diwan_id): + pattern = re.compile(rf":DIWAN_ID:\s*{re.escape(diwan_id)}\s*$", re.MULTILINE) + for path in CONTENT_ROOT.rglob("*.org"): + if ".diwan" in path.parts: + continue + try: + text = path.read_text(encoding="utf-8", errors="ignore") + except OSError: + continue + if pattern.search(text): + return True + return False + + +def unique_id(base): + if not id_in_use(base): + return base + for i in range(2, 50): + candidate = f"{base}-{i}" + if not id_in_use(candidate): + return candidate + return f"{base}-{uuid.uuid4().hex[:6]}" + + +# --- herdr ----------------------------------------------------------------- +# Real `herdr` CLI (workspace/tab/pane/agent over a JSON socket API). Verified +# live against herdr 0.9.5 — see `herdr --skill` for the authoritative +# reference if this drifts on a future version. + +def herdr_available(): + return shutil.which("herdr") is not None + + +class HerdrError(RuntimeError): + pass + + +def herdr_json(*args): + res = subprocess.run(["herdr", *args], cwd=CONTENT_ROOT, capture_output=True, text=True) + if res.returncode != 0: + raise HerdrError(f"herdr {' '.join(args)} failed: {res.stderr.strip() or res.stdout.strip()}") + try: + return json.loads(res.stdout) + except json.JSONDecodeError: + raise HerdrError(f"herdr {' '.join(args)}: non-JSON output: {res.stdout.strip()}") + + +def herdr_diwan_workspace_id(): + data = herdr_json("workspace", "list") + for ws in data["result"]["workspaces"]: + if ws.get("label") == DIWAN_WORKSPACE_LABEL: + return ws["workspace_id"] + created = herdr_json("workspace", "create", "--label", DIWAN_WORKSPACE_LABEL, "--no-focus") + return created["result"]["workspace"]["workspace_id"] + + +def herdr_find_tab_by_label(workspace_id, label): + data = herdr_json("tab", "list", "--workspace", workspace_id) + for t in data["result"]["tabs"]: + if t.get("label") == label: + return t["tab_id"] + return None + + +def herdr_tab_root_pane(workspace_id, tab_id): + data = herdr_json("pane", "list", "--workspace", workspace_id) + for p in data["result"]["panes"]: + if p.get("tab_id") == tab_id: + return p["pane_id"] + return None + + +def herdr_agent_alive(name): + try: + herdr_json("agent", "get", name) + return True + except HerdrError: + return False + + +def herdr_agent_start_args(name, pane_id): + args = ["agent", "start", name, "--kind", AGENT_KIND, "--pane", pane_id] + if AGENT_KIND == "claude" and CLAUDE_PERMISSION_MODE: + args += ["--", "--permission-mode", CLAUDE_PERMISSION_MODE] + elif AGENT_KIND == "pi" and PI_APPROVE_FLAG: + args += ["--", PI_APPROVE_FLAG] + return args + + +def herdr_start_agent_in_new_tab(workspace_id, name, prompt): + created = herdr_json( + "tab", "create", "--workspace", workspace_id, + "--cwd", str(CONTENT_ROOT), "--label", name, "--no-focus", + ) + tab_id = created["result"]["tab"]["tab_id"] + pane_id = created["result"]["root_pane"]["pane_id"] + try: + herdr_json(*herdr_agent_start_args(name, pane_id)) + herdr_json("agent", "prompt", name, prompt) + except HerdrError: + # don't leave an agent-less orphan tab behind on failure (e.g. name + # rejected by herdr's naming rule) + try: + herdr_json("tab", "close", tab_id) + except HerdrError: + pass + raise + + +def herdr_dispatch_fresh(name, prompt): + """First dispatch for `name` (new tick run, or a heading's first diwan do).""" + if not herdr_available(): + print(f"[dry-run] herdr not found — would create a '{name}' tab in the " + f"'{DIWAN_WORKSPACE_LABEL}' workspace, start a {AGENT_KIND} agent, and prompt it.") + print(f" prompt: {prompt}") + return + try: + ws_id = herdr_diwan_workspace_id() + herdr_start_agent_in_new_tab(ws_id, name, prompt) + except HerdrError as e: + print(f"warning: herdr dispatch for '{name}' failed: {e}", file=sys.stderr) + + +def herdr_dispatch_continuation(name, prompt): + """Resume `name`: prompt it if its agent is still live, otherwise find its + tab by label and restart an agent there. Recreates the tab from scratch + only if that tab itself is gone (e.g. closed by hand).""" + if not herdr_available(): + print(f"[dry-run] herdr not found — would prompt/resume '{name}' in the " + f"'{DIWAN_WORKSPACE_LABEL}' workspace.") + print(f" prompt: {prompt}") + return + try: + if herdr_agent_alive(name): + herdr_json("agent", "prompt", name, prompt) + return + ws_id = herdr_diwan_workspace_id() + tab_id = herdr_find_tab_by_label(ws_id, name) + pane_id = herdr_tab_root_pane(ws_id, tab_id) if tab_id else None + if pane_id is None: + herdr_start_agent_in_new_tab(ws_id, name, prompt) + return + herdr_json(*herdr_agent_start_args(name, pane_id)) + herdr_json("agent", "prompt", name, prompt) + except HerdrError as e: + print(f"warning: herdr resume for '{name}' failed: {e}", file=sys.stderr) + + +# --- commands -------------------------------------------------------------- + +def cmd_bare(args): + if not herdr_available(): + print("herdr not found on PATH. Install herdr: https://herdr.dev/") + print("Once installed, `diwan` will attach you to the 'diwan' workspace.") + return + try: + ws_id = herdr_diwan_workspace_id() + herdr_json("workspace", "focus", ws_id) + except HerdrError as e: + print(f"warning: couldn't focus the '{DIWAN_WORKSPACE_LABEL}' workspace: {e}", file=sys.stderr) + os.execvp("herdr", ["herdr"]) + + +def cmd_tick(args): + DIWAN_DIR.mkdir(exist_ok=True) + if git_is_dirty(): + if git_commit_all("chore: snapshot manual edits before tick"): + print("Committed pending manual edits before starting tick.") + + state = load_state() + last = state.get("last_tick_commit") + head = git_head() + + if last is None: + changed = True + range_desc = f"(initial)..{head[:8]}" + range_arg = "(initial import)" + else: + changed = git_diff_has_changes(last, head, excludes=[".diwan"]) + range_desc = f"{last[:8]}..{head[:8]}" + range_arg = f"{last}..{head}" + + if not changed and not args.force: + print(f"No changes since last tick ({range_desc}); nothing to do. Use --force to run anyway.") + return + + tab = "tick-" + datetime.now(timezone.utc).strftime("%Y%m%d-%H%M%S") + prompt = ( + f"You are a diwan tick agent for the content repo at {CONTENT_ROOT}. " + f"Read {TOOL_ROOT}/PROTOCOL.md and AGENTS.md, then follow {TOOL_ROOT}/TICK.md exactly. " + f"Change range since last completed tick: {range_arg}." + ) + herdr_dispatch_fresh(tab, prompt) + + history = state.get("tick_history", []) + history.append(tab) + dropped, kept = history[:-MAX_TICK_TABS], history[-MAX_TICK_TABS:] + state["tick_history"] = kept + save_state(state) + git_commit_all(f"diwan: record tick dispatch {tab}", diwan_identity=True) + + print(f"Dispatched tick agent: herdr terminal '{tab}' ({range_desc}).") + if dropped and herdr_available(): + try: + ws_id = herdr_diwan_workspace_id() + for old_tab in dropped: + old_tab_id = herdr_find_tab_by_label(ws_id, old_tab) + if old_tab_id: + herdr_json("tab", "close", old_tab_id) + print(f"Pruned {len(dropped)} tick terminal(s) past the retention window: {', '.join(dropped)}.") + except HerdrError as e: + print(f"warning: couldn't prune old tick terminal(s) {dropped}: {e}", file=sys.stderr) + elif dropped: + print(f"note: {len(dropped)} older tick terminal(s) ({', '.join(dropped)}) " + f"are past the retention window — herdr not on PATH, clean up by hand.") + + +def cmd_do(args): + target = args.target + diwan_id_arg = args.diwan_id + + m = re.match(r"^(.+):(\d+)$", target) + heading_mode = bool(m) and Path(m.group(1)).is_file() + + if heading_mode: + path_str, lineno = m.group(1), int(m.group(2)) + path = Path(path_str) + lines = read_lines(path) + idx = lineno - 1 + if not (0 <= idx < len(lines)): + print(f"error: {target}: line out of range", file=sys.stderr) + sys.exit(1) + parsed = parse_headline(lines[idx]) + if not parsed: + print(f"error: {target} is not an org headline", file=sys.stderr) + sys.exit(1) + level, rest = parsed + + existing_id = get_property(lines, idx, "DIWAN_ID") + if diwan_id_arg and existing_id and diwan_id_arg != existing_id: + print( + f"error: heading already has DIWAN_ID={existing_id}, " + f"conflicts with --diwan-id={diwan_id_arg}", + file=sys.stderr, + ) + sys.exit(1) + + is_continuation = existing_id is not None + if is_continuation: + diwan_id = existing_id + if not HERDR_NAME_RE.match(diwan_id): + print( + f"error: existing DIWAN_ID '{diwan_id}' on {target} is not a valid " + f"herdr agent name (^[a-z][a-z0-9_-]{{0,31}}$) — fix the heading's " + f":DIWAN_ID: property by hand before retrying.", + file=sys.stderr, + ) + sys.exit(1) + else: + base = diwan_id_arg or slugify(heading_title_for_slug(rest)) + diwan_id = base if diwan_id_arg else unique_id(base) + if not HERDR_NAME_RE.match(diwan_id): + print( + f"error: --diwan-id '{diwan_id}' is not a valid herdr agent name " + f"(^[a-z][a-z0-9_-]{{0,31}}$)", + file=sys.stderr, + ) + sys.exit(1) + assign_diwan_id(lines, idx, level, diwan_id) + write_lines(path, lines) + git_commit_all( + f"diwan: assign DIWAN_ID={diwan_id} to {path_str}:{lineno}", + diwan_identity=True, + ) + + title = heading_title_for_slug(rest) + prompt = ( + f"You are a diwan do agent, DIWAN_ID={diwan_id}, for the content repo at {CONTENT_ROOT}. " + f'Your heading is {path_str}:{lineno} ("{title}"). ' + f"Read {TOOL_ROOT}/PROTOCOL.md and AGENTS.md, then follow {TOOL_ROOT}/DO.md." + ) + if is_continuation: + herdr_dispatch_continuation(diwan_id, prompt) + else: + herdr_dispatch_fresh(diwan_id, prompt) + else: + diwan_id = diwan_id_arg or f"adhoc-{uuid.uuid4().hex[:8]}" + herdr_dispatch_continuation(diwan_id, target) + + print(diwan_id) + + +def main(): + parser = argparse.ArgumentParser(prog="diwan") + sub = parser.add_subparsers(dest="command") + + p_tick = sub.add_parser("tick", help="run a diwan tick pass") + p_tick.add_argument("--force", action="store_true", help="dispatch even if nothing changed since last tick") + + p_do = sub.add_parser("do", help="dispatch/resume a diwan do session") + p_do.add_argument("target", help="file:lineno pointing at a heading, or a free-text prompt") + p_do.add_argument("--diwan-id", dest="diwan_id", default=None) + + args = parser.parse_args() + if args.command is None: + cmd_bare(args) + elif args.command == "tick": + cmd_tick(args) + elif args.command == "do": + cmd_do(args) + + +if __name__ == "__main__": + main()