From d52f733f14b442e96f5cd41e3eacd5e1411485dc Mon Sep 17 00:00:00 2001 From: marc Date: Mon, 9 Nov 2026 10:40:19 +0100 Subject: [PATCH] rename TOOL_ROOT/CONTENT_ROOT to DIWAN_DIR/DIWAN_NOTES_DIR DIWAN_DIR: where diwan itself is installed (was TOOL_ROOT / DIWAN_TOOL_ROOT). DIWAN_NOTES_DIR: the content repo diwan is invoked against (was CONTENT_ROOT); now also has its own env var override, not just git rev-parse --show-toplevel discovery, for cron/systemd use where cwd may not be the content repo. The content repo's own runtime-bookkeeping directory (state.json, .lock) is renamed STATE_DIR internally to avoid colliding with the new DIWAN_DIR name; its on-disk path is unchanged (still a .diwan/ subdirectory of the content repo). --- DESIGN.md | 10 +++++----- diwan | 43 +++++++++++++++++++++++-------------------- 2 files changed, 28 insertions(+), 25 deletions(-) diff --git a/DESIGN.md b/DESIGN.md index ceeca22..dcfb3a6 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -29,7 +29,7 @@ Three layers, each independently simple: - **Git**, in the *content* repo, no remote (for Marc's prototype content repo — a real deployment may add one, e.g. for Nextcloud sync, but that's the content repo's business, not this one's). Every automated write to the content repo is a commit. This *is* the revert mechanism (principle #2) — not a bespoke undo log. Commits are authored as `diwan ` when diwan itself is the actor, or the user's own git identity when `diwan tick` snapshots whatever manual edits were lying around at the start of a run (so no tick ever operates on a dirty/mixed workspace, and "who changed this" is answerable from `git log --author` alone). This repo (the tool) has its own, unrelated git history — just ordinary source control for diwan's own development. - **herdr**, a terminal-multiplexer-for-agents with a JSON-over-socket CLI (`herdr workspace/tab/pane/agent ...`). diwan doesn't invent a session store: one herdr workspace labeled `diwan` holds every tick run and every open `do` session as a separate, independently addressable, persistent tab. See "herdr integration" below for the exact calls. -- **The `diwan` CLI** (`./diwan` at this repo's root), a single dependency-free Python script (shebang runs it through `nix shell nixpkgs#python3`, so no system Python install needed on NixOS; a real flake package would wrap this properly instead). It does only the mechanical, deterministic parts: git bookkeeping, enough org parsing to read/write a `DIWAN_ID` property and insert a `Close diwan session` checkbox item, and herdr dispatch. It never triages, writes prose, or makes judgment calls — that's the dispatched agent's job, guided by `TICK.md`/`DO.md`. It distinguishes two roots explicitly (see the script's own docstring): `TOOL_ROOT` (where it and its docs are installed — `Path(__file__).resolve().parent`, overridable via `DIWAN_TOOL_ROOT` for whatever a future flake's install layout needs) and `CONTENT_ROOT` (whatever notes repo it's invoked against, found via `git rev-parse --show-toplevel` from the current directory). `diwan` must be run from inside (or below) a content repo — it has no content of its own to act on. +- **The `diwan` CLI** (`./diwan` at this repo's root), a single dependency-free Python script (shebang runs it through `nix shell nixpkgs#python3`, so no system Python install needed on NixOS; a real flake package would wrap this properly instead). It does only the mechanical, deterministic parts: git bookkeeping, enough org parsing to read/write a `DIWAN_ID` property and insert a `Close diwan session` checkbox item, and herdr dispatch. It never triages, writes prose, or makes judgment calls — that's the dispatched agent's job, guided by `TICK.md`/`DO.md`. It distinguishes two roots explicitly (see the script's own docstring): `DIWAN_DIR` (where it and its docs are installed — `Path(__file__).resolve().parent`, overridable via the `DIWAN_DIR` env var for whatever a future flake's install layout needs) and `DIWAN_NOTES_DIR` (whatever notes repo it's invoked against, found via `git rev-parse --show-toplevel` from the current directory by default, overridable via the `DIWAN_NOTES_DIR` env var for e.g. a cron/systemd timer that doesn't cd into it first). `diwan` must be run from inside (or below) a content repo, or have `DIWAN_NOTES_DIR` set — it has no content of its own to act on. Everything else (what counts as "trivial", where an inbox item belongs, how to answer a chat question) is intentionally **not** code — it's prose in `TICK.md`/`DO.md` that whatever LLM is dispatched has to interpret. This is the harness-agnosticism principle made concrete: the CLI's job ends at "here's a prompt, here's a terminal," everything judgment-shaped is left to the model. @@ -54,7 +54,7 @@ Two halves: a deterministic CLI half (`cmd_tick` in `diwan`) and a judgment-driv 1. If the content repo's working tree is dirty, commit everything under the user's own git identity (`chore: snapshot manual edits before tick`) — this is what makes "no manual commit requirement" (PROTOCOL.md) actually true. 2. Load the content repo's `.diwan/state.json`'s `last_tick_commit`. Diff that commit against current `HEAD` (in the content repo), excluding `.diwan/` itself (so diwan's own bookkeeping commits never count as "something changed"). No prior tick → treat as "always changed." 3. If nothing changed and `--force` wasn't passed: print and exit. This is the only gate on whether a tick does anything at all. -4. Otherwise, pick a tab name `tick-` (lowercase + digits + hyphens only — see "herdr integration" for why that's not negotiable) and dispatch a fresh agent there (`herdr_dispatch_fresh`), in a pane whose working directory is the *content* repo, with a prompt that says: read `/PROTOCOL.md` and `AGENTS.md` (the latter resolved relative to the agent's own cwd, i.e. the content repo), follow `/TICK.md`, here's the commit range. **This dispatch is fire-and-forget** — the CLI call returns as soon as the prompt is submitted, it does not wait for the tick to finish. +4. Otherwise, pick a tab name `tick-` (lowercase + digits + hyphens only — see "herdr integration" for why that's not negotiable) and dispatch a fresh agent there (`herdr_dispatch_fresh`), in a pane whose working directory is the *content* repo (`DIWAN_NOTES_DIR`), with a prompt that says: read `/PROTOCOL.md` and `AGENTS.md` (the latter resolved relative to the agent's own cwd, i.e. the content repo), follow `/TICK.md`, here's the commit range. **This dispatch is fire-and-forget** — the CLI call returns as soon as the prompt is submitted, it does not wait for the tick to finish. 5. Record the new tab name in the content repo's `state.json`'s `tick_history` (kept to the last `MAX_TICK_TABS`, default 3, override via `DIWAN_MAX_TICK_TABS`), commit that under the diwan identity, and — if this pushed any older tick tabs out of the retention window — close those tabs for real via `herdr tab close`. Note what's *not* here: `last_tick_commit` is deliberately **not** updated by the CLI at dispatch time. It's updated by the dispatched agent itself, as the literal last step of `TICK.md`, only if everything else succeeded. If a tick agent crashes or is killed partway, the next tick sees the same "changed" range again rather than silently skipping work — correctness over tidiness. @@ -66,7 +66,7 @@ Note what's *not* here: `last_tick_commit` is deliberately **not** updated by th 4. Write one short, titled, checkbox subheading per notable thing this run did into `diwan.org`'s Reports log (not one paragraph for the whole run), each with an optional `:LINK:` property and a 1-2 line signed body; delete any already-`[X]` subheadings first. 5. Only if 1–4 all succeeded: update `state.json.last_tick_commit` to current `HEAD` and commit under the diwan identity. -**Verified live** (2026-11-08, herdr 0.9.5 + Claude Code, `claude` kind, against the old mixed tool+content repo, before the 2026-11-09 split): dispatched for real, ran to completion with zero permission prompts (see "Known gaps" for why that almost didn't work), correctly left two already-open `:session:`-tagged sessions untouched (back when `:session:` was still spelled `:diwan:`, before the tag rename — see "Known gaps" for what's unverified post-rename), correctly found nothing to triage, wrote a proper signed report, and updated `state.json` correctly. A second immediate `diwan tick` correctly no-op'd. **Not yet re-verified** against the post-split layout (tool repo + separate `diwan-dev-notes` content repo, `TOOL_ROOT`/`CONTENT_ROOT` as two different directories) — see "Known gaps". +**Verified live** (2026-11-08, herdr 0.9.5 + Claude Code, `claude` kind, against the old mixed tool+content repo, before the 2026-11-09 split): dispatched for real, ran to completion with zero permission prompts (see "Known gaps" for why that almost didn't work), correctly left two already-open `:session:`-tagged sessions untouched (back when `:session:` was still spelled `:diwan:`, before the tag rename — see "Known gaps" for what's unverified post-rename), correctly found nothing to triage, wrote a proper signed report, and updated `state.json` correctly. A second immediate `diwan tick` correctly no-op'd. **Not yet re-verified with a real dispatched agent** against the post-split layout (tool repo + separate `diwan-dev-notes` content repo, `DIWAN_DIR`/`DIWAN_NOTES_DIR` as two different directories) — path resolution itself has been dry-run tested (see below), but no agent has actually followed the resulting prompt yet. See "Known gaps". ## What `diwan do` actually does @@ -96,8 +96,8 @@ Lifecycle: **fresh** (no property) → `diwan do` mints an ID, creates the tab+a ## Known gaps / explicit non-goals (don't "fix" these without discussing) -- **Tool/content repo split just happened (2026-11-09), not yet re-verified live.** This repo used to be both diwan's own dev copy *and* a throwaway test-content repo at once; it's now tool-only, with content split out to a sibling `diwan-dev-notes` repo. The CLI was updated to resolve `TOOL_ROOT` (its own install dir, `Path(__file__).resolve().parent`, overridable via `DIWAN_TOOL_ROOT`) separately from `CONTENT_ROOT` (`git rev-parse --show-toplevel` from the invoking directory), and the dispatch prompts now pass `TOOL_ROOT`-absolute paths for `PROTOCOL.md`/`TICK.md`/`DO.md`. This has been syntax-checked and smoke-tested for path resolution, but **not yet run through a full live tick/do pass** against the new two-repo layout — do that before trusting it for real. -- **No flake yet.** The split prepared the ground (tool repo is self-contained, no content-repo assumptions baked into its own files) but there's no `flake.nix` in this repo yet, and the shebang still runs via a bare `nix shell nixpkgs#python3` rather than a proper package. Until that lands, "installed on `PATH`" means manually symlinking/aliasing `diwan` somewhere on `PATH` yourself. `DIWAN_TOOL_ROOT` exists specifically so a future flake's install layout (e.g. docs copied into `$out/share/diwan/` alongside a wrapped binary) can override the default `Path(__file__).resolve().parent` guess if it doesn't hold. +- **Tool/content repo split just happened (2026-11-09), not yet re-verified live.** This repo used to be both diwan's own dev copy *and* a throwaway test-content repo at once; it's now tool-only, with content split out to a sibling `diwan-dev-notes` repo. The CLI was updated to resolve `DIWAN_DIR` (its own install dir, `Path(__file__).resolve().parent`, overridable via the `DIWAN_DIR` env var) separately from `DIWAN_NOTES_DIR` (`git rev-parse --show-toplevel` from the invoking directory by default, overridable via the `DIWAN_NOTES_DIR` env var), and the dispatch prompts now pass `DIWAN_DIR`-absolute paths for `PROTOCOL.md`/`TICK.md`/`DO.md`. This has been syntax-checked and smoke-tested for path resolution (including both env var overrides, from outside any repo), but **not yet run through a full live tick/do pass with a real dispatched agent** against the new two-repo layout — do that before trusting it for real. +- **No flake yet.** The split prepared the ground (tool repo is self-contained, no content-repo assumptions baked into its own files) but there's no `flake.nix` in this repo yet, and the shebang still runs via a bare `nix shell nixpkgs#python3` rather than a proper package. Until that lands, "installed on `PATH`" means manually symlinking/aliasing `diwan` somewhere on `PATH` yourself. `DIWAN_DIR` exists specifically so a future flake's install layout (e.g. docs copied into `$out/share/diwan/` alongside a wrapped binary) can override the default `Path(__file__).resolve().parent` guess if it doesn't hold. - **Retired-session teardown.** `TICK.md` step 3 says a closed session has "nothing for you to do to the heading's content" and that herdr-side teardown "is not your job" — but nothing else currently closes that tab either. Right now a `Close diwan session` checked off (`[X]`) just... sits there, tab still open. This is a real gap, not yet decided whether the CLI should gain a path for it or whether `TICK.md` should just say to run `herdr tab close` directly. - **No real conversation continuity across resumption.** Covered above — intentional for now, but worth revisiting if resumed `do` sessions turn out to need more context than the heading text carries. - **`pi` kind is wired up and smoke-tested, but not yet exercised against real heading content.** `DIWAN_HARNESS=pi` gets its own equivalent of the Claude autonomy flag (`--approve`, see "`pi` kind, verified live" above) and has been confirmed live for ad-hoc `diwan do` dispatch and continuation. Still untested: `diwan tick` with `DIWAN_HARNESS=pi`, and `diwan do` against a real `:` heading (DIWAN_ID minting, `:chat:` protocol, scope discipline) rather than a free-text ad-hoc prompt. diff --git a/diwan b/diwan index 6633758..273f2c6 100755 --- a/diwan +++ b/diwan @@ -15,12 +15,15 @@ over a JSON socket API — see `herdr --skill`), verified live against herdr have run instead of failing, so the rest of the pipeline stays testable without herdr installed. -Two separate roots: TOOL_ROOT is where diwan itself (this script, plus +Two separate roots: DIWAN_DIR is where diwan itself (this script, plus PROTOCOL.md/TICK.md/DO.md) is installed — `diwan` is meant to live on PATH -via a nix flake, independent of any one person's notes. CONTENT_ROOT is -whatever notes repo it's being invoked against, discovered from the current -directory via `git rev-parse --show-toplevel` — diwan must be run from -inside (or below) that repo. +via a nix flake, independent of any one person's notes; override with the +DIWAN_DIR env var if the default (this script's own directory) doesn't +hold. DIWAN_NOTES_DIR is whatever notes repo it's being invoked against, +discovered from the current directory via `git rev-parse --show-toplevel` +by default (diwan must then be run from inside, or below, that repo) or +set explicitly via the DIWAN_NOTES_DIR env var (e.g. for a cron/systemd +timer that doesn't cd into it first). """ import argparse import contextlib @@ -35,27 +38,27 @@ import uuid from datetime import datetime, timezone from pathlib import Path -TOOL_ROOT = Path(os.environ.get("DIWAN_TOOL_ROOT", str(Path(__file__).resolve().parent))) +DIWAN_DIR = Path(os.environ.get("DIWAN_DIR", str(Path(__file__).resolve().parent))) -def find_content_root(): +def find_notes_dir(): res = subprocess.run( ["git", "rev-parse", "--show-toplevel"], capture_output=True, text=True ) if res.returncode != 0: print( "error: not inside a git repo — run diwan from inside the notes repo " - "you want it to act on.", + "you want it to act on, or set DIWAN_NOTES_DIR.", file=sys.stderr, ) sys.exit(1) return Path(res.stdout.strip()) -CONTENT_ROOT = find_content_root() -DIWAN_DIR = CONTENT_ROOT / ".diwan" -STATE_FILE = DIWAN_DIR / "state.json" -LOCK_FILE = DIWAN_DIR / ".lock" +DIWAN_NOTES_DIR = Path(os.environ["DIWAN_NOTES_DIR"]) if "DIWAN_NOTES_DIR" in os.environ else find_notes_dir() +STATE_DIR = DIWAN_NOTES_DIR / ".diwan" +STATE_FILE = STATE_DIR / "state.json" +LOCK_FILE = STATE_DIR / ".lock" MAX_TICK_TABS = int(os.environ.get("DIWAN_MAX_TICK_TABS", "3")) AGENT_KIND = os.environ.get("DIWAN_HARNESS", "claude") DIWAN_WORKSPACE_LABEL = "diwan" @@ -84,7 +87,7 @@ TODO_KEYWORDS = {"TODO", "IN-PROGRESS", "WAITING", "DONE", "CANCELLED"} def run_git(args, check=True, capture=False): return subprocess.run( - ["git", *args], cwd=CONTENT_ROOT, check=check, capture_output=capture, text=True + ["git", *args], cwd=DIWAN_NOTES_DIR, check=check, capture_output=capture, text=True ) @@ -238,7 +241,7 @@ def assign_diwan_id(lines, idx, level, diwan_id, child_text="Close diwan session def id_in_use(diwan_id): pattern = re.compile(rf":DIWAN_ID:\s*{re.escape(diwan_id)}\s*$", re.MULTILINE) - for path in CONTENT_ROOT.rglob("*.org"): + for path in DIWAN_NOTES_DIR.rglob("*.org"): if ".diwan" in path.parts: continue try: @@ -274,7 +277,7 @@ class HerdrError(RuntimeError): def herdr_json(*args): - res = subprocess.run(["herdr", *args], cwd=CONTENT_ROOT, capture_output=True, text=True) + res = subprocess.run(["herdr", *args], cwd=DIWAN_NOTES_DIR, capture_output=True, text=True) if res.returncode != 0: raise HerdrError(f"herdr {' '.join(args)} failed: {res.stderr.strip() or res.stdout.strip()}") try: @@ -328,7 +331,7 @@ def herdr_agent_start_args(name, pane_id): def herdr_start_agent_in_new_tab(workspace_id, name, prompt): created = herdr_json( "tab", "create", "--workspace", workspace_id, - "--cwd", str(CONTENT_ROOT), "--label", name, "--no-focus", + "--cwd", str(DIWAN_NOTES_DIR), "--label", name, "--no-focus", ) tab_id = created["result"]["tab"]["tab_id"] pane_id = created["result"]["root_pane"]["pane_id"] @@ -424,8 +427,8 @@ def cmd_tick(args): tab = "tick-" + datetime.now(timezone.utc).strftime("%Y%m%d-%H%M%S") prompt = ( - f"You are a diwan tick agent for the content repo at {CONTENT_ROOT}. " - f"Read {TOOL_ROOT}/PROTOCOL.md and AGENTS.md, then follow {TOOL_ROOT}/TICK.md exactly. " + f"You are a diwan tick agent for the content repo at {DIWAN_NOTES_DIR}. " + f"Read {DIWAN_DIR}/PROTOCOL.md and AGENTS.md, then follow {DIWAN_DIR}/TICK.md exactly. " f"Change range since last completed tick: {range_arg}." ) herdr_dispatch_fresh(tab, prompt) @@ -513,9 +516,9 @@ def cmd_do(args): title = heading_title_for_slug(rest) prompt = ( - f"You are a diwan do agent, DIWAN_ID={diwan_id}, for the content repo at {CONTENT_ROOT}. " + f"You are a diwan do agent, DIWAN_ID={diwan_id}, for the content repo at {DIWAN_NOTES_DIR}. " f'Your heading is {path_str}:{lineno} ("{title}"). ' - f"Read {TOOL_ROOT}/PROTOCOL.md and AGENTS.md, then follow {TOOL_ROOT}/DO.md." + f"Read {DIWAN_DIR}/PROTOCOL.md and AGENTS.md, then follow {DIWAN_DIR}/DO.md." ) if is_continuation: herdr_dispatch_continuation(diwan_id, prompt)