Design/progressive disclosure#145
Merged
Merged
Conversation
Canonical knowledge files grow without bound while their entries stay valid, so nothing can be dropped (time-sharding and a load-excluded cold bucket are already rejected). At scale an agent that legitimately wants system understanding reads every decision, then every learning, and exhausts its context window. Headings-first consumption is necessary but not sufficient — an agent can always choose to read the whole file — and consolidation does not help (the 2026-07-16 pass moved LEARNINGS only 98 -> 88 because the corpus is dense with distinct signal, not redundancy). Design: each canonical file becomes a BOUNDED ROOT — a staging zone (where `add` already writes) plus a `## Themes` section carrying a "just enough" per-theme gist and a link to a theme file holding that theme's bodies. Staging IS the watermark (un-digested by construction), so no state file is needed. An agent-driven, human-gated pass folds staged entries into themes and refreshes the touched gists. The structure is self-similar, so an overgrown theme file can itself become a root — nesting deferred, not precluded. Verified against the code: the add anchor (first line-start `## [`, falling back to AfterHeader) always lands above `## Themes` even when staging is empty, so the add path needs zero change; and because the root is bounded, the `ctx agent` packet needs no rewire either. Scope LEARNINGS/DECISIONS/CONVENTIONS; CONSTITUTION (small by design) and TASKS (auto-archived) excluded. Implementation remains open; the pass codified as a reusable skill is the final deliverable. Spec: specs/progressive-disclosure.md Signed-off-by: Jose Alekhinne <jose@ctx.ist>
Decompose specs/progressive-disclosure.md into specs/plans/pd-m1.md: 15 tasks across 5 epics, each with a falsifiable acceptance criterion, plus a test matrix where every invariant row is exercised by a named task. Milestone 1 deliberately moves nothing. The pass that relocates entry bodies (M2+) is the clobber/data-loss risk class, so the refusal machinery lands first: Parse (byte-exact round-trip), Validate, and the cross-file invariants (pairing, uniqueness, links). Staging is kept raw so the mover gets byte-exact bodies and nothing is silently normalized. T10-T12 are the milestone's measurement gate: they prove the design's load-bearing premise that `add` needs zero change because its anchor always lands above `## Themes` -- including the empty-staging AfterHeader fallback. If they fail, the spec's Layout section is wrong and must be revisited via /ctx-plan rather than patched downstream. Blocking-TBD gate found one real blocker: the spec's validate said "exactly one ## Themes" while the pass "creates it on first run" -- a contradiction for an un-migrated root. Resolved IN THE SPEC (validate accepts zero-or-one; zero = first run; 2+ refuses), since a resolution living only in the plan would be minting. Four deferrable TBDs are carried in Out of scope, annotated with the milestone at which each graduates to blocking. TASKS.md gains epic anchors only (E1-E5, partitioning T01-T15); the plan is the single source of truth for milestone progress. Spec: specs/progressive-disclosure.md Signed-off-by: Jose Alekhinne <jose@ctx.ist>
insert.AfterHeader returned content[:insertPoint] + entry, truncating the file at the insertion point and silently discarding the tail. Its sole caller is beforeFirstEntry's fallback, taken when a knowledge file has no `## [` entry — so `ctx learning add` / `ctx decision add` against an entry-less LEARNINGS.md/DECISIONS.md destroyed any content below the H1 header and its comment block. Same family as the clobber bug index.Validate guards: silent loss of persisted memory, git-only recovery. Masked today because an entry-less file from `ctx init` has nothing after the comment block (empty tail), so nothing was lost yet — but any hand-written section below the preamble of an as-yet-entry-less file was one add away from deletion. Fix: preserve the tail (the pattern sibling Task already uses). Empty tail returns the original bytes unchanged, so every file shape that works today is byte-identical; only the losing shape changes. When a tail follows, the entry is separated from it by a `---` rule. The package had no tests. Added insert_test.go (tail-preserved, empty-tail-unchanged, primary-path-unchanged) and a TestMain calling lookup.Init() — without it desc.Text returns "" and strings.Index(s,"") == 0 silently rewrites every insert anchor to offset 0, so tests would pass exercising a path production never takes. Verified end-to-end against the built binary: a Notes section below the preamble survives `ctx learning add` and the entry lands above it. Found by the layout proof for specs/progressive-disclosure.md; the fix stands alone and ships alone. Spec: specs/fix-afterheader-tail-truncation.md Signed-off-by: Jose Alekhinne <jose@ctx.ist>
Two gotchas from fixing the AfterHeader truncation: - desc.Text() returns "" in a test binary without lookup.Init(), and strings.Index(s, "") == 0 turns anchor-based inserts into offset-0 matches — tests pass exercising a path production never takes. - An insert helper returning content[:i]+x without content[i:] is append-with-truncation, not insert; a lone content[:i] is the smell. Spec: specs/fix-afterheader-tail-truncation.md Signed-off-by: Jose Alekhinne <jose@ctx.ist>
Prove the progressive-disclosure premise that `ctx X add` needs no accommodation: its insert anchor always lands a new entry in the staging zone for all three file kinds, populated AND empty staging. - LEARNINGS/DECISIONS (beforeFirstEntry): entry lands above `## Themes`, including the empty-staging AfterHeader fallback. - CONVENTIONS (AppendAtEnd): new `###` section lands inside the trailing `## Recent` staging zone, below `## Themes`. The gate fired first: T10's empty-staging case caught the pre-existing AfterHeader tail-truncation bug (destroying the Themes section). That bug is fixed separately and merged; these proofs are now green with unchanged assertions. Amendment logged in the plan. Plan: specs/plans/pd-m1.md Signed-off-by: Jose Alekhinne <jose@ctx.ist>
Milestone 1 of progressive disclosure: the refusal machinery and the structural vocabulary, with tests. Nothing moves — no entry body is relocated and no gist is authored (M2+ moves bodies, the clobber risk class, so the guards land first). New package internal/disclosure: - Parse splits a root (preamble | staging | themes) per kind and round-trips byte-for-byte via Reconstruct — the mover needs verbatim bodies, so nothing is normalized. Parse is total; all refusal is in Validate. - Validate is the fail-loud precondition: zero-or-one ## Themes (zero = not-yet-migrated first run), no entry below ## Themes, staging parses into discrete entries. It never regenerates from "what it recognized" — the failure mode behind the historic clobber bug. - CheckPairing / CheckUniqueness / CheckLinks are the cross-file invariants: gists <-> theme files 1:1, every entry in exactly one place (identity is timestamp+title, since same-second entries share a timestamp), every theme link resolves. Supporting: internal/config/disclosure (structural vocabulary), internal/err/disclosure (entity.Sentinel guards + err.disclosure.* i18n text). internal/compliance runs the invariants against this repo's own .context/ (vacuous pass — un-migrated) and proves-both-ways with a planted double-Themes violation. E4 (layout proofs, T10-T12) landed earlier; this commit is E1-E3 + E5. Full suite green (714 pkgs), lint 0, make audit all-pass; canonical knowledge files byte-identical. Plan: specs/plans/pd-m1.md Signed-off-by: Jose Alekhinne <jose@ctx.ist>
Resolve the M2-blocking gist-format TBD in the spec (### Gist format): one bullet `- <name> — <gist> → [<name>](<noun>/<slug>.md)`, gist one line ≤~140 chars describing coverage, authored by the pass. Then decompose M2 into specs/plans/pd-m2.md: 14 tasks / 3 epics. M2 is the digesting pass in DRY-RUN only: a read-only `ctx disclosure inspect` CLI (built on M1's Parse) and a `ctx-digest` skill that proposes a theme per staged entry, authors proposed gists, and shows the plan -- moving nothing. The mover (append->verify->remove, gist write-back) is M3, kept behind this fully-exercised read+plan path. T12 is the measurement gate: the dry-run must produce a coherent theme plan and leave the fixture byte-identical. Rolling-wave gate satisfied (M1 DoD all [x]). TASKS.md gains E1-E3 epic anchors only, partitioning T01-T14. Spec: specs/progressive-disclosure.md Signed-off-by: Jose Alekhinne <jose@ctx.ist>
The dry-run pass's read foundation: the disclosure inspect model plus a
read-only CLI, built on M1's Parse. Nothing writes.
- disclosure.KindFor infers a root's kind from its canonical filename;
StagedEntries lists the staging zone's un-digested entries;
Inspect assembles the {kind, staging, themes} view (json-tagged).
- `ctx disclosure inspect <file> [--json]`: reports staged entries and
current themes for LEARNINGS/DECISIONS/CONVENTIONS, human or JSON.
Rejects a non-knowledge file with a typed, path-naming error. It is
read-only and verified to leave the file byte-identical.
New surfaces follow the project's conventions: err/disclosure sentinel +
format wrapper, config/embed/cmd Use/DescKey constants, commands.yaml +
write.yaml i18n text (bijection holds), write/disclosure output package,
and registration in bootstrap/group.go under the artifacts group.
Verified end-to-end against a built binary (human + JSON + rejection).
The mover (M3) will consume this same Inspection. E3 (the ctx-digest
dry-run skill) is next.
Plan: specs/plans/pd-m2.md
Signed-off-by: Jose Alekhinne <jose@ctx.ist>
The dry-run half of progressive disclosure, as a shipped skill. It inspects a knowledge root (ctx disclosure inspect --json), proposes a theme per staged entry (semantic, human-overridable), authors a one-line coverage gist per theme (spec ### Gist format), and presents the digest plan -- moving nothing. The skill explicitly writes no file and names the apply step (a later milestone) as where the move happens. Copilot CLI copy created and in sync. Measurement gate (T12) driven on a fixture of 5 staged learnings: the dry-run produced a coherent 2-theme plan with gists, every entry assigned once, and left the fixture byte-identical. Milestone 2 complete: all 14 tasks [x], DoD met, make audit passes, 717 packages green, canonical knowledge files untouched. The mover (append->verify->remove, gist write-back) is M3, which consumes the Inspection this milestone builds. Plan: specs/plans/pd-m2.md Signed-off-by: Jose Alekhinne <jose@ctx.ist>
Persist two learnings (the /ctx-task-out measurement gate that caught the AfterHeader data-loss bug; the new-package audit gauntlet), the PD-M3 task pointer, and the session handover. Signed-off-by: Jose Alekhinne <jose@ctx.ist>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.