worktree: Collapse untracked directories in Status. Fixes #181

[cedric-appdirect]

Why

Worktree.Status() lists every untracked file individually, while reference git status reports a fully untracked directory as a single entry by default (status.showUntrackedFiles=normal, see git-status docs and status.showUntrackedFiles).

On worktrees with many untracked, non-ignored files — e.g. dotfiles repositories opened over a home directory as reported in #181 — Status() enumerates, lstats and allocates a merkletrie node, a Change and a FileStatus per untracked file. In a reproduction with a 30k-untracked-file worktree Status() takes ~2s where git status takes ~0.1s; the reporter measured 326s on their home directory.

What

• New StatusOptions.UntrackedFiles mirroring git's status.showUntrackedFiles:
  • UntrackedFilesNormal (zero value, new default): untracked directories are reported as a single entry, like git status (?? dir/).
  • UntrackedFilesAll: every untracked file listed individually — go-git's historical behavior.
  • UntrackedFilesNo: untracked entries omitted, like git status -uno.
• The collapse is implemented in the diff machinery: a new noder.Collapser interface lets a directory noder declare its subtree representable by one change; Changes.addRecursive then emits a single change for the directory instead of descending.
• The filesystem noder gets a CollapseUntrackedDirs option (requires Index, same contract as IgnoreMatcher). A directory collapses only when it has no tracked descendants and contains at least one visible (non-ignored) file, validated with an early-exit scan — so fully untracked directories are usually validated with one ReadDir per nesting level instead of a full enumeration. Empty untracked directories produce no entry, matching git.
• Internal callers that enumerate untracked files (Add, AddGlob, Clean, and the KeepReset conflict check) are pinned to UntrackedFilesAll and behave exactly as before.

The design mirrors upstream git's traversal (dir.c): UntrackedFilesNormal maps to DIR_SHOW_OTHER_DIRECTORIES | DIR_HIDE_EMPTY_DIRECTORIES, the tracked-descendant check to directory_exists_in_index(), and the early-exit visibility scan to read_directory_recursive()'s check_only mode.

Behavior change (v6)

Status() output for untracked files changes from per-file listing to collapsed directory entries (git parity). Consumers needing the old listing can opt into StatusOptions{UntrackedFiles: UntrackedFilesAll}. Verified against reference git status --porcelain on equivalent fixtures, including nested .gitignore un-ignore rules and tracked/untracked mixes.

Benchmarks

A/B comparison of the main-branch binary vs this branch on an identical fixture (5,000 untracked files across 550 directories, 100 tracked files; Apple M4 Pro; 7 runs each, -benchtime=5x), via benchstat:

                  │  before (main)  │  after (this PR)  │
                  │     sec/op      │   sec/op  vs base │
StatusUntracked5k    309.3m ± 29%    185.4m ± 31%  -40.04% (p=0.001 n=7)

The included BenchmarkStatusUntrackedDir reproduces this on demand (20k files / 2,200 dirs: All 1476ms/op vs Normal 903ms/op) and the status map holds 1 entry instead of 20,000. Note: the remaining cost is dominated by the gitignore.ReadPatterns pre-walk, which still scans the whole worktree; making that walk lazy/incremental (as upstream's prep_exclude does) is a natural follow-up PR.

Test plan

• New unit tests: utils/merkletrie (collapse insert/delete/non-collapse), utils/merkletrie/filesystem (collapse truth table: empty dirs, tracked descendants, no-index no-op, ignored contents, file→dir replacement, symlinks)
• New worktree tests: default collapse, UntrackedFilesAll, UntrackedFilesNo, and an Add("dir") regression test
• Existing TestStatusIgnored updated to the git-parity output (verified against git status --porcelain)
• go test .: 758 passed, 0 failed
• go test -race ./...: 4659 passed; 4 failures in plumbing/transport/http backend E2E tests that also fail on pristine main (local git backend infrastructure limitation)
• golangci-lint run clean on changed packages

AI disclosure

This contribution was produced with AI assistance (Kimi k3), including profiling analysis of #181, implementation and tests. Commits carry an Assisted-by trailer per AI_POLICY.md. The behavior was cross-checked against reference git on equivalent fixtures.