Agent/indexed statistical groups

docs/docs/guides/memory.md

@@ -68,15 +68,21 @@ In this example, loading 10 million rows consumed roughly 800 MB of heap memory.
 
 ## 3. The `.sys.gc` Function
 
-Call `(.sys.gc 0)` to signal that the runtime should reclaim unused memory. Currently this is a lightweight hook that returns `0` — Rayforce uses deterministic ref counting and eager page release via `madvise` during buddy coalescing, so most memory is reclaimed automatically when references are dropped.
+Call `(.sys.gc)` to run allocator maintenance after references have been
+dropped. It drains cross-thread frees, flushes slab caches, coalesces free
+blocks, reclaims empty oversized pools, and incrementally releases aged free
+pages. Values themselves use deterministic reference counting; this is not a
+tracing collector.
 
 ```lisp
 (.sys.gc)  ; 0
 ```
 
 !!! note "Note"
 
-    Because Rayforce uses deterministic ref counting (not tracing GC), memory is freed immediately when the last reference is released. The buddy allocator coalesces blocks and releases pages back to the OS automatically. `(.sys.gc 0)` exists as a hook for future use.
+    Because Rayforce uses deterministic reference counting, objects become
+    reclaimable immediately when their last reference is released. `(.sys.gc)`
+    performs the allocator-side consolidation and page-return pass.
 
 ## 4. The `.sys.info` Function
 

docs/docs/language/functions.md

@@ -165,7 +165,7 @@ Operations on vectors as collections.
 | `lead` | unary | Shift values one row forward; last row is null/sentinel | `(lead [10 20 30])` → `[20 30 0Nl]` |
 | `deltas` | unary | Adjacent differences; first row is null | `(deltas [10 15 13])` → `[0Nl 5 -2]` |
 | `ratios` | unary | Adjacent ratios as f64; first row is null | `(ratios [2 4 8])` → `[0Nf 2.0 2.0]` |
-| `fills` | unary | Forward-fill nullable vectors | `(fills (as 'I64 (list 0N 2 0N)))` → `[0Nl 2 2]` |
+| `fills` | unary | Forward-fill nullable vectors, or every column of a table | `(fills (as 'I64 (list 0N 2 0N)))` → `[0Nl 2 2]` |
 | `sums` | unary | Running sum; nulls are skipped | `(sums [1 2 3])` → `[1 3 6]` |
 | `avgs` | unary | Running average over non-null values | `(avgs [2 4 6])` → `[2.0 3.0 4.0]` |
 | `mins` | unary | Running minimum | `(mins [3 1 2])` → `[3 1 1]` |

docs/docs/language/math.md

@@ -578,6 +578,8 @@ For common time-series scans, prefer the built-in DAG forms. They avoid per-row
 (mins [3 1 2])              ; [3 1 1]
 (maxs [3 1 2])              ; [3 3 3]
 (fills (as 'I64 (list 0N 2 0N))) ; [0Nl 2 2]
+(fills (table [t value]
+              (list [1 2 3] (as 'I64 (list 10 0N 30))))) ; fills each column
 (differ [1 1 2 2])          ; [true false true false]
 (msum 3 [1 2 3 4])          ; [1 3 6 9]
 (mavg 3 [1 2 3 4])          ; [1.0 1.5 2.0 3.0]

docs/docs/namespaces/index.md

@@ -15,6 +15,7 @@ Rayfall's builtins are organised under dotted namespaces. Names beginning with `
 | [`.idx.*`](idx.md) | Accelerator indexes: bloom, hash, sort, zone. |
 | [`.ipc.*`](ipc.md) | TCP client IPC and the server connection-hook accessor. |
 | [`.log.*`](log.md) | Write-ahead log: open, write, sync, snapshot, roll, replay, validate. |
+| [`.mem.*`](mem.md) | Value sizing and scoped time/allocation measurement. |
 | [`.os.*`](os.md) | Process environment: `getenv`, `setenv`. |
 | [`.repl.*`](repl.md) | Interactive REPL control — attach the local REPL to a remote server. |
 | [`.sys.*`](sys.md) | System info and shell-style commands: build, info, mem, prof, querylog, gc, exec, listen, timeit, env, cmd. |
@@ -33,4 +34,4 @@ When the server is started with `-U <password>`, the following dotted builtins a
 - `.sys.exec`, `.sys.cmd`, `.sys.listen`, `.sys.querylog.enable`
 - `.time.timer.set`, `.time.timer.del`
 
-`.attr.*`, `.col.*`, `.graph.*`, `.idx.*`, and pure read/inspect builtins (`.db.splayed.get`, `.db.parted.get`, `.db.parted.tables`, `.fs.size`, `.fs.list`, `.log.write`, `.log.sync`, `.log.validate`, `.sys.build`, `.sys.info`, `.sys.mem`, `.sys.prof`, `.sys.querylog`, `.sys.gc`, `.sys.env`, `.sys.args`, `.sys.timeit`, `.ipc.handle`, `.time.now`) are always available.
+`.attr.*`, `.col.*`, `.graph.*`, `.idx.*`, `.mem.*`, and pure read/inspect builtins (`.db.splayed.get`, `.db.parted.get`, `.db.parted.tables`, `.fs.size`, `.fs.list`, `.log.write`, `.log.sync`, `.log.validate`, `.sys.build`, `.sys.info`, `.sys.mem`, `.sys.prof`, `.sys.querylog`, `.sys.gc`, `.sys.env`, `.sys.args`, `.sys.timeit`, `.ipc.handle`, `.time.now`) are always available.

docs/docs/namespaces/mem.md

@@ -0,0 +1,71 @@
+# `.mem.*` — memory measurement
+
+Value-level memory inspection and explicitly scoped query measurement. These
+operations are unrestricted and do not change the measured value.
+
+## Reference
+
+| Function | Arity | Flags | Description |
+|---|---|---|---|
+| [`.mem.objsize`](#mem-objsize) | unary | — | Logical bytes retained by a value. |
+| [`.mem.ts`](#mem-ts) | 1 | special form | Evaluate once and return result plus time, allocation, peak-memory, result-size, and worker statistics. |
+
+## `.mem.objsize` { #mem-objsize }
+
+Signature: `(.mem.objsize value)`.
+
+Returns the logical byte size of the complete `ray_t` graph reachable from
+`value`. Each object header and live payload is counted; a shared child is
+counted once. The number deliberately excludes buddy-block slack, global
+symbol dictionaries, and opaque native handles, making it stable across
+allocator tuning. It is an in-memory size, not a serialized-wire size.
+
+```lisp
+(.mem.objsize 42)       ;; 32: one ray_t header
+(.mem.objsize [1 2 3])  ;; 56: 32-byte header + three I64 cells
+```
+
+## `.mem.ts` { #mem-ts }
+
+Signature: `(.mem.ts expression)`.
+
+`.mem.ts` is a special form: it evaluates `expression` exactly once inside a
+process-wide measurement scope. Secondary-worker allocations and worker busy
+time are included. Result sizing and construction of the returned statistics
+dictionary happen after the time/allocation scope closes.
+
+```lisp
+(set m (.mem.ts (sum (til 1000000))))
+(at m 'result)
+(at m 'peak-live-bytes)
+```
+
+The returned dictionary has these fields:
+
+| Key | Meaning |
+|---|---|
+| `result` | The expression's result. |
+| `time-ns` | Monotonic wall-clock duration. |
+| `memory-bytes` | Net process-memory delta from the shared per-query profiler/query-log measurement. |
+| `allocated-bytes` | Sum of allocator block bytes obtained during the scope, including temporary allocations. |
+| `freed-bytes` | Sum of allocator block bytes released before the measured evaluation returned. |
+| `net-bytes` | Allocated minus freed bytes at the measurement boundary; it may be negative when the expression releases pre-existing values. |
+| `peak-live-bytes` | Maximum positive live-byte delta above the starting boundary. |
+| `result-bytes` | Logical `.mem.objsize` of `result`. |
+| `alloc-count` | Number of allocator blocks obtained. |
+| `free-count` | Number of allocator blocks released. |
+| `workers` | Pool workers, including worker 0 (the main thread), that executed at least one parallel task. Zero means no pool dispatch occurred. |
+| `worker-busy-ns` | Busy nanoseconds summed across participating workers. |
+| `parallelism` | `worker-busy-ns / time-ns`. |
+
+Only one `.mem.ts` scope may be active in a process, so nested calls return a
+`state` error. Allocation instrumentation is normally inactive; ordinary
+queries pay only a predicted-not-taken check at allocator entry/exit. While a
+scope is active, atomic accounting adds intentional overhead. For performance
+benchmarks, time normal executions separately and use `.mem.ts` as an
+additional memory-measurement execution.
+
+The common fields (`time-ns`, `memory-bytes`, `workers`, `worker-busy-ns`, and
+`parallelism`) come from the same per-query measurement lifecycle used by the
+profiler and query log. The allocation counters and `result-bytes` are the
+additional `.mem.ts`-specific detail.

docs/docs/namespaces/sys.md

@@ -16,7 +16,7 @@ Process-level introspection (build, memory, host info) and command-style operati
 | [`.sys.prof`](#sys-prof) | variadic | — | Last profiled query's per-step statistics as a table. |
 | [`.sys.querylog`](#sys-querylog) | variadic | — | Ambient per-query statistics ring as a table. |
 | [`.sys.querylog.enable`](#sys-querylog-enable) | variadic | restricted | Toggle query-statistics logging. |
-| [`.sys.gc`](#sys-gc) | variadic | — | Garbage-collect hint (no-op; returns 0). |
+| [`.sys.gc`](#sys-gc) | variadic | — | Run allocator maintenance and return `0`. |
 | [`.sys.env`](#sys-env) | variadic | — | Count or list of globally bound names. |
 | [`.sys.exec`](#sys-exec) | unary | restricted | Run a shell command; return its exit code. |
 | [`.sys.cmd`](#sys-cmd) | unary | restricted | Dispatch a colon-command string. |
@@ -209,7 +209,11 @@ The `-Q` startup flag is the equivalent for enabling logging from launch.
 
 ## `.sys.gc` { #sys-gc }
 
-Signature: `(.sys.gc)`. No-op stub — Rayforce uses reference counting, so there's no global collector to kick. Kept as a registered builtin so existing scripts that call it don't error. Returns `0`.
+Signature: `(.sys.gc)`. Runs allocator maintenance: drains cross-thread frees,
+flushes slab caches, coalesces free buddy blocks, reclaims empty oversized
+pools, and incrementally returns aged free pages to the operating system.
+Rayforce values remain deterministically reference-counted—this is allocator
+GC, not a tracing object collector. Returns `0`.
 
 ## `.sys.env` { #sys-env }
 

docs/docs/queries/attributes.md

@@ -1,15 +1,15 @@
 # Column Attributes
 
-Semantic properties stamped onto numeric columns — `sorted`, `unique`, `grouped`, `parted`.  Where an [accelerator index](indexes.md) is a *physical* structure (a hash table, a permutation), a column attribute is an *assertion about meaning*: this column is non-descending, these values are distinct.  The `.attr.*` family layers over the same storage as `.idx.*`, but its contract is semantic, not structural.
+Semantic properties stamped onto columns — `sorted`, `unique`, `grouped`, `parted`.  Where an [accelerator index](indexes.md) is a *physical* structure (a hash table, a permutation), a column attribute is an *assertion about meaning*: this column is non-descending, these values are distinct.  The `.attr.*` family layers over the same storage as `.idx.*`, but its contract is semantic, not structural.
 
 !!! note "v1 status"
-    The four attributes, their strict verify-on-set kernels, and the `(.attr.*)` Rayfall surface are shipped, along with the as-of-join fast paths that consume them.  All attributes are **numeric-only** in v1 — the same numeric type set as `.idx.*`; symbol / string columns are deferred.
+    The four attributes, their strict verify-on-set kernels, and the `(.attr.*)` Rayfall surface are shipped. `sorted` and `unique` are numeric; the backing-index attributes `grouped` and `parted` also support symbol columns.
 
 ## Why Attributes Are Separate from Indexes
 
 An [accelerator index](indexes.md) answers *"what structure is attached?"* — a zonemap, a hash table, a sort permutation.  A column attribute answers a different question: *"what is semantically true of these values?"*  The two markers (`sorted`, `unique`) carry no allocation at all — they are cheap flags.  The two backing-index attributes (`grouped`, `parted`) do reuse the index layer underneath (so a `grouped` or `parted` column also shows up via `.idx.has?` and `.idx.info`), but `.attr.*` exists to assert a *property* the engine can trust, not merely to build a structure.
 
-The payoff is that a consumer — most importantly the [as-of join executor](joins.md#pre-sorted-fast-path) — can trust a stamped attribute unconditionally and skip work it would otherwise have to do defensively.
+The payoff is that consumers can trust a stamped attribute unconditionally. Besides the [as-of join executor](joins.md#pre-sorted-fast-path), filters and grouped aggregation use the physical layout directly.
 
 ## The Four Attributes
 
@@ -22,15 +22,17 @@ The payoff is that a consumer — most importantly the [as-of join executor](joi
 
 The two **markers** are independent and may coexist.  A column holds at most **one backing index** at a time: setting `grouped` or `parted` replaces any prior backing index.  Markers may coexist with a backing index.
 
-`parted` is **verify-only** — it never reorders data.  It checks that the column is already block-laid-out (each distinct value occupies one contiguous, ascending run) and errors if it is not.  The caller is responsible for ordering the column first.
+`parted` is **verify-only** — it never reorders data.  It checks that the column is already block-laid-out (each distinct value occupies one contiguous run; numeric keys must also be ascending) and errors if it is not.  The caller is responsible for ordering the column first.
+
+For `where: (in key set)` on a parted key, the filter builds its row selection from the matching `[start, len)` ranges without scanning the key column or expanding full morsels into row ids. If the same query groups by that key and uses supported aggregates, Rayforce skips the filter and group discovery entirely and streams each selected partition directly into the aggregate kernel. The equivalent `grouped` path consumes hash-index row slices.
 
 ## Setting, Reading, and Dropping
 
 ### Signatures
 
 `(.attr.set 'name v)   (.attr.get v)   (.attr.drop v)`
 
-- `(.attr.set 'name v)` — assert attribute `name` on numeric vector `v`.  Strict verify: it scans `v` and **errors on violation**, so a stamped attribute never lies.  Returns the column with the attribute recorded.
+- `(.attr.set 'name v)` — assert attribute `name` on vector `v`. Strict verification **errors on violation**, so a stamped attribute never lies. Returns the column with the attribute recorded.
 - `(.attr.get v)` — return the symbol vector of currently-held attributes (the empty symbol vector when none are set).
 - `(.attr.drop v)` — remove all attributes from `v`.
 
@@ -85,7 +87,7 @@ An unknown attribute name is rejected the same way:
 (.attr.set 'bogus [1 2 3])      ; ⇒ error (domain): unknown attribute
 ```
 
-Setting `sorted` runs an O(n) ascending scan; `unique` and `grouped` build / consult the hash to detect duplicates; `parted` walks the column verifying that each distinct value forms one contiguous ascending block.
+Setting `sorted` runs an O(n) ascending scan; `unique` and `grouped` build / consult the hash to detect duplicates; `parted` walks the column verifying that each distinct value forms one contiguous block (and ascending block order for numeric keys).
 
 ## Combining Attributes
 
@@ -160,7 +162,7 @@ For a partitioned join `(asof-join [Key Time] L R)`, if the single numeric equal
 
 ## Caveats and Limits
 
-- **Numeric types only (v1).**  Attributes accept the same numeric type set as `.idx.*` (boolean, integer, float, and temporal types).  Symbol / string columns are deferred — including the `parted` equality key in a partitioned as-of join, which must be a numeric ID.
+- **Type support.** `sorted` and `unique` accept numeric types. `grouped` and `parted` also accept `SYM`; `STR` remains unsupported.
 - **Strict verify, no silent stamping.**  `.attr.set` errors on violation rather than recording a property it cannot confirm; this is what lets consumers trust the stamp unconditionally.
 - **Conservative propagation.**  Only copy / rebind preserve attributes; every transform drops them.  Re-assert with `.attr.set` after a transform.
 - **One backing index per column.**  `grouped` and `parted` share the [accelerator-index](indexes.md) slot, so a column carries at most one of them at a time; the markers are separate and free.

docs/docs/queries/indexes.md

@@ -16,8 +16,11 @@ The executor consults indexes at five sites.  Each site has a specific consumpti
 | `.idx.zone` | `filter` — comparison predicates (EQ/NE/LT/LE/GT/GE) | O(1) all/none short-circuit: if the constant falls outside the column's [min, max] range the filter returns 0 rows without scanning; if every row must pass the whole table is returned.  Integer and float families both supported. |
 | `.idx.bloom` | `filter` — EQ on integer-family columns | Definite-absent proof: if all k probe bits are clear the key is absent and the filter returns 0 rows without scanning.  False positives fall through to the scan; there are no false absences. |
 | `.idx.hash` | `filter` — EQ on integer-family columns | Direct rowsel from hash-chain probe: matching rows are collected in O(matches) without a full column scan.  Null-free columns only (HAS_NULLS → scan). |
+| `part` backing (`.attr.set 'parted`) | `filter` — EQ predicates | Locate the value block and build the rowsel directly from its contiguous `[start, len)` span, including dense blocks. |
 | `.idx.sort` | `filter` — range predicates (EQ/LT/LE/GT/GE) | Binary search over the row-id permutation to identify the matching span, then a rowsel is built from that span.  NE (two disjoint spans) falls back.  A selectivity guard fires when the span exceeds n/128 (~0.78%) — see [Performance Characteristics](#performance-characteristics) for the measured rationale. |
 | `.idx.hash` | `filter` — IN predicates on integer-family columns | Hash-chain probe for each set element; result is the union rowsel over all matches. |
+| `part` backing (`.attr.set 'parted`) | `filter` — IN predicates | Match partition keys once and build the rowsel from their contiguous physical spans. |
+| `.idx.hash` / `part` backing | `where key IN …` + `group by key` | Resolve only the requested key slices and aggregate them directly, skipping filter scan and group discovery. Part slices stream as contiguous ranges. |
 | `.idx.sort` | ORDER BY — single ascending key | The pre-built permutation is reused directly rather than re-sorting.  Descending ORDER BY falls back to recompute (reversing the perm would swap tie-group positions relative to stable DESC sort). |
 | `.idx.sort` | `distinct` | Walk the sort permutation once to collect first-occurrence row ids per distinct value — O(n) with no hash table.  SYM/GUID/STR columns fall back. |
 | `.idx.hash` | `find` | Hash-chain probe returns the minimum row id matching the needle, or a provably-absent signal.  Integer-family needles only; float and cross-family needles fall through to the scan. |
@@ -71,8 +74,10 @@ Set `RAY_IDX_STATS=1` before running a process.  At exit, per-site consult and h
 idx_route filter_zone   consults=3 hits=2
 idx_route filter_bloom  consults=1 hits=1
 idx_route filter_hash   consults=5 hits=5
+idx_route filter_part   consults=3 hits=3
 idx_route filter_range  consults=4 hits=3
 idx_route in            consults=2 hits=2
+idx_route group-slice   consults=2 hits=2
 idx_route find          consults=8 hits=7
 idx_route sort          consults=6 hits=6
 idx_route distinct      consults=2 hits=2

docs/docs/queries/pivot.md

@@ -147,6 +147,21 @@ Running computations like cumulative sums can be computed using the DAG-backed t
 
 Use `scan` when the accumulator is a custom function. Use `sums`, `avgs`, `mins`, `maxs`, and `prds` for built-in running aggregates, and `msum`, `mavg`, `mmin`, `mmax`, `mcount`, `mvar`, and `mdev` for fixed trailing windows. Constant-window calls are lazy-aware DAG operations and materialize with morsel-based kernels.
 
+When the calculation must restart for each key, use the partitioned `window`
+form. A positive `frame:` value is the trailing row count, including the
+current row:
+
+```lisp
+(window {from: trades
+         part: [Sym]
+         order: [Time]
+         frame: 3
+         funcs: {avg3: (avg Price)}})
+```
+
+Use `frame: 'running` for an unbounded cumulative frame and omit `frame:` (or
+use `frame: 'whole`) to aggregate over the whole partition.
+
 ### Rank and Order
 
 ```lisp