interactive: DDIR server#763
Merged
Merged
Conversation
Introduce `interactive::server::Server`, a per-worker registry that hosts
interpreted DDIR programs in a long-running timely computation and lets
them share results by name. This is the interpreter-driven successor to
the legacy `dd_server` crate (which hot-loaded compiled `.so`s): "install"
means render a `scope_ir::Program`, not `dlopen`.
Resolves the two named-trace binding points the batch backends only
`panic!` on:
- `Source::Trace(name)` -> `import_core` the registered trace into the
new dataflow (keeping its `ShutdownButton`);
- `Export(name, _)` -> arrange and register the trace for later imports.
Lifecycle:
- install: build a dataflow over imported traces + positional inputs,
publishing its exports; unique-name and known-import validation up front.
- tick: advance all inputs to the next epoch, run to quiescence, then
compact every trace to the epoch (importers hold the shared TraceBox
back to what they still need, so this sheds only unneeded history).
- drop_program: evict a program, gated on its published traces having no
live importers (an importer refcount). Presses the import shutdown
buttons and closes inputs so timely reclaims the dataflow.
Shared traces live at the host time `u64`; each program body still renders
in its own `iterative::<PointStamp>` scope, as in `vec::evaluate`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A multi-worker command-driven driver over `interactive::server::Server`. The main thread streams command lines to worker 0, which injects them into a timely `Sequencer`; the resulting total order is replayed on every worker, so install/tick/drop stay collective while `feed` is applied on worker 0 only (the arrangement's exchange pact routes data to key owners). Commands: install / feed / tick / drop / list / help / exit. Runs a script file (first positional arg) or an interactive stdin REPL; `-w N` selects worker count. Demo programs: - producer.ddp publishes input 0 as the trace "edges"; - reach_import.ddp imports "edges" + roots, maintains reachability; - server_demo.txt cross-dataflow shared-trace + incremental update demo; - drop_demo.txt exercises the drop gate and teardown. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Derive serde Serialize/Deserialize on the scope-tree IR (`scope_ir::*`),
the scalar AST (`parse::{Term, UnOp, BinOp, Projection, Reducer}`), and
`ir::LinearOp`, so a lowered `scope_ir::Program` (and a `Value`) can ride a
timely `Sequencer` to every worker. `timely`'s exchange `Data` is a blanket
impl over `Send + 'static + Serialize + Deserialize`, so no `Ord`/`Debug` is
required.
Also add `parse::pipe::parse_term`, a public entry to parse a single scalar
`Term` from a string; a closed term evaluates to a constant `Value`, which
the server uses to accept structured (ADT) values on input.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…feeds Robustness: parse/lower/validate on the intake thread and ship a typed, serializable `Command` (carrying a `scope_ir::Program`) through the `Sequencer`. Workers only execute well-typed commands, so a malformed program or value can no longer panic a worker — it's caught (the parser still `panic!`s, so the intake wraps it in `catch_unwind`) and reported at the prompt, which keeps running. A main-thread panic hook keeps the caught parser panics quiet while leaving worker panics visible. Teardown: `drop_program` now calls `worker.drop_dataflow(id)` (id captured via `next_dataflow_index`) to remove the operators and free their state immediately, instead of the best-effort step nudge. The importer-refcount gate is what makes the unilateral removal safe. Per-program probes: each program carries its own `ProbeHandle` and `tick` waits only on live programs. A single shared probe stranded a dropped program's handle at its last frontier and wedged `tick` forever (surfaced by the -w4 drop demo). Feeds: `feed` takes a value (a comma-separated int row, or any closed `Term` literal for ADT data like ASTs), an optional `time=` (default the current epoch; `update_at` lets inputs be scheduled into the future), and an optional `diff=`. Adds examples/programs/echo.ddp and value_demo.txt (structured values, time-scheduled input, and bad-input robustness). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`peek <trace> [key]` reads the current contents of a registered trace and prints them (worker 0). It builds a transient dataflow that imports the trace, optionally filters to one key, exchanges every row to worker 0, and accumulates net multiplicities as of the current epoch — so the result is the complete, consolidated snapshot even when the trace is sharded across workers, not each worker's slice. The dataflow is dropped (`drop_dataflow`) as soon as it has drained, and uses its own probe so it never interferes with installed programs. An unknown trace is a clean error, not a crash. This is the read-back-out counterpart to `inspect`: rather than watching changes stream past, ask for a relation's contents on demand. Adds examples/programs/peek_demo.txt. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The server's helper programs and command scripts were sitting in
examples/programs/ next to ordinary DDIR source, conflating two different
artifacts: a .ddp is a dataflow you `install`, while a session .txt is a
stream of server commands you run the server *on*. Separate them:
examples/server/
README.md -- what the server is, the two file kinds, the
commands, and what each demo shows
programs/ -- producer.ddp, reach_import.ddp, echo.ddp
(server-oriented: they use import/export, so unlike
../programs/ they are not ddir_vec-runnable)
sessions/ -- shared_trace.txt, drop.txt, values.txt, peek.txt
(renamed from *_demo.txt; install paths updated)
No code change beyond a usage pointer in the example's header.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
first version of a server for ddir, accepting commands for a dataflow lifecycle.