Skip to content

Schema v2 migration + four analysis levels (epic) #69

Description

@rahlk

Epic: migrate codeanalyzer-python to canonical schema v2 + four analysis levels

Migrate the analyzer from the v1 payload (PyApplication + a separate program_graphs section) to the CLDK canonical schema v2 — one additive Code Property Graph tree — exposed as four gated analysis levels (-a 1|2|3|4), in both the analysis.json and Neo4j projections. This is a breaking major release (the codeanalyzer-backend "existing analyzer → schema v2" path). It supersedes the L3 approach in #67: dataflow moves inline into the tree, and the interprocedural SDG moves from -a 3 to -a 4.

Target shape

application → symbol_table{module} → types/functions → callables{ body{}, cfg[], cdg[], ddg[], summary[] }
call_graph[], param_in[], param_out[]   — cross-function edges, at application scope
  • can://python/<app>/<file>/<type>/<callable-sig> durable ids (≥ callable); <callable-id>@<line>:<col> / @tag ordinal ids (< callable).
  • source stored once per module + byte-offset spans (every node's text slices off it); the per-callable code field is dropped.
  • Payload envelope: schema_version: "2.0.0", language, max_level, k_limit.

Levels

Level Grows Adds edges
1 tree to callable + call nodes in body (callee:null)
2 callee: null→id backfill call_graph
3 rest of body + @entry/@exit cfg, cdg, ddg (syntactic, prov:["ssa"])
4 synthetic param vertices param_in, param_out, summary, + ddg (semantic, prov:["points-to"])

--graphs sdg requires -a 4 (requesting it at -a 3 is a flag error).

Key decisions

  • Reuse the existing dataflow kernels (cfg/dominance/defuse/sdg/summaries/alias); a new identity-mapping layer translates internal (signature, node_id) keys → can:// ids at emit time, which keeps the JSON and Neo4j projections in lockstep.
  • Scalpel (python-scalpel) is the primary L4 points-to oracle behind the frozen may_alias interface, with TypeBasedAliasOracle as an automatic fallback.
  • Analyzer-only scope. The python-sdk Pydantic revision is a separate follow-up (via cldk-sdk-frontend). Taint stays deferred to the SDK; the backward slicer stays internal as the L3/L4 validation gate (no product "queryable slicer" framing, no taint_flows section).
  • Neo4j is a re-identify + one prov property + source/span parity + MAJOR bump (schema.neo4j.json2.0.0); the near-identity projection structure is unchanged. Always full-depth.

Invariants / gates

  • Additive superset: analysis.json(-a 1) ⊆ … ⊆ analysis.json(-a 4), modulo the callee: null→id refinement and the L3→L4 DDG ssapoints-to widening.
  • Two-projection agreement: JSON node/edge ids == Neo4j merge-key values.
  • Per-level gates (validated per stage): L1 canonical-schema conformance; L2 no dangling call-graph endpoints; L3 intraprocedural backward slice == hand-computed set; L4 no dangling SDG endpoints + PARAM arity + a summary for a known transitive flow.

Stages

Each stage is an independently shippable PR-unit with its own gate. Child issues:

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions