Add UK bus-spending imputation plan and DfT calibration targets#153
Add UK bus-spending imputation plan and DfT calibration targets#153vahid-ahmadi wants to merge 2 commits into
Conversation
The published Populace UK population miscalibrates the two bus consumption variables: weighted bus_fare_spending lands ~2x the DfT fare total (and is concentrated in too few households at implausibly high amounts), while bus_subsidy_spending lands well below the DfT net-support total. The incumbent enhanced-FRS build anchors both to DfT Annual Bus Statistics; that anchoring was never ported to the Populace UK build. This adds the missing UK build artifacts, mirroring build/us conventions: - build/uk/bus_calibration_targets.py: a TargetRegistry with DfT-anchored household-sum targets for bus_fare_spending (~GBP 4.0bn) and bus_subsidy_spending (~GBP 3.5bn) — England DfT BUS05a/BUS05b totals uplifted to UK by the ONS mid-2023 population ratio. These feed the calibration solver and aggregate_admin_gate. - build/uk/bus_source_stages.json: a source manifest declaring the two imputation stages (LCFS fares, ETB subsidy) as weighted-QRF draws over household predictors, each followed by a support_clip to the donor's realized range so the imputation does not over-concentrate spending. - build/uk/bus_imputation.py: uk_bus_plan(), the donor graph and stage names, mirroring us_plan (transforms injected by the caller; no stubs/fallbacks). - tests/test_uk_bus.py: plan assembly + donor citations, manifest contract, and target value/provenance checks. Executable stage transforms are injected by the build caller; survey microdata is supplied as plain tables at call time. No incumbent data-package reference. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds calibrate_bus_spending_levels(): the executable level-calibration the declarative targets and stages describe. It mirrors the incumbent enhanced-FRS step exactly (scale = target / actual; column *= scale), scaling each bus variable's weighted total to its DfT target. Value-scaling only — the set of spending households and the distribution shape are unchanged, as in the incumbent build (the spender share is set by the imputation, not calibration). Verified on the published populace_uk_2023 household table: bus_fare_spending 7.360bn -> 4.000bn (scale 0.543) bus_subsidy_spending 0.968bn -> 3.500bn (scale 3.615) both landing exactly on their DfT targets. Tests cover exact-target scaling, spender-set preservation, registry-default targets, and clear errors on a missing column or a zero aggregate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Holding off on merging this as-is. The imputation plan shape is fine, but the calibration targets need to be Ledger-backed before they land in Populace: |
) (#322) populace-fit capped scikit-learn below 1.9 because quantile-forest imported sklearn.tree._tree.DTYPE, which sklearn 1.9 removed (and sklearn 1.9 also changed the sample-weight signature of the forest bootstrap helpers). The cap made the package uninstallable alongside any sklearn>=1.9 consumer: installing populace-fit into such an environment downgraded scikit-learn 1.9 -> 1.8, breaking float-pinned reproduction tests built under 1.9 (hit in populace-dynamics). quantile-forest 1.4.2 (2026-06-21) tracks both sklearn-1.9 changes (zillow/quantile-forest#152 restores DTYPE via an np.float32 fallback, #153 fixes the sample-weight bootstrap). Requiring quantile-forest>=1.4.2 lets the sklearn<1.9 cap be dropped (>=1.5), so populace-fit installs and passes its suite under both sklearn 1.8 and 1.9 and can coexist in one environment with an sklearn-1.9 consumer. populace-fit is the only package that imports quantile_forest (verified across the repo). Its QRF forests do their own weighted bootstrap and call .fit() without sample_weight, and the sign gate is a native HistGradientBoostingClassifier, so nothing in this package touches the sklearn-1.9-changed forest bootstrap path directly. Test evidence (populace-fit suite, editable install of populace-frame + populace-fit, quantile-forest 1.4.2 in both): - scikit-learn 1.8.0: 76 passed - scikit-learn 1.9.0: 76 passed Also documents the coexistence guarantee in populace-fit's README. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
MaxGhenis
left a comment
There was a problem hiding this comment.
The diagnosis and the plan shape are right: published populace UK bus fares are ~2x the DfT total and subsidy is well under, and anchoring to DfT BUS05 with a QRF-plus-support-clip imputation is the correct fix. The DfT figures check out — England fare receipts £3.4bn and net government support £3.0bn for the year ending March 2025 (note DfT revised these tables on 24 March 2026, and the precise tables are BUS05aii and BUS05i/ii).
But this has to be rebuilt against the architecture that landed after it opened, and it conflicts with main today (agent-assisted review):
-
Spec-only packages.
build/ukis now resource-only —build/uk/__init__.pymoved tobuild/uk_runtime, so editing it is a modify/delete conflict, andtest_spec_only_country_packages.pyfails on any.pyunderbuild/uk.bus_imputation.pyandbus_calibration.pyare executable and belong inbuild/uk_runtime;bus_source_stages.jsoncan stay as a spec. -
TargetSpecno longer takesaggregation.bus_calibration_targets.pypassesaggregation="sum"andtest_uk_bus.pyasserts it — both break on rebase. The green CI here is stale (it ran when the branch was 107 commits behind main). -
Targets must be Ledger-backed, per my earlier comment and the
ledger_targets/LedgerTargetReferencesurface that has since landed. Don't embed the £3.4bn fare receipts, the £3.0bn net support, or the 68.3/57.7 uplift as Python literals — the DfT facts and the England-to-UK derivation belong in Ledger with lineage, referenced from here.
Two substantive points for the rebuild:
- Vintage: the population is 2023 but the targets are period-2025 DfT. Calibrate 2023 to the year-ending-March-2023 figure, or say why 2025 is the right anchor.
- The England-to-UK population uplift is crude — England receipts already include London, the dominant bus market, so scaling by population likely overstates the UK total, and Scotland/Wales fund their own subsidy. A per-nation or GB anchor would be better; at minimum caveat it.
Also, the executable calibrator grew this past its "declarative plan + targets" description — I'd split it out or move it to runtime as part of the rework. Direction is good; this is a rework, not a tweak.
Problem
The published Populace UK population miscalibrates both bus consumption variables (verified against the incumbent enhanced-FRS build and DfT Annual Bus Statistics, at the same simulated year):
bus_fare_spendingbus_subsidy_spendingFare is ~2× too high and over-concentrated; subsidy is ~3.6× too low — exactly the failure mode that occurs when the survey imputation isn't anchored to DfT. The incumbent build applies a DfT calibration (
calibrate_bus_fare_spending/calibrate_bus_subsidy_spending); that step was never ported to the Populace UK build.What this adds (mirrors
build/us)build/uk/bus_calibration_targets.py— aTargetRegistrywith DfT-anchored household-sum targets forbus_fare_spending(~£4.0bn) andbus_subsidy_spending(~£3.5bn): England DfT BUS05a/BUS05b totals uplifted to UK by the ONS mid-2023 population ratio (1.18). Feeds the calibration solver andaggregate_admin_gate.build/uk/bus_source_stages.json— a source manifest declaring the two imputation stages (LCFS fares, ETB subsidy) as weighted-QRF draws, each followed by asupport_clipto the donor's realized range (kills the over-concentrated high-value tail).build/uk/bus_imputation.py—uk_bus_plan(), donor graph, stage names; mirrorsus_plan(transforms injected by the caller — no stubs/fallbacks).tests/test_uk_bus.py— plan assembly, donor citations, manifest contract, target value/provenance.Notes
test_no_incumbent_data_package_references_in_live_tree); survey microdata is supplied as plain tables at call time.populace_uk_2023.rail_subsidy_spendingand the road-fuel litre proxies share the same missing-anchor class of bug.Full
populace-buildtest suite passes locally (uv run pytest), ruff clean.🤖 Generated with Claude Code