Skip to content

NixOS support - #379

Open
mrosseel wants to merge 409 commits into
brickbots:mainfrom
mrosseel:nixos
Open

NixOS support#379
mrosseel wants to merge 409 commits into
brickbots:mainfrom
mrosseel:nixos

Conversation

@mrosseel

Copy link
Copy Markdown
Collaborator

Summary

  • Full NixOS-based system for PiFinder (replaces Raspbian)
  • Declarative system configuration via Nix flake
  • SD card image, netboot, and migration bootstrap tarball builds
  • Software update via nixos-rebuild with GitHub release/PR channels

Test plan

  • Flash SD image and verify boot
  • Test WiFi AP and client mode switching
  • Test software update UI channels
  • Test hostname rename via web UI

🤖 Generated with Claude Code

@mrosseel mrosseel added the testable Ready for testing via PiFinder software update label Feb 12, 2026
@mrosseel mrosseel added testable Ready for testing via PiFinder software update and removed testable Ready for testing via PiFinder software update labels Feb 14, 2026
@mrosseel
mrosseel force-pushed the nixos branch 4 times, most recently from 59c9e31 to e2c57cd Compare May 27, 2026 16:03
mrosseel and others added 3 commits May 28, 2026 09:15
The Nix derivation was overwriting pifinder-build.json with
"nix-${gitRev}" at build time, so even released devices reported a
random short-sha instead of the release version. Three writers became
two, with consistent semantics everywhere:

- pifinder-src.nix: drop the cat > pifinder-build.json block and the
  gitRev arg — the derivation now copies the source file through
  verbatim, no version invention.
- flake.nix: drop the pifinderGitRev _module.args plumbing.
- services.nix: drop pifinderGitRev / gitRev from the pifinder-src
  import.
- release.yml: reorder so the version stamp is written into the
  working tree BEFORE the nix build (so the store path bakes in the
  release version, not the previous stamp), then re-stamp with the
  resulting store_path after the build, commit, push, tag.

Result: SD image, cachix closure, and committed JSON all agree on
the released version. Matches the flow already documented in
nixos/RELEASE.md.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Captures the decision to self-host Attic at cache.pifinder.eu, backed by
SQLite + local disk initially with Cloudflare R2 as the eventual chunk
store. Covers considered alternatives (cachix.org, Magic Nix Cache,
nix-casync, harmonia) and the operational consequences for CI publishing,
on-device updates, and failure fall-through to cache.nixos.org.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces pifinder.cachix.org with the self-hosted Attic instance at
https://cache.pifinder.eu/pifinder (ADR 0004) on both the device-side
substituter list and the CI publish path. cachix.org is removed
entirely with no fallback, so this is the first build that proves
Attic stands on its own.

services.nix:
  substituters       = ["https://cache.pifinder.eu/pifinder"
                        "https://cache.nixos.org"]
  trusted-public-keys = ["pifinder:8UU/O3oLkaJHHUyqEcPGl+9F1m4MqDca39Ewl49jBmE="
                         "cache.nixos.org-1:..."]
  (pifinder.cachix.org and its key removed.)

build.yml — build-native, build-emulated, build-migration-tarball:
  - remove cachix/cachix-action steps
  - remove `cachix push` (replaced by `attic push pifinder:pifinder`)
  - add a "Setup Attic substituter" step that runs
      nix profile install nixpkgs#attic-client
      attic login pifinder https://cache.pifinder.eu \"\$ATTIC_TOKEN\"
      attic use pifinder:pifinder
    before the build, so the build itself substitutes from
    cache.pifinder.eu.
  build-emulated swaps cachix/install-nix-action for
  DeterminateSystems/nix-installer-action — no cachix dependencies left.

First post-cachix build is expected to be slow: pifinder.cachix.org's
warmed-up paths are gone. Once it completes and `attic push` lands,
subsequent builds substitute from cache.pifinder.eu.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
github-actions Bot and others added 14 commits May 28, 2026 12:28
…cceed)

Brickbots/PiFinder runs the workflow on pull_request:synchronize from
PR brickbots#379 (mrosseel:nixos -> brickbots:main). With ATTIC_TOKEN now also
set on brickbots, build-emulated's 'Push to Attic' step succeeds — the
last failing step. But stamp-build then tries to checkout the PR head
ref (mrosseel:nixos) and 'git push' a pifinder-build.json commit there,
which can't work from brickbots' Actions runner (no write access to the
fork). The PR run therefore failed at the stamp step even after attic
was wired correctly.

Gating stamp-build on github.event_name == 'push' keeps the canonical
stamp on the mrosseel:nixos push run (where it works) and skips it on
brickbots PR runs (which only need to verify the build).

Net effect: both repos' CI runs in parallel without stomping —
- Both build and push the same closure to cache.pifinder.eu (attic
  FastCDC-dedups, so the second push is a no-op),
- Only mrosseel stamps pifinder-build.json,
- build-migration-tarball already gates on github.ref == refs/heads/nixos
  so it only runs on mrosseel push.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sync the fork with upstream up to 0e0ec03 (39 commits since the last
sync at e4b623a): typed positioning model (PointingEstimate / ImuSample
/ AlignedResult), polar alignment, comet vectorization, single-instance
lock, resolution-flexible UI, i18n fonts.

Conflict resolution (18 files):
- Adopt upstream's typed positioning model across solver, integrator,
  state, imu_pi, status, main, base, console, object_details, fonts,
  menu_structure, auto_exposure, camera_interface.
- Keep NixOS layers: software.py (store-path upgrade UI), utils.py
  (build_json, writable comet_file, robust pifinder_dir), sys_utils.py.
- plot.py: restore top-level `import pandas` (upstream added module-level
  uses; the fork had made it lazy) and drop the redundant lazy imports.
- solver.py taken verbatim from upstream; the cedar-detect dev-spawn is
  proposed upstream separately (brickbots#478).
- Drop fork-deleted nox/pip tooling (requirements.txt, version.txt).

deps: add xlrd to python-packages.nix (pyerfa already present).

Verification: ruff clean; 479 unit tests pass. Remaining failures are not
from this merge -- 6 test_software + 4 test_t9_search pre-exist on
origin/nixos; 5 test_comets fail on skyfield 1.53 (upstream vectorization,
has a runtime fallback).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Skyfield's propagate() lays a batched Kepler orbit out as
(3, #orbits, #times) but sets output_shape = (3,) + t1.shape, so a
batched orbit propagated to a single scalar time raises "cannot reshape
array of size 3N into shape (3,)" on skyfield >= 1.46. (The fork uses
nixpkgs' skyfield 1.53; upstream pins 1.45, which tolerated it.)

Give every comet the same target time as an (N, 1) column so output_shape
matches the (3, N, 1) result, then squeeze the time axis. Verified against
the per-comet path (0 AU difference) and the existing tests/test_comets.py
oracle (7/7 pass).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…che)

The upstream merge auto-merged catalogs.py entirely to the fork's side,
silently dropping upstream's T9 search (brickbots#464: KEYPAD_DIGIT_TO_CHARS,
search_by_t9) and the catalog disk cache; test_t9_search (an upstream
test) failed as a result. Take upstream's catalogs.py wholesale, matching
the upstream main.py and menu_structure the merge already adopted. Trade-
off: drops the fork's priority-fast-path / background-loader in favour of
upstream's cache + loader (the agreed "take upstream catalogs" choice).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
MIN_NIXOS_VERSION was intentionally bumped 2.5.0 -> 3.0.0 (d705057, prep
for 3.0), but test_software still asserted 2.5.x/2.6.x as qualifying.
Shift the qualifying mock releases and _meets_min_version cases into the
3.x line so the suite matches the current minimum; below-min (2.4.0) and
draft cases are unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
main.py created (and server.py switched) the pifinder_logconf.json symlink
relative to the cwd, which on NixOS is the read-only /nix/store -> the
service crash-looped at startup (OSError: Read-only file system).

Follow the same split config.json already uses: the logconf presets stay
read-only in the source tree (utils.pifinder_dir/python/logconf_*.json) and
the active selection is persisted as a bare filename in the writable data
dir (PiFinder_data/log_config), resolved via utils.active_logconf_path().
Storing the name (not a store-path symlink) keeps the choice valid across
upgrades (which GC old store paths) and reboots.

No NixOS workaround needed; also removes the write-to-source-tree
antipattern on Raspberry Pi OS (upstreamable).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The upstream merge kept nixos's outer tetra3_dir (python/PiFinder/tetra3)
while taking upstream's solver.py, which built the DB path as
tetra3_dir/data/default_database.npz. But the submodule nests the package
at tetra3/tetra3, so the DB is actually at tetra3/tetra3/data -> the solver
crashed at startup with FileNotFoundError.

Load it by its canonical name, Tetra3("default_database"); tetra3 resolves
the bundled DB from its own package data dir regardless of the inner/outer
layout. Validated on-device (loads from python/tetra3/data).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
release.yml now installs Nix + Attic like build.yml and pushes the release closure to a dedicated, never-GC'd `pifinder-release` cache. Devices and the migration first-boot trust it ahead of the dev `pifinder` cache and cache.nixos.org. Removes the last Cachix usage from active config.

Docs (RELEASE.md, ADR 0004, NIXOS_STATUS.md) document the dev-vs-retained-release two-cache split and the per-cache retention caveat. The pifinder-release trusted-public-key is a placeholder until the cache is bootstrapped server-side.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The 6 .direnv/ nix-direnv cache files were tracked but regenerate on
every direnv reload, so rebases baked divergent copies into the nixos
stack. Gitignore + untrack stops that churn going forward.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…erivations

Replace the 524-line nixos/pkgs/python-packages.nix (26 manually packaged PyPI deps, each with hand-chased hashes and build patches) with a uv-managed workspace realized into the Nix store via uv2nix.

Changes:
* Deps declared in python/pyproject.toml, pinned in python/uv.lock (117 pkgs)
* nixos/pkgs/uv-python.nix builds the runtime/dev virtualenvs; the 5 native packages (python-libinput, python-prctl, python-pam, dbus-python, pygobject) keep their build patches as uv2nix overrides
* flake.nix: add pyproject-nix/uv2nix/pyproject-build-systems inputs, thread via specialArgs, devShell uses the uv2nix devEnv
* libcamera Python bindings stay a Nix overlay (not on PyPI)

All four nixosConfigurations + the devShell evaluate; the aarch64 build is to be validated by CI.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…cache

DeterminateSystems/magic-nix-cache-action now returns HTTP 418 and the
GitHub Actions cache rate-limits it (Twirp ResourceExhausted, "rate limit
exceeded"), so `nix develop` cannot fetch the dev environment and every
lint/test/type-check job fails before ruff/pytest/mypy even run — which all
testable PRs inherit.

Mirror build.yml/release.yml and substitute from the self-hosted Attic cache
cache.pifinder.eu (ADR 0004) instead, falling back to cache.nixos.org when
ATTIC_TOKEN is unavailable (e.g. fork PRs) so the job never hard-fails.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…s_utils failures

Changes:
* networking.nix: pifinder-wifi-fallback service+timer brings up PiFinder-AP when no client connects within 45s (or when AP is forced via the UI), so a device with an unreachable saved network stays reachable and AP survives reboot.
* sys_utils.py: persist chosen WiFi mode to PiFinder_data/wifi_mode for the fallback service to restore.
* software.py: show a clear 'No internet - check WiFi' screen instead of an empty channel list when GitHub is unreachable.
* utils.py: log why get_sys_utils() falls back to the no-op fake (previously swallowed) so a failed NM/dbus/pam import is diagnosable.
* remove dead Raspberry Pi OS scripts switch-ap.sh/switch-cli.sh (dhcpcd/hostapd; inert on NixOS).

Needs on-device validation of the AP fallback timing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mrosseel and others added 30 commits July 17, 2026 23:44
Each sweep image pair now gets img_NNN_XXms_metadata.json carrying the
driver-applied exposure/gain, sensor die temperature, sensor black levels,
Lux, colour gains, and raw-frame ADU statistics (mean/median/std/percentiles,
saturated fraction). frame_metadata.json aggregates the same records.

Also gate the sweep UI COMPLETE transition on sweep_metadata.json existing:
the camera process writes it a few seconds after the last image, and
enriching before that silently skipped the SQM/solve/noise-floor details
on every sweep.
Each tracker (BlackLevelTracker, CloudEstimator, WingEstimator,
RadiometerAccumulator) gains dump(): config, derived stats, and the full
window sample contents. The solver publishes these into sqm_details as
window_black_level / window_clouds / window_wings / window_radiometer on
every update, and each sweep image JSON snapshots the whole sqm_details,
so a sweep records the pipeline state frame by frame.
…ot the old base

activate_system compared the persisted camera-type against --default-camera,
i.e. the RUNNING generation base camera. On a device whose build had that
camera as its base (mr: imx477-base build), upgrading onto an imx462-base
build concluded camera is the base, nothing to do and rebooted into the
wrong DTB, killing the camera. Ask the new store path for a matching
specialisation instead, and run the NEW build set-extlinux-default so the
camera-to-entry mapping matches the entries just written.
…erization

Plan B for the stuck-pedestal incident (2026-07-17):
- BlackLevelTracker: an accepted pedestal is now a lease (max_age_seconds,
  default 900 s) instead of a latch — without a fresh accepting refit it
  expires and callers fall back to the profile constant, the same state
  every session boots in. Regression test replays the incident.
- max_intercept_stderr tightened 1.0 -> 0.6 ADU: honest ramp fits on the
  27-sweep archive run 0.10-0.69; the cloud-poisoned fit passed at 0.97.

Calibration wizard (dark-frame path the imx296 tester will run):
- persists a per-frame report JSON (median/MAD/percentiles, requested AND
  driver-reported exposure, sensor temp) beside the fitted scalars — the
  ramp shape is where short-exposure pedestal elevation shows, and three
  fitted numbers cannot preserve it
- dark-current fit uses driver-reported exposures (the imx477 delivers
  intermittent half-exposure frames; a requested-exposure fit is silently
  corrupted)
- dark ladder extended from one decade to near sensor minimum, covering
  the elevation zone (imx462: +2.9 ADU at 25 ms, 0 at ~300 ms)

Sweep instrumentation:
- per-image records now carry the full driver metadata dict (new sensors
  report different keys) and the live camera-side radiometer sample
  (the sqm_details snapshot freezes while the solver starves mid-sweep)
- camera_pi keeps last_frame_metadata; both image_metadata sites publish
  actual_exposure_us
…mat stragglers

mypy: the B-V cache globals are Optional, so assignments cannot narrow
them across statements; parse into locals and use those for savez/len.
format: 4 files the CI format session was silently reformatting on every
run (timeentry, three tests).
Telemetry captured only IMU and solve streams — blind to the radiometric
side. A third event type records the camera-side radiometer sample at
~1 Hz: t (capture epoch, s), exp (driver-reported exposure, s), bg
(background median, ADU), mad (median absolute deviation, ADU), grad
(quadrant gradient, ADU), seq (frame sequence). Dedupes on sequence;
replay players skip unknown event types, so old builds read new files.

A full-night radio stream answers what sweep snapshots cannot: how
auto-exposure actually behaves over a session (lever-arm availability
for the black-level fit), a continuous sky-steadiness timeline, and
real-night input for replaying pedestal estimators.
The marking menu down slot is the one readable at the eyepiece without
moving your head — it goes to SWEEP, the everyday reference-meter action.

SQM Correct is deleted rather than kept: a magnitude-additive session
knob silently absorbs ADU-space, brightness-dependent errors (tonight:
a stuck pedestal), masking exactly the fault class it appears to fix,
and staling whenever the layers beneath it change. Its legitimate
output — the unit-vs-reference-meter difference — is already recorded
as evidence by every sweep run with a reference reading. The feature
never shipped upstream (fork-only), so now is the cheapest moment to
remove it.

Removed: UISQMCorrect screen, menu entry and corrected indicator,
_apply_sqm_correct and both solver call sites, the sqm_correct_delta
shared state, and all test references. CONTEXT.md records the decision.
python/result and python/result-lib are nix build output symlinks that were
committed before .gitignore's result/result-* rule existed, so they stayed
tracked and kept getting re-snapshotted into unrelated commits. Untrack them;
the existing ignore rule keeps them out from now on.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PXDxJkU5CzAegv3FcMEwEq
The NixOS migration UI flow (UIMigrationConfirm/UIMigrationProgress) was removed
from ui/software.py in 'refactor(software): delete the NixOS migration UI flow',
but a later sqm-fix merge reintroduced the test's import and spec builders for
those deleted classes, breaking test collection. Drop the stale references.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PXDxJkU5CzAegv3FcMEwEq
test_radec_entry exercised an older dependency-injection API (BlinkingCursor
time_provider; CoordinateConverter/CoordinateEntryLogic calc_utils_provider;
LayoutConfig no-arg) that radec_entry dropped for module-level calc_utils/time.
Patch calc_utils/time at the module level and pass a display stub to
LayoutConfig so the suite matches the implementation. 6 stale tests were failing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PXDxJkU5CzAegv3FcMEwEq
Surface the ten shielded optical-black rows the IMX290/462 sensor already
transmits and use their trimmed mean as the complete per-frame pedestal for
radiometric SQM. This makes IMX462 SQM zero-touch: no lens cap, dark frames,
or calibration wizard from the user.

Kernel: a second sensor source pad (MEDIA_BUS_FMT_SENSOR_DATA) routes the OB
rows to the receiver as a separate metadata stream without altering the
1920x1080 image matrix.

libcamera: the imx290 cam-helper unpacks the shielded pixels, trims the outer
5% each side and averages the central 90% (preserving sub-ADU resolution on
the 16-bit SensorBlackLevels scale), publishing {l,l,l,l+1}. The +1 marker
lets Python tell a measured value from a static tuning tuple.

Python: camera_pi reads SensorBlackLevels for imx290/imx462 and passes the
native-ADU value as the radiometer sample's optical_black_pedestal. A valid OB
value is the complete pedestal; calibrated then profile pedestals remain the
fallback when OB is missing or unmarked.

Validated on mr2 (IMX462, gain 30) by a same-frame cupboard test: active-green
vs OB dark-accumulation slopes agree to well under 1 ADU/s, so OB is a valid
complete pedestal. Kernel is built via nixos-hardware; pifinder-fast/
pifinder-kernel-cross give a fast x86_64 cross build to seed the binary cache
so CI substitutes the kernel instead of compiling it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015EWb2JET2JE1t4ya3HZPNL
Add the OB pedestal to the SQM architecture note and glossary, and record the
decision in ADR-0023. The justification is the same-frame cupboard test on
mr2's IMX462 (active-green vs OB slopes agree to <1 ADU/s), scoped to IMX462;
IMX296/IMX477 OB remain open. Defines 'optical black' as a glossary term and
sets the pedestal precedence (valid OB > calibrated > profile bias).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015EWb2JET2JE1t4ya3HZPNL
python/.venv, mypy/pytest/ruff caches and __pycache__ directories are not
needed at runtime and inflated the source output to ~935 MiB, making Attic
cache pushes absurdly slow. Removing them drops it to ~19 MiB (cache push
~10-15 s, device upgrade fetch ~30-43 MiB).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015EWb2JET2JE1t4ya3HZPNL
…stal

Gate BlackLevelTracker creation behind SQM_BLACK_LEVEL_TRACKER_ENABLED
(default False) so the radiometer pedestal is OB when present, else the static
profile bias_offset — no tracker in between. Lets us observe the optical-black
pedestal's effect in isolation during rollout/validation. Reversible: flip the
flag (or revert) to restore the tracker fallback.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015EWb2JET2JE1t4ya3HZPNL
All aperture/annulus/wing radii are defined for the ~1.0-scale Bayer-green
photometry images (imx462 green: 490px vs the 512 solve image). The mono
imx296 does photometry on its full-res 1088px frame (scale 2.125), where the
fixed radii sit inside the star profile: the r=5 aperture holds only 83-94%
of a star's flux (measured growth curves, 2026-07-18 sweeps), every local
background annulus lands on the PSF itself, and the wing estimator's sky
ring eats the wings so it measures f~1 and stays permanently inert.

Convert all radii from solve-image pixels to the photometry image's pitch
(_scaled_photometry_radii) and give WingEstimator a set_scale() that
rescales its patch geometry and clears the rolling window on change. Both
the production path (solver.update_sqm) and the calibration wizard's SQM
preview use the shared helper.

Effect by sensor:
- imx462/imx290: radii round to (5, 10, 17) - essentially unchanged.
- imx296: radii become (11, 21, 38); on the 2026-07-18 reference sweeps
  this moves the median error vs a hand-held meter from -0.20 to +0.00
  and lets the estimator track real focus drift.
- hq: also gets larger radii; its sqm_band_offset (like imx296's) should
  be refit against reference sweeps after this change.

Validated: 121 tests pass across test_sqm, test_solver_sqm,
test_sqm_calibration, test_sweep_frame_record, test_solver_shmem
(9 new tests cover the scaling).
The sensor's OB clamp pins raw black to a target that moves with sensor
state: on the 2026-07-18 imx296 reference sweeps the delivered level was
55.9-56.3 ADU while the device's wizard calibration (bias 58) and the
profile constant (60) both over-subtracted. At dark-site signal levels a
2-4 ADU over-subtraction produces a strong SQM-vs-exposure slope
(-0.9..-1.5 mag/decade) and kills short exposures outright
(background - pedestal <= 0, 'background unresolved').

The BlackLevelTracker already measures the in-session intercept but was
suppressed whenever a wizard calibration existed - precisely backwards: a
stored bias is one moment's measurement of a moving level. Flip the
priority in both SQM paths (radiometric and stellar): a confidently fitted
tracked intercept wins for bias; the wizard's dark-current rate remains
authoritative and is added on top (the intercept fit cannot separate dark
from sky). The tracker's own stderr / deviation-band / lease gates bound
how far it can stray, and an accepted fit >2 ADU from the static bias now
logs at info.

Replayed against the 2026-07-18 imx296 sweeps (radiometric feed, wizard
anchor 58): tracker conditions on all four (stderr 0.26-0.32), fits
55.9-56.3, and the recomputed SQM exposure slopes collapse from
-0.9..-1.5 to -0.01..+0.52 with the residual offset a tight constant
(sd 0.04) for a later band-offset refit.

Also: the published black_level_tracked flag now reflects the leased
pedestal actually used, not the raw last fit.

Validated: 136 tests pass across the SQM suites (4 updated/new).
Rich's 2026-07-18 calibration reports (imx296 + imx462, wizard run
interleaved with the reference sweeps) show two capture defects:

1. The first ~3 frames of every stage report actual_exposure_us at the
   PREVIOUS exposure (req=1us, actual=999999us) yet pass the frame gate,
   because _capture_and_wait checks the exposure_time field, which echoes
   the committed setting rather than what the sensor delivered. Stale
   full-length frames land in the bias/dark stacks.

2. After the exposure change lands, the sensor's optical-black clamp
   re-settles to a new level over ~3 more frames (imx296: 56 -> 59 ADU).
   Frames in that window measure neither the old nor the new black level.
   (On the imx296 the settled clamp is genuinely exposure-dependent:
   ~56 ADU at 1s vs ~59 ADU at the 29us minimum - a min-exposure bias is
   only valid for its own regime, which is why the tracked in-session
   pedestal takes priority in operation.)

Together these explain the wizard bias of 58 on a sensor whose 1s clamp
level is 56: a median over stale-1s frames at 56, settling frames, and
settled-29us frames at 59.

Fix: _capture_and_wait gates on actual_exposure_us (exposure_time only as
fallback), re-requests a capture when a stale frame arrives, and uses an
exposure-proportional tolerance; bias and dark stages discard
CLAMP_SETTLE_DISCARD_FRAMES frames after every exposure change (the dark
ramp changes exposure per step). Adds ~30-40s to a wizard run.

Evidence: dumps/logs/rich-imx296/sqm_calibration_report_imx296_*.json.

Validated: 141 tests pass across the SQM suites (5 new).
feat(sqm): per-frame optical-black pedestal for IMX290/IMX462 (zero-touch)
Telemetry menu under Settings > Telemetry:
- Record: inline On/Off toggle (no submenu), same pattern as Test Mode.
- Sections: one multi-select checklist with inline checkmarks (like the
  Catalogs filter) for what to record — IMU, SQM, Solves, Targets and
  Images. Toggling applies live to an in-progress recording.
- Max Size: session cap (250 MB / 500 MB / 1 GB / 2 GB / Unlimited).
- Load: unchanged.

Images ships OFF by default, preserving the old telemetry_images=false:
one 512x512 PNG is written per solve, so at ~1 solve/s it costs roughly
300 MB/hour against a few MB/hour for every other section combined.

Session size cap. Frames are written by the camera process, so the
recorder measures the session directory from its flush loop rather than
counting its own writes. On reaching the cap, frame capture is suspended
(the one unbounded consumer) while the event log keeps running, so a
capped session stays useful instead of going dark. Default 1 GB.

Also logs the SQM *ingredients*, not just the end products, so a session
stays recomputable if the SQM/airglow maths changes in code later:
- radio event gains the raw per-frame inputs — red/blue Bayer backgrounds,
  optical-black pedestal, photometry image size (green bg and exposure
  were already there).
- the session header snapshots the FULL camera profile constants plus the
  airglow calibration and camera_type — the constants those ingredients
  were produced under.
The derived published 'sqm' and applied 'floor' are still logged, but
explicitly as a what-the-device-showed audit trail, not ground truth.

The airglow import is optional: the model is part of the SQM stack only on
branches that carry it (deepchart has the radiometer but not airglow), and
there a session simply records no airglow constants.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

testable Ready for testing via PiFinder software update

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants