MAG L1C day-boundary follow-ups: T018/T019, MAG validation data, and gap-management scope

[sapols]

Description of the issue

Follow-up to #2925. The PR addressing that issue (#3323) implements the simplest useful form of cross-day timeline inheritance: when the processing day opens with a gap (including days with no normal-mode data at all), the generated timestamps continue the previous day's timeline, taken from an optional previous-day L1C dependency. Several related items were deliberately left out to keep that change small and un-opinionated about gap management; this issue tracks them.

Remaining work

1. T018 - continue the previous day's L1C timeline (MAG's top priority)

Per @alastairtree (comment below and review on sds-data-manager#1471): the previous-day dependency should be the L1C product rather than norm L1B - MAG is mostly in burst mode, so the previous day's L1C is the timeline that actually exists every day and is the best continuity source. Target dependency set to generate day N L1C (per sensor): day N L1B burst, day N L1B norm, day N-1 L1C (if it exists). This corresponds to SDC Data Validation Document section 4.3.3, case T018.

Two halves:

• Processing side: DONE in MAG L1C: continue previous day's timeline across the day boundary #3323 (commit 8f49ade switched previous_day_dataset from the previous day's norm L1B to its L1C, per the review discussion there). Nothing further needed in imap_processing.
• Delivery side (infra design, the remaining work): confirmed 2026-07-15 in sds-data-manager: the orchestration does not apply date_range when selecting input files (the base IMAPJobHandler.get_science_files_inputs queries only the job's own partition window; Hi's wider queries work via custom_behavior/hi.py), and the trigger sensor's range parser (int(date_range[0][0])) reads only the first character, so negative values like "-1d" fail. Past-day L1C delivery therefore needs a custom_behavior/mag.py (or generic orchestration support). There is also the self-descriptor wrinkle: the L1C job would consume its own previous-day output, so the query must target exactly day N-1 (a plain date_range window cannot exclude day N's own output) and self-trigger loops must be avoided. This design belongs to the infrastructure team; sds-data-manager#1471 is on hold pending it.

Verification / report-back: once delivery of the previous-day L1C ships, regenerate a few known day-boundary gap days, confirm the continued timeline in the real L1C products, and report back on #3323 (promised there).

2. T019 vs T024 specification conflict

T019 (no NM data at all; NM timeline created from BM) says to anchor the synthetic NM timeline at the first BM timestamp plus half the NM sampling period. That conflicts with the day-aligned grid already shipped and validated against T024 (fix #2274): for T024's first BM timestamp 12:38:02.000442, the validated first NM timestamp is the day-aligned 12:38:02.500, whereas the T019 rule gives 12:38:02.250442. Both cannot hold for the same burst-only scenario. Needs a decision from the MAG team (a live discussion is probably more effective than email). Note that #3323 adds a third behavior for this scenario when a previous-day file exists (inherit the previous day's grid); the T019-vs-T024 question is specifically about burst-only days with no usable neighbor.

3. MAG-approved validation data for T017-T019

The SDC Data Validation Document lists the input/output for T017-T019 as "No data provided". The unit tests merged with #3323 (from #3318) lock in the intended leading-gap behavior with synthetic scenarios, but MAG-approved validation data should replace or confirm them when it arrives. (Requested from the MAG team by email; still pending.)

4. Gap management beyond gaps at the beginning of the day

The #3323 change is deliberately un-opinionated about everything except leading gaps:

• Mid-day and trailing gaps still generate timestamps anchored on the current day's own samples.
• Whether following-day data is ever needed (current position: no - the previous day is enough, and any cross-day misalignment lands at the day boundary, which is as good a place as any).
• How inherited cadence should interact with rate changes near the boundary.

These need MAG team input on their exact expectations before further behavior is committed.

5. Timestamp precision (investigated during #2925 - resolved, recorded for reference)

The L1C pipeline carries the timeline in float64 array columns. At 2025-era TTJ2000 magnitudes float64 resolves only multiples of 128 ns, so stored L1C epochs can differ from their exact integer values by up to 64 ns (growing to 128 ns after mid-2036 and 256 ns after 2073 as epochs cross float64 binade boundaries). Per MAG expectations relayed in this review comment, nanosecond precision is not expected at the instrument level and the ~1 us atol used across the MAG validation tests reflects the expected precision - so float64 storage stays well within expectations for the mission lifetime, an int64 end-to-end timeline refactor is not needed, and the #3323 grid-continuity tests assert at that same ~1 us precision. Recorded here so the storage quantization is a known, accepted property rather than a future surprise.

Related, and already landed in #3323: generate_missing_timestamps used to route integer gap bounds through np.rint (float64), quantizing every generated timestamp onto that same 128 ns grid. With the fix, regenerated L1C products can show gap-fill timestamps at rates of 8+ vec/s (whose periods are not multiples of 128 ns) shifted by up to 128 ns - and occasionally one row more or fewer at a gap edge - relative to previously generated products. The new values are the exact intended grid; fills at 1, 2, and 4 vec/s are byte-identical to before. Diffs against archived products in those cases are the bug fix, not a regression.

Production wiring reminder

The previous-day L1C only arrives once the orchestration can deliver past-day inputs at all (the custom_behavior design in item 1). IMAP-Science-Operations-Center/sds-data-manager#1471 is on hold pending that design - its current date_range-on-norm-L1B config is a superseded placeholder, not a working mechanism. Until the delivery work lands and deploys, the timeline-continuation path is dormant.

[sapols]

Just to document it: this issue does NOT include the known "§7.3.4 step 3 doc divergence" item; our pipeline implementation differs from that documentation but MAG has blessed it as acceptable (Maxine worked on that code with them in person).

[alastairtree]

Of the stuff in this ticket MAG team would like T018 - inherit from the previous day's L1C be implemented. The changes in #3323 are nice but do not really address the main gap/continuity problem because we are mostly in burst mode and so continuity from the previous days L1C is the best way to close those day boundary gaps.

[sapols]

@alastairtree Thanks for making the priority clear. I've rewritten item 1 of this issue to be the actionable T018 ticket: previous-day L1C as the continuity source, with your day-N dependency list. The processing side is a small change on top of #3323 (its timeline-continuation mechanism doesn't care which product supplies the previous day's timeline), and I'll have that PR ready quickly.

UPDATE: I ended up doing this inside #3323 itself (8f49ade).

The long pole is delivery: I confirmed the orchestration doesn't yet apply date_range to input selection at all (details in item 1 and #1471), so past-day L1C delivery needs infra work (likely custom behavior, like Hi's). I've flagged the relevant people on the sds PR.