Skip to content

feat: pgdog.min_lsn read-your-writes replica routing floor#1156

Open
mediprtl wants to merge 1 commit into
pgdogdev:mainfrom
mediprtl:feat/min-lsn-read-routing
Open

feat: pgdog.min_lsn read-your-writes replica routing floor#1156
mediprtl wants to merge 1 commit into
pgdogdev:mainfrom
mediprtl:feat/min-lsn-read-routing

Conversation

@mediprtl

@mediprtl mediprtl commented Jul 7, 2026

Copy link
Copy Markdown

Sticky reads via a per-read LSN floor - #912

Part of #912 (@levkk asked for a PR there). Opt-in read-your-writes that keeps
reads on replicas instead of pinning tables to the primary.

A client sets pgdog.min_lsn (startup options=-c or SET pgdog.min_lsn='X/Y'),
intercepted like pgdog.role. The load balancer routes the read only to a
replica whose replayed LSN (already tracked by the LSN monitor) is >= that floor.
If none qualify it errors recoverably with NoReplicaCaughtUp (SQLSTATE 58000)
carrying a catch-up ETA from the replica's replication lag, or — opt-in —
falls back to the primary.

Addressing the concerns from #912

  • No client-side LSN polling. Our consumers already hold the commit LSN — it
    rides on the async message that triggers the read — so there's no round-trip to
    race. (PG 19's WAIT FOR LSN makes client-side LSN a first-class primitive too.)
  • Opt-in. No param → unchanged behavior. It doesn't replace the row-change /
    per-driver-plugin direction you described — it's a small primitive those can
    build on.
  • Conservative. A stale monitor view only costs an offload opportunity; it
    never serves a stale read.

Testing

pgdog.min_lsn has run continuously in our production and sandbox for 3+ weeks,
built on the LSN monitor PgDog already ships (it reads the monitor's tracked
replayed LSN — no new poller).

Production — 2 read replicas

  • On the order of ~8M reads carrying pgdog.min_lsn over the window.
  • >99.9% were served from a caught-up replica. The remaining <0.1% never
    caught up before the client stopped retrying and fell back on the client side.
    No stale reads — routing is conservative (a read only goes to a replica
    whose replayed LSN is ≥ the floor).
  • Reaching a replica took ~4.6M NoReplicaCaughtUp responses (clients
    retried): most reads were served immediately or after one short wait; the floor
    only holds a read back until the replica catches up.
  • Real lag, not sub-second: replica lag averaged ~2 min and peaked at
    ~50 min over the window. The floor adapts — the reported ETA averaged ~1 min
    and tracked actual lag (max ~26 min).
  • Stable — no PgDog crashes or restarts attributable to the feature.

Sandbox — 1 read replica

  • ~1M NoReplicaCaughtUp responses over ~4 weeks; lower lag / lighter load.

Coverage — unit tests (replica-filter routing, NoReplicaCaughtUp +
ETA-from-lag) and a pgx integration test in integration/load_balancer/pgx/.

Notes / happy to adjust

  • min_lsn_primary_fallback is currently a bool in [general] — glad to fold it
    into an existing routing enum if you'd prefer.
  • replay_lag_seconds lives on the pool's LsnStats wrapper; can move it into
    pgdog_stats::LsnStats if that's the better home.

Add an opt-in read-your-writes floor. A client sets the `pgdog.min_lsn`
connection parameter (startup option or SET) to a commit LSN it must observe;
PgDog then routes the read only to a replica that has replayed at least that far.

- Reads carrying `pgdog.min_lsn` are filtered to replicas whose replayed LSN
  (from the existing LSN monitor) is >= the floor.
- When no replica qualifies, PgDog raises NoReplicaCaughtUp (SQLSTATE 58000)
  carrying a catch-up ETA ("eta ~Ns"), derived from the replica's replication lag
  in time (now() - pg_last_xact_replay_timestamp()), so a client can size a retry.
  The rejection logs at debug (expected backpressure), not error.
- Optional `min_lsn_primary_fallback`: an unmet floor falls back to the primary
  instead of erroring.
- `pgdog.min_lsn` is registered as an untracked parameter; inert when unset or on
  setups without replicas.

Includes unit tests (routing, NoReplicaCaughtUp, ETA-from-lag) and a pgx
integration test.
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants