Skip to content

fix: unbreak integration tests on the read-only build branch, and cover it with a test - #60

Merged
cb1kenobi merged 5 commits into
use-readonlyfrom
fix/readonly-flush-databases-sandbox-import
Jul 27, 2026
Merged

fix: unbreak integration tests on the read-only build branch, and cover it with a test#60
cb1kenobi merged 5 commits into
use-readonlyfrom
fix/readonly-flush-databases-sandbox-import

Conversation

@dawsontoth

Copy link
Copy Markdown
Contributor

Targets use-readonly. Fixes the integration-test failures on #41 and adds the regression test the branch was missing.

Why the tests were failing

import { flushDatabases } from 'harper' is the whole problem.

Harper loads components through a sandboxed VM loader, and inside it harper is a SyntheticModule built from a hardcoded allowlistgetHarperExports() in harper's security/jsLoader.ts. That list is not the package's index.ts exports. flushDatabases is a public package export (since 5.1.0) but is absent from the allowlist in the pinned 5.1.23, and still absent on harper main.

A named import of a non-allowlisted symbol fails at link time, which kills the entire component:

SyntaxError: The requested module 'harper' does not provide an export named 'flushDatabases'

So @harperfast/nextjs never loaded and no Next.js server started. 15 of 21 tests failed; the 6 that passed only hit Harper's own operations/REST endpoints. This is only visible in Harper's stderr.log (the harper-logs-node-* CI artifacts) — Playwright just reports missing elements, which is what made it look like a Next.js problem.

HARPER_READONLY itself was never broken; it works fine in 5.1.23.

The fix is a namespace import, which always links, plus a guarded call. Verified locally: 24 passed / 1 skipped, and unit tests green.

Worth your attention

The flush is currently a no-op on every released Harper. The warning this PR adds fires on all of them, which means the flushDatabases() call would never have run even if the import had linked. The real fix is one line in harper — adding flushDatabases to getHarperExports(). Happy to open that PR if you agree; until it lands, read-only children still open the databases and replay the on-disk WAL, so they only miss very recent writes.

That's not hypothetical — the new test demonstrates it. The prerendered page renders an empty list while GET /Dog/rex returns Rex, so I left that one assertion test.fixme'd with a pointer to why. It's the assertion that will prove the flush works once Harper exposes it.

--install-links silently served stale plugin code. It copies file:../.. instead of symlinking, and npm treats the copy as satisfying the tree and re-extracts it from its cache — so a rebuilt dist/ never reached the fixtures, even after deleting node_modules/@harperfast. npm run install:fixtures now removes the copy and npm's hidden .package-lock.json. Without that second removal npm just restores the cache. This one cost me a while, and would have made this branch very confusing to iterate on.

The new fixture can't use the bare harper specifier. Real apps declare harper as a dependency (as nextjs-example does), but the harness deep-copies the fixture directory and harper installs to ~577MB, so the test passes harper's resolved entry path through the environment instead. The mechanism under test is unchanged — a build child loading Harper's storage layer against a locked database — but it is a deviation from how a real app would be written, so flag it if you'd rather take the 577MB.

The page also needs /* turbopackIgnore: true */; both bundlers otherwise try to resolve the runtime path at build time and fail with "Cannot find module as expression is too dynamic".

On the missing regression test

Nothing in the suite covered what this branch fixes — every fixture either never touches Harper data or marks its data pages force-dynamic, so no next build child ever opened the databases. The suite passed just as well with your read-only change reverted; I checked.

fixtures/next-16-static-data closes that: a seeded Dog table and a /dogs page with no force-dynamic, so Next.js prerenders it in a child process that loads Harper. With HARPER_READONLY removed it reproduces your reported error exactly —

While lock file: <root>/database/data/LOCK: Resource temporarily unavailable

— and passes with it. That's the before/after I used to confirm the test actually discriminates.

Also

applications.moduleLoader: 'none' in integrationTests/fixture.ts does not mean "no sandbox". importScoped only skips the VM when the value is exactly 'native', so 'none' falls through to the most sandboxed loader — which is why CI hit the synthetic-module path at all. Left alone here since changing it would alter what the suite covers, but it's misleading and probably not what was intended.


I did not run a cross-model review (the cross-model-review skill isn't installed in my environment), so this has had no independent review pass. Generated by Claude Opus 5.

dawsontoth and others added 3 commits July 27, 2026 13:11
…atal

Harper loads components through a sandboxed VM loader where the `harper` module
is a SyntheticModule built from a hardcoded allowlist (`getHarperExports` in
harper's security/jsLoader.ts), not the package's index.ts exports.
`flushDatabases` is a public package export but is absent from that allowlist in
harper 5.1.23 — and on harper main — so the named import failed at *link* time:

  SyntaxError: The requested module 'harper' does not provide an export named
  'flushDatabases'

That is fatal to the whole component, so @harperfast/nextjs never loaded and no
Next.js server started: 15 of the 21 integration tests failed, every one that
loads a page. The 6 that passed only touch Harper's own operations/REST
endpoints.

A namespace import always links; the property is simply undefined where it isn't
exposed. Guard the call and warn instead, so read-only builds still work on
Harper versions that don't expose the flush.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`--install-links` copies `file:../..` instead of symlinking it, so each fixture
holds a snapshot of dist/ rather than a live view. npm treats that snapshot as
satisfying the tree and re-extracts it from its cache on reinstall, so a rebuilt
dist/ never reached the fixtures: `npm run install:fixtures` silently left tests
running stale plugin code, even after deleting node_modules/@HarperFast.

Remove the copy and npm's hidden lockfile before installing. Without dropping
.package-lock.json npm just restores the cached copy.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Nothing in the suite exercised the failure this branch fixes. Every fixture
either never touches Harper data or marks its data pages `force-dynamic`, so no
`next build` child ever opened the RocksDB databases the parent already held —
the tests passed just as well with the read-only change reverted.

Adds a next-16-static-data fixture: a seeded Dog table and a /dogs page with no
`force-dynamic`, so Next.js prerenders it in a build child that loads Harper.
Verified to reproduce the reported error when HARPER_READONLY is not set:

  While lock file: <root>/database/data/LOCK: Resource temporarily unavailable

Fixtures can't depend on `harper` (the harness deep-copies the fixture directory
and the package is ~577MB), so the test passes harper's resolved entry path in
through the environment; `fixture()` now accepts env vars for that.

One assertion is test.fixme'd: the read-only child reads on-disk state, so it
prerenders an empty list even though `GET /Dog/rex` returns Rex. That is the hole
`flushDatabases()` is meant to close, and it stays open while Harper doesn't
expose that export to components.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces regression coverage for Harper's read-only mode during static page generation in Next.js builds. It adds a new integration test fixture (next-16-static-data), updates the fixture installation script to prevent stale plugin caching, and modifies the plugin to import harper as a namespace to safely handle the best-effort flushDatabases call. The review feedback suggests adding defensive guards against undefined globals in the test fixture page and wrapping the flushDatabases call in a try/catch block to prevent transient database errors from failing the entire build.

Comment thread fixtures/next-16-static-data/app/dogs/page.js Outdated
Comment thread src/plugin.ts
@dawsontoth
dawsontoth marked this pull request as ready for review July 27, 2026 17:17
The flush runs before the try/finally that restores the env var, so a throw from
flushDatabases() left HARPER_READONLY='true' on a thread that goes on to serve
requests — the restore never ran. The surrounding comment already called the
flush best-effort; now the code actually is. Raised by gemini-code-assist.

Also make the fixture page fail with a clear message when
HARPER_FIXTURE_HARPER_ENTRY is missing, instead of a bare TypeError on
`tables.Dog`. Deliberately not falling back to an empty list: an empty list is a
meaningful result in this fixture, so swallowing a Harper load failure that way
would make the test pass for the wrong reason.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The warning said the flush was skipped but not what to do about it. Harper does
expose flushDatabases under `applications.moduleLoader: native`, which bypasses
the VM loader and resolves the real package, so the message now names that
setting and its trade-off (no per-application tagged logging or config).

Verified rather than assumed: next-16-static-data-native.pw.ts runs the same
fixture under the native loader, where the warning does not fire and the
statically generated page *does* see the seeded row. Its counterpart in
next-16-static-data.pw.ts asserts the empty list on the default loader, so the
pair pins the behavior down in both directions and replaces the test.fixme.

`fixture()` now takes an options object so a test file can override the
`applications` config.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@cb1kenobi
cb1kenobi merged commit 501e089 into use-readonly Jul 27, 2026
7 checks passed
@cb1kenobi
cb1kenobi deleted the fix/readonly-flush-databases-sandbox-import branch July 27, 2026 17:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants