test: provider conformance testkit + weekly live smoke against real APIs#11
Merged
Merged
Conversation
Add @refkit/provider-testkit (private, workspace-only) with
searchConformant() and expectLicenseMap(): reusable assertions that run a
provider's search through fixture or live fetch and check schema validity,
id/providerId identify the provider, licenseVersion is only on CC-family
licenses, and D8 image-URL rules for image providers.
id-rule: settled on exact prefix match `ref.id.startsWith(`${provider.id}:`)`
plus `ref.source.providerId === provider.id`, dropping the plan draft's OR
fallback — verified every current provider factory (including dual-factory
packages like openverse/openverse-audio, pexels/pexels-video,
pixabay/pixabay-video, polyhaven/ambientcg) stamps referenceId(providerId, …)
and source.providerId with the exact same string as provider.id, so the
exact match is sufficient and still catches a provider stamping the wrong
provider's id.
Also registers packages/provider-testkit/vitest.config.ts in the root
aggregator (vitest.config.ts enumerates package paths explicitly).
…l checks; field-wise license map compare - D8 thumbnail: fail only when the thumbnail reuses the item's landing page (=== canonicalUrl or === source.sourceUrl); extensionless CDN thumbnails (openverse /thumb/, smithsonian deliveryService, unsplash query-param URLs) no longer false-fail — isLikelyImageUrl is now only a fast-accept, never a failure condition. Verified the REAL openverse factory + its real fixture passes searchConformant (would have thrown before this fix). - Preview rule unchanged: image providers must declare image/* mediaType. - Add negative self-tests for the previously unguarded checks: foreign id prefix, source.providerId mismatch, text/html preview mediaType; plus a passing extensionless-CDN thumbnail case. - expectLicenseMap: replace key-order-sensitive JSON.stringify equality with field-wise compare (license+version exact; jurisdiction only when the expectation specifies it), widen case type to mapRightsUrl's shape, and add self-tests for key-order insensitivity and jurisdiction handling. - JSDoc caveat on enforceImageUrls for mixed-modality factories.
Add packages/provider-testkit/src/live.ts exporting liveSmoke(), an env-gated describe.skipIf wrapper around searchConformant() that runs one real query against the live API (30s timeout) and asserts refs.length > 0. Exported via a separate "./live" package export so consuming test files import from '@refkit/provider-testkit/live', keeping vitest out of the main entry. Add src/__tests__/live.test.ts to all 18 provider packages: keyless (met, artic, wikimedia-commons, openverse+openverse-audio, gutendex, poetrydb, rijksmuseum, polyhaven+ambientcg, internet-archive) and BYOK (unsplash/UNSPLASH_KEY, pexels+pexels-video/PEXELS_KEY, pixabay+pixabay-video/PIXABAY_KEY, flickr/FLICKR_KEY, smithsonian/SI_KEY, brave/BRAVE_TOKEN, freesound/FREESOUND_TOKEN, jamendo/JAMENDO_CLIENT_ID, europeana/EUROPEANA_KEY), matching packages/mcp/src/cli.ts's defaultProviders construction exactly. Text providers (gutendex, poetrydb) use query 'love'. Add @refkit/provider-testkit devDependency to every touched package still missing it, plus vitest as a direct dependency of provider-testkit (live.ts imports it). Root package.json gets a test:live script (REFKIT_LIVE=1 vitest run). Carry-over from W3.2 review: hoist europeana's inline pageOnly fixture to module scope (ITEM_PAGE_ONLY) and include it in the conformance test's fixture array, exercising the landing-page-rejection path. europeana keeps page-only items via their thumbnail (not dropped), so the count assertion holds unchanged. Hermetic default (pnpm test:run): 304 passed, 22 new skipped (live suites), totals otherwise unchanged. Spot-checked REFKIT_LIVE=1 against the real met and poetrydb APIs — both green.
No changeset: testkit is private, live suites and the workflow are dev-only, and provider devDependency additions do not change published artifacts.
…of deriving from the token table
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Wave 3 of the hardening plan (test infrastructure):
@refkit/provider-testkit(private, unpublished):searchConformantencodes the provider-conformance rules (schema validity, provider id stamping, licenseVersion-only-on-CC-families, D8 no-landing-page-as-media) as reusable assertions;expectLicenseMapfor data-driven license-mapping checks. Adopted in openverse + europeana as proof.REFKIT_LIVE=1; BYOK suites additionally key-gated) reuse the same conformance assertions against real APIs — upstream drift becomes visible instead of silent. Defaultpnpm test:runstays fully hermetic (verified: zero live suites selected without the env)..github/workflows/live-smoke.yml, Mondays 03:17 UTC + manual dispatch, read-only permissions): keyless providers always; BYOK when repo secrets exist.No changesets: testkit is private, live suites/workflow are dev-only, and provider devDependency additions don't change published artifacts (verified against
fileswhitelists).Test Plan
pnpm test:run— 304 passed / 22 skipped (hermetic; zero regressions)pnpm -r typecheck— all packages cleanREFKIT_LIVE=1against live Met + PoetryDB APIs — 8/8 green