Skip to content

zlib, vfs: add ZIP archive support and an archive vfs provider#64339

Open
pipobscure wants to merge 1 commit into
nodejs:mainfrom
pipobscure:ziparchives
Open

zlib, vfs: add ZIP archive support and an archive vfs provider#64339
pipobscure wants to merge 1 commit into
nodejs:mainfrom
pipobscure:ziparchives

Conversation

@pipobscure

Copy link
Copy Markdown
Contributor

Summary

node:zlib — ZIP archive support (ZipEntry, ZipFile, ZipBuffer)

  • Read, create, and mutate ZIP archives entirely in pure JS, reusing the existing deflate/zstd bindings (no new native code).
  • ZipEntry: reads and builds individual archive members, supporting store, deflate, and zstd compression methods. Includes a streaming variant (createStream()/contentStream()) for incremental production/consumption without buffering a whole member in memory.
  • ZipBuffer: an in-memory, always-writable view over an archive buffer. Entries can be added, replaced, or deleted; toBuffer() serializes the current live set into a fresh archive with a new central directory.
  • ZipFile: a disk-backed archive, read-only by default and writable via { writable: true }. Adding an entry appends it where the central directory used to be and rewrites the central directory in place; deleting rewrites the central directory only (no growth). compact() streams a fresh archive with dead entries removed, without touching the still-open file.
  • Every method has a synchronous *Sync counterpart, documented as event-loop-blocking (mirroring node:fs's own wording). An internal busy-flag guard prevents a sync call from racing an in-flight async mutation on the same ZipFile.
  • New error codes: ERR_ZIP_ENTRY_CORRUPT, ERR_ZIP_ENTRY_NOT_FOUND, ERR_ZIP_ENTRY_TOO_LARGE, ERR_ZIP_INVALID_ARCHIVE, ERR_ZIP_NOT_WRITABLE, ERR_ZIP_UNSUPPORTED_FEATURE.
  • zlib.getMaxZipContentSize()/setMaxZipContentSize() bound decompressed entry size by default, to guard against zip-bomb-style entries.
  • Encrypted entries and unsupported compression methods are rejected consistently across content(), contentSync(), and contentStream().

node:vfsArchiveProvider

  • Mounts a ZipBuffer or ZipFile as a virtual file system. provider.readonly reflects the underlying archive's own writability.
  • Directories are recognized both explicitly (an entry ending in /) and implicitly (any entry prefixed "<dir>/").
  • Since a ZIP member can't be edited or read in place, writes are buffered in memory and only committed as a new archive entry when the handle is closed.
  • Full synchronous method parity (openSync, statSync, readdirSync, readFileSync/writeFileSync, mkdirSync, rmdirSync, unlinkSync, renameSync, ...), backed by ZipBuffer/ZipFile's own sync surface and documented as event-loop-blocking.

Test plan

  • test/parallel/test-zlib-zip*.js, test/pummel/test-zlib-zip-slow.jsZipEntry/ZipFile/ZipBuffer creation, round-tripping, writability, sync parity, zstd, fuzzing, hardening (encrypted entries, unsupported methods, truncated/garbage archives), zip64 boundaries, metadata, and property-based tests.
  • test/parallel/test-vfs-archive-provider.js — construction validation, stat/readdir over explicit and implicit directories, full async and sync CRUD against both a ZipBuffer and a ZipFile source, read-only enforcement (EROFS).
  • Full node:zlib/node:vfs parallel suite run against a local build (171 tests) to confirm no regressions.

P.S.: my CLA should be on file and I wrote this myself so COO is declared

@nodejs-github-bot nodejs-github-bot added lib / src Issues and PRs related to general changes in the lib or src directory. needs-ci PRs that need a full CI run. labels Jul 7, 2026
@pipobscure pipobscure changed the title zlib, vfs: add ZIP archive read/write support and an archive vfs provider zlib, vfs: add ZIP archive support and an archive vfs provider Jul 7, 2026
@codecov

codecov Bot commented Jul 7, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.82899% with 120 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.31%. Comparing base (4ee2117) to head (2a56a15).

Files with missing lines Patch % Lines
lib/internal/zip.js 95.22% 102 Missing and 9 partials ⚠️
lib/internal/vfs/providers/archive.js 98.28% 7 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #64339      +/-   ##
==========================================
+ Coverage   90.23%   90.31%   +0.07%     
==========================================
  Files         741      743       +2     
  Lines      240979   243856    +2877     
  Branches    45401    46181     +780     
==========================================
+ Hits       217449   220236    +2787     
- Misses      15112    15172      +60     
- Partials     8418     8448      +30     
Files with missing lines Coverage Δ
lib/internal/errors.js 97.71% <100.00%> (+<0.01%) ⬆️
lib/vfs.js 100.00% <100.00%> (ø)
lib/zlib.js 98.36% <100.00%> (+0.02%) ⬆️
lib/internal/vfs/providers/archive.js 98.28% <98.28%> (ø)
lib/internal/zip.js 95.22% <95.22%> (ø)

... and 26 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@pipobscure

pipobscure commented Jul 7, 2026

Copy link
Copy Markdown
Contributor Author

Just to show where this is coming from and where I'm going with this.

pipobscure#3 is a follow on to handle resolving inside a vfs/archive. And based on both of those this is how one could nicely build SEA apps and/or distribute bundled apps.

@pipobscure pipobscure force-pushed the ziparchives branch 3 times, most recently from 0f2b879 to 4258e94 Compare July 7, 2026 19:20
pipobscure added a commit to pipobscure/node that referenced this pull request Jul 7, 2026
Codecov flagged low patch coverage on lib/internal/zip.js and
lib/internal/vfs/providers/archive.js in nodejs#64339. Add tests exercising
Zip64 extra-field parsing, DOS date/time edge cases, streaming-entry
state guards, decodeMemberStream/decodeMemberSync's duplicated error
branches, the ZipBuffer/ZipFile iteration protocols, several on-disk
ZipFile error paths, and ArchiveFileHandle's direct read/write/stat/
truncate surface plus a handful of provider-level error branches the
existing tests didn't reach.
Add ZIP archive read/write support to zlib, plus a VFS provider that
mounts a ZIP archive as a browsable, read/write file system.

zlib gains three classes and a pair of helper functions:

* ZipEntry represents a single archive member: name, metadata, and
  content, created from a Buffer/stream (`create()`/`createStream()`)
  or read back out of raw entry bytes (`read()`). It is otherwise
  immutable; every accessor and static creator has a synchronous and
  an asynchronous form.
* ZipFile wraps a real ZIP file on disk (`open()`/`openSync()`),
  exposing get/add/delete/stream by name plus `compact()` to reclaim
  space left behind by deleted entries. Opened read-only unless
  `{ writable: true }` is passed.
* ZipBuffer is the equivalent fully in-memory representation,
  serializable back to a Buffer with `toBuffer()`/`toBufferSync()`.
  Always writable, since nothing is written until asked to serialize.
* `createZipArchive()`/`createZipArchiveSync()` build a brand-new
  archive (as a stream or a Buffer) from a list of entries.

Every read path enforces `getMaxZipContentSize()`/
`setMaxZipContentSize()` limits and rejects malformed local/central
directory records, guarding against zip-bomb and corrupt-archive
inputs. New ERR_ZIP_* error codes cover corrupt entries, missing
entries, oversized entries, invalid archives, non-writable archives,
and unsupported ZIP features.

node:vfs gains ArchiveProvider, a VirtualProvider backed by an
already-open ZipBuffer or ZipFile: mounting one exposes the archive's
entries through the same fs-like API any other VFS provider offers.
Directories are inferred from entry-name prefixes rather than stored
explicitly. Because a ZIP member can only be written or read in full,
never edited in place, a file opened for writing is buffered and only
committed as a new archive entry when its handle is closed. The
provider's readonly flag mirrors the underlying archive's own
writable state.

Signed-off-by: Philipp Dunkel <pip@pipobscure.com>
@bakkot

bakkot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

See also #45651

Comment thread doc/api/vfs.md

The resolved absolute path used as the root.

## Class: `ArchiveProvider`

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ZipProvider might be better. We might have other forms of "Archive"` later.

@pipobscure pipobscure Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It’s agnostic to the algorithm. So as along at the API shape stays the same it should work with others as well. But I take note and have no objections to changing it.

@pipobscure

pipobscure commented Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

See also #45651

Thanks @bakkot !!!

I think the time has come for it on the one hand, and on the other I added some „motivation“ links earlier. Here some more detail:

Based on this, we can modify the loader to directly load from an archive. If we do that, we get application bundles. pipobscure#3

Which can then in turn be used to easily create SEA apps. https://github.com/pipobscure/experimental-sea

So the world had changed enough that it‘s worth proposing again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lib / src Issues and PRs related to general changes in the lib or src directory. needs-ci PRs that need a full CI run.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants