fs: add native fast paths for recursive readdir and buffer I/O#64396
Open
anonrig wants to merge 4 commits into
Open
fs: add native fast paths for recursive readdir and buffer I/O#64396anonrig wants to merge 4 commits into
anonrig wants to merge 4 commits into
Conversation
nodejs-github-bot
added
c++
Move readdirSync({ recursive: true }) into a C++ walk that uses
scandir dirent types so most entries avoid an extra stat(). Add
readFileBuffer and writeFileBuffer bindings so Buffer readFileSync
and writeFileSync avoid open/stat/read/write/close round-trips in JS,
matching the existing utf8 fast paths. Throw ERR_FS_FILE_TOO_LARGE
from C++ when the file exceeds 2 GiB.
On a ~2k-entry tree, recursive readdir is about 2x faster than the
JS walk; buffer read/write see smaller single-digit gains.
Signed-off-by: Yagiz Nizipli <yagiz@nizipli.com>
Member
|
format cpp ;) |
Always call the native readdirRecursiveSync binding and remove the unused JS fallback. Document readFileBuffer, writeFileBuffer, and readdirRecursiveSync on the fs internal binding typings. Signed-off-by: Yagiz Nizipli <yagiz@nizipli.com>
Replace the extra S_IS* macros with a switch on S_IFMT, avoid a temporary string when encoding relative paths, and tighten the directory-descent type checks. Signed-off-by: Yagiz Nizipli <yagiz@nizipli.com>
Use primordials Array instead of the global, return undefined explicitly for the JSDoc check, and match clang-format on the directory descent condition. Signed-off-by: Yagiz Nizipli <yagiz@nizipli.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #64396 +/- ##
==========================================
- Coverage 90.25% 90.19% -0.06%
==========================================
Files 741 741
Lines 241207 241475 +268
Branches 45430 45514 +84
==========================================
+ Hits 217698 217809 +111
- Misses 15084 15167 +83
- Partials 8425 8499 +74
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds C++ fast paths for common
fssync operations, following the same pattern as the existingreadFileUtf8/writeFileUtf8bindings:fs.readdirSync(path, { recursive: true })— full directory walk in C++ (readdirRecursiveSync). Uses scandir dirent types so most entries skip the extrainternalModuleStatthe JS walk does per entry. Symlinks /UV_DIRENT_UNKNOWNstillstat/lstatas needed for correct descent andDirenttypes.fs.readFileSync(path)/encoding: null— single nativereadFileBuffer(open / fstat / read / close). ThrowsERR_FS_FILE_TOO_LARGEwhen size exceeds 2 GiB.fs.writeFileSync(path, buffer)— single nativewriteFileBufferforArrayBufferViewdata whenflushis not set.Public API and results (sorted) stay the same; sequential/parallel readdir recursive and read/write file tests pass.
Benchmarks (arm64 macOS, Release)
Tree: ~2110 entries. Short runs (
n=25for readdir).readdirRecursiveSyncreaddir+internalModuleStatfs.readdirSync({ recursive: true })withFileTypesBuffer I/O (same binary, short
n):readFileSyncsmallreadFileSync64 KiBwriteFileSync1 KiB bufferLarger trees show recursive readdir closer to ~3× (stats dominate).
Test plan
test-fs-readdir-recursive(parallel + sequential)test-fs-readdir-types/test-fs-readdir-types-symlinkstest-fs-readfile(+ empty, fd, flags, utf8 fast path)test-fs-write-file*(sync, buffer, typedarrays, flush)