Cache cwd path absolutization via once_cell_cache feature for a large increase in effective parallelism

[alexander-beedie]

For a one-line/no-code update this unlocks a lot of extra speed/parallelism 😄

Example: spark and streamlit now type-check ~35% faster on my machine 🚀

I think this might also positively impact #3993 by reducing getcwd syscalls (these are likely not a big contributor though), so would be marginally additive to #3994.

Summary

Overview

• Profiling across the mypy_primer corpus showed std::env::current_dir() syscalls eating a surprising amount of CPU. The Absolutize trait delegates to the path-absolutize crate, which calls current_dir() on every absolutize(), even for already-absolute paths.

• A project absolutizes once per module (plus per glob/config site), so a large one can issue thousands of unnecessary getcwd syscalls per run.

• This is especially damaging under parallelism: current_dir() is a process-global syscall that contends across threads, so the cost/impact grows with thread count.

Solution

Enabling path-absolutize's "once_cell_cache" feature resolves the cwd just once, via a thread-safe cell, and reuses it. This completely eliminates the parallel contention without touching any pyrefly code/logic.

Test Plan

No type-checking logic (or actual code!) was changed.
All test results are identical before/after.

Benchmarks¹

Timed single-threaded (j1) and multi-threaded (mt) across ~60 of the mypy_primer projects.

mods = how many modules are in the project.
old = pyrefly on main branch.
new = main + this PR.

Results below omit trivially-sized or very fast-running projects.
Time in seconds, min of 4 runs after warmup.

project	mods	old:j1	new:j1	Δ%	old:mt	new:mt	Δ%
streamlit	641	1.656	1.489	−10%	0.437	0.284	−35%
spark	1240	2.977	2.738	−8%	0.628	0.410	−35%
pylox	305	0.233	0.228	−2%	0.178	0.124	−30%
pycryptodome	303	0.282	0.276	−2%	0.176	0.127	−28%
core	9649	14.287	13.996	−2%	2.232	1.614	−28%
strawberry	239	0.435	0.398	−9%	0.177	0.129	−27%
pywin32	471	0.653	0.646	−1%	0.227	0.177	−22%
scrapy	435	0.772	0.755	−2%	0.234	0.183	−22%
mypy	1093	2.599	2.603	+0%	0.499	0.399	−20%
django-stubs	704	0.611	0.604	−1%	0.280	0.225	−20%
dd-trace-py	2840	4.629	4.601	−1%	0.758	0.610	−19%
poetry	441	0.860	0.764	−11%	0.283	0.228	−19%
ignite	359	0.812	0.766	−6%	0.284	0.230	−19%
kornia	496	0.801	0.755	−6%	0.281	0.230	−18%
hydpy	449	1.721	1.595	−7%	0.550	0.451	−18%
helion	408	2.391	2.291	−4%	0.549	0.452	−18%
twisted	860	1.476	1.387	−6%	0.338	0.281	−17%
pandera	393	0.924	0.906	−2%	0.282	0.235	−17%
prefect	838	3.078	3.024	−2%	0.671	0.562	−16%
pip	406	1.224	1.180	−4%	0.334	0.280	−16%
sympy	1550	2.179	2.169	−0%	0.457	0.383	−16%
zulip	1975	4.348	4.145	−5%	0.773	0.663	−14%
bokeh	497	1.372	1.292	−6%	0.387	0.335	−14%
scikit-learn	655	1.528	1.498	−2%	0.390	0.338	−13%
xarray	237	2.865	2.858	−0%	0.749	0.650	−13%
pandas	1458	5.482	5.412	−1%	1.059	0.924	−13%
spack	710	1.712	1.661	−3%	0.383	0.342	−11%
narwhals	313	1.938	1.867	−4%	0.547	0.493	−10%
materialize	877	1.896	1.806	−5%	0.598	0.541	−10%
pwndbg	272	1.180	1.180	+0%	0.602	0.549	−9%
rotki	2361	4.149	4.095	−1%	0.655	0.601	−8%
scipy-stubs	824	4.114	4.120	+0%	0.987	0.924	−6%
static-frame	165	3.958	3.966	+0%	1.142	1.084	−5%
jax	672	5.868	5.754	−2%	1.134	1.082	−5%
colour	715	7.471	7.673	+3%	1.889	1.823	−3%
cryptography	222	0.441	0.439	−1%	0.180	0.174	−3%
setuptools	321	0.702	0.653	−7%	0.181	0.175	−3%
pandas-stubs	346	5.440	5.327	−2%	1.335	1.295	−3%
sphinx	243	1.165	1.120	−4%	0.284	0.275	−3%
freqtrade	333	1.440	1.373	−5%	0.284	0.278	−2%
pytest	243	0.757	0.765	+1%	0.233	0.228	−2%
ibis	563	1.076	1.014	−6%	0.286	0.281	−2%
PyGithub	321	0.442	0.441	−0%	0.178	0.175	−2%
psycopg	251	0.549	0.505	−8%	0.179	0.177	−1%
scipy	930	2.761	2.660	−4%	0.606	0.601	−1%

Footnotes

1. Test machine: Apple Silicon M3 Max. ↩

[yangdanny97]

wow, nice!

[meta-codesync]

@yangdanny97 has imported this pull request. If you are a Meta employee, you can view this in D110782573.

[samwgoldman]

Awesome! Thank you @alexander-beedie :)

[yangdanny97]

Review automatically exported from Phabricator review in Meta.

[meta-codesync]

@samwgoldman merged this pull request in 197d490.

[alexander-beedie]

Awesome! Thank you @alexander-beedie :)

You're welcome ;)