Skip to content

IL: add ILPreNamespace, make ILPreTypeDef creation lazy - #20092

Open
auduchinok wants to merge 7 commits into
dotnet:mainfrom
auduchinok:il-pre-namespace
Open

IL: add ILPreNamespace, make ILPreTypeDef creation lazy#20092
auduchinok wants to merge 7 commits into
dotnet:mainfrom
auduchinok:il-pre-namespace

Conversation

@auduchinok

@auduchinok auduchinok commented Jul 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

This PR groups lazy type definitions into lazy namespaces structure, allowing to make nested namespaces lazy and defer lazy type defs creation. It improves the custom module readers that we have in Rider.

In addition to that, the PR adds tests asserting the type defs are not eagerly read, which discovered several issues. This PR, #20088, and #20090 all fix these issues. The constraint fix has the most effect for the most users, since without it the type defs were effectively not lazy at all.

The following was measured with the other two PRs included.

Retained memory after ParseAndCheckProject (retain-project, avg of 3):

Project main 8c0e444de branch + other PRs vs main
Console — 168 refs / 3 src 77.7 MB 59.9 MB −17.8 (−22.9 %)
FSharp.Common — 486 refs / 58 src 1319.2 MB 766.0 MB −553.2 (−41.9 %)
FCS — 124 refs / 397 src 2308.7 MB 2289.2 MB noise

Type defs actually read out of the references:

Project top-level TypeDef rows read before constraint fix after namespace levels realised
Console — 168 refs 3 905 3 905 / 3 905 (100 %) 2 280 / 3 905 (58 %) 428 → 300
FSharp.Common — 486 refs 106 980 107 356 (≈100 %) † 34 877 / 106 980 (33 %) 11 262 → 3 964
FCS — 124 refs not instrumented — — —

† Denominator is top-level rows only while the read counter also fires for nested type defs.

ParseAndCheckProject — warm (1 warm-up, then samples with the IL reader cache cleared each time):

Project main: time (min / median) branch: time (min / median) main: allocated branch: allocated
Console — 168 refs / 3 src 109 / 122 ms 95 / 110 ms (−13 % / −10 %) 110.2 MB 96.4 MB (−12.5 %)
FSharp.Common — 486 refs / 58 src 1832 / 2027 ms 1699 / 1968 ms (−7 % / −3 %) 3167 MB 2670 MB (−15.7 %)
FCS — 124 refs / 397 src 3060 ms (min) 2906 ms (min) 14 347 MB 14 340 MB (−0 %)

auduchinok and others added 7 commits July 27, 2026 14:34
@auduchinok
IL: add ILPreNamespace, make ILPreTypeDef creation lazy
6c6966b
@auduchinok @claude
IL: shrink per-ILTypeDefs namespace overhead
230e9f4 
Hold the child namespaces + their by-name lookup in a single
InterruptibleLazy<struct (ILPreNamespace[] * Dictionary<..>)> instead of two
separate lazies, and share one pre-computed empty instance for namespace-less
levels (every nested-type container - the vast majority of ILTypeDefs).

On a single-file FCS check against a project with ~486 references this cut the
namespace InterruptibleLazy wrappers from ~171,600 to ~8,800 and the retained
IL-namespace machinery from ~16.5 MB to ~11.3 MB.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@auduchinok @claude
IL: simplify namespace grouping, restore metadata order when flattening
65e8fcc 
Follow-up cleanup on the ILPreNamespace work:

 - ImportILNamespaceLevel now buckets a level's flat entries and child
   pre-namespaces in one pass instead of via multisetDiscriminateAndMap
 - drop the now-unused readBlobHeapAsSplitTypeName/seekReadPreTypeDef and
   the mkILPreTypeDefEntry signature entry
 - TryFindPreTypeDef is a self-recursive member; simplify namespacesOf
 - flattening a grouped table (AsArray/AsList/the enumerator) restores the
   TypeDef row order, which static linking relies on to emit the types of a
   --standalone assembly in the reader's order
 - tests for laziness, lookup, hybrid levels, duplicate namespace nodes,
   imported entity order and metadata order of a read module

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@auduchinok @claude
IL: restore the TypeDef order where it is needed, not in ILTypeDefs
9e331e8 
Flattening a grouped table walks namespace by namespace, which is not the
TypeDef row order when a namespace is split across the table. Doing the
restoration inside ILTypeDefs cost a flag in the namespaces payload plus a
sort in AllPreTypeDefs, in a memory-critical type; static linking is the only
order-sensitive consumer, and one stable sort by MetadataIndex there covers
it. Verified on a --standalone build: all 296 of FSharp.Core's top-level
types are emitted in FSharp.Core's metadata order.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@auduchinok @claude
WIP: shared namespace-level machinery, import over it, benchmarks
62ef1cd 
Temporary commit of the in-progress work on top of 9e331e8:

- il.fs/il.fsi: one namespace-bucketing implementation (ILPreNamespaceOfEntries)
  reached both by the grouped reader and, via ilTypeDefsAsNamespaceLevel, by any
  flat table; DelayInitValue extracted into illib for the three hand-rolled lazies;
  fixed a race where AsArrayOfPreNamespaces could return an empty array.
- import.fs: ImportILNamespaceLevel and its copy of the bucketing removed;
  ImportILTypeDefs walks the two arrays directly. Plus three unrelated memory
  tweaks (Nullness.GetFlags, shared noTypars, closure capture in ImportILTypeDef).
- CompilerImports.fs: skip addConstraintSources for non-F# CCUs (also ported to
  its own branch off main, to be proposed and measured separately).
- tests: namespace/cancellation tests updated, SurfaceArea baseline.
- benchmarks: NamespaceImportBenchmarks with the retained-memory and
  retain-project probes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@auduchinok @claude
Don't force namespace contents when building the initial TcEnv
4b59802 
AddModuleOrNamespaceRefsToNameEnv runs over the root namespaces of every
referenced assembly, and two checks in it forced each one's contents -
which for an imported namespace means importing every type in it:

- Entity.DemangledModuleOrNamespaceName forced the ModuleOrNamespaceType
  only to read its ModuleOrNamespaceKind. Only FSharpModuleWithSuffix
  demangles, and only a suffixed name can be of that kind, so testing the
  name first avoids the read.

- `modref.IsModule && EntityHasWellKnownAttribute ... AutoOpenAttribute`:
  IsModule forces the contents while the attribute check only reads
  entity_attribs, so the operands are swapped.

Retained memory when checking ReSharper.FSharp's FSharp.Common (486
references): 874.7 -> 780.4 MB (-10.8%). Framework-only projects are
unchanged, since their root namespaces get imported anyway.

Also un-skips `Type defs 02 - assembly import`: no type of the reference
is imported now, closing dotnet#16166.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@auduchinok @claude
Add end-to-end tests for what checking a file reads out of a reference
7ef2ed6 
The reader tests above them pin the API in isolation; these pin the
guarantee it exists for. A synthetic reference assembly records every
GetTypeDef, member, nested-type and attribute read, and each test checks
a file against it and asserts what came out - so a walk introduced far
from the reader (addConstraintSources did exactly that) fails here.

What they pin: naming Ns1.A reads the root level and Ns1's own types and
nothing else; members, nested types and the base type stay unread until
something needs them; a member lookup is what forces the base type's
namespace; attributes are read for the types that enter scope, so an open
pays for all of a namespace's types and a qualified use does not.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@auduchinok
auduchinok requested a review from a team as a code owner July 28, 2026 20:09
@github-actions github-actions Bot added the AI-Tooling-Check-Scanned-Clean Tooling check: diff analyzed, no interesting infrastructure files label Jul 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

AI-Tooling-Check-Scanned-Clean Tooling check: diff analyzed, no interesting infrastructure files

Projects

F# Compiler and Tooling
Status: New

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@auduchinok