Avoid mono collection hangs from recursive type growth

[naganohara-yoshino]

Description

The mono item collector currently detects infinite recursive monomorphization
with a depth-based recursion limit. This can be too late for non-regular
recursive instantiations, where each recursive step produces much larger
instance arguments.

This PR starts checking the structural length of instance.args once recursive
instantiation is partway to the recursion limit. If the arguments already exceed
the type length limit, rustc emits the recursion limit diagnostic instead of continuing to process enormous types.

This keeps the diagnostic consistent with the error produced when the same code
is compiled with a smaller #![recursion_limit].

Related Issue

Fixes #156272.

[rustbot]

r? @dingxiangfei2009

rustbot has assigned @dingxiangfei2009.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

Why was this reviewer chosen?

The reviewer was selected based on:

• Owners of files modified in this PR: compiler
• compiler expanded to 73 candidates
• Random selection from 18 candidates

[naganohara-yoshino]

With this change, the code now terminates and reports the recursion-limit diagnostic:

#![allow(dead_code)]

enum Foo<A> {
    Fst,
    Snd(Box<dyn Fn() -> Foo<(A, A)>>),
}

fn recursive<A>(x: Foo<A>) {
    match x {
        Foo::Fst => (),
        Foo::Snd(f) => recursive(f()),
    }
}

fn main() {
    let p0: Foo<()> = Foo::Fst;
    recursive(p0);
}

error: reached the recursion limit while instantiating `recursive::<((((((..., ...), ...), ...), ...), ...), ...)>`
  --> ./app/src/main.rs:11:24
   |
11 |         Foo::Snd(f) => recursive(f()),
   |                        ^^^^^^^^^^^^^^
   |
note: `recursive` defined here
  --> ./app/src/main.rs:8:1
   |
 8 | fn recursive<A>(x: Foo<A>) {
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^
   = note: the full name for the type has been written to 'main.long-type-8599535653221736848.txt'
   = note: consider using `--verbose` to print the full type name to the console

error: aborting due to 1 previous error

UI test is added accordingly.

[rustbot]

This PR modifies tests/ui/issues/. If this PR is adding new tests to tests/ui/issues/,
please refrain from doing so, and instead add it to more descriptive subdirectories.

[rustbot]

Some changes occurred in compiler/rustc_attr_parsing

cc @jdonszelmann, @JonathanBrouwer

rustc_codegen_gcc is developed in its own repository. If possible, consider making this change to rust-lang/rustc_codegen_gcc instead.

cc @antoyo, @GuillaumeGomez

rustc_codegen_cranelift is developed in its own repository. If possible, consider making this change to rust-lang/rustc_codegen_cranelift instead.

cc @bjorn3

Some changes occurred in compiler/rustc_ast/src/expand/typetree.rs

cc @ZuseZ4

Some changes occurred in check-cfg diagnostics

cc @Urgau

Some changes occurred in compiler/rustc_codegen_llvm/src/builder/autodiff.rs

cc @ZuseZ4

Some changes occurred in compiler/rustc_builtin_macros/src/autodiff.rs

cc @ZuseZ4

Some changes occurred in compiler/rustc_codegen_llvm/src/llvm/enzyme_ffi.rs

cc @ZuseZ4

Some changes occurred to diagnostic attributes.

cc @mejrs

This PR modifies bootstrap.example.toml.

If appropriate, please update CONFIG_CHANGE_HISTORY in src/bootstrap/src/utils/change_tracker.rs.

These commits modify the Cargo.lock file. Unintentional changes to Cargo.lock can be introduced when switching branches and rebasing PRs.

If this was unintentional then you should revert the changes before this PR is merged.
Otherwise, you can ignore this comment.

[rustbot]

This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

[bjorn3]

Looks like you made a mistake with rebasing. Try git fetch upstream && git rebase da80ed0708a09dc096c184345d6eb42cbcd50a1e --onto upstream/main (assuming you named rust-lang/rust "upstream") to fix it. Afterwards git log should show that the last commit before your own is on the upstream/main branch.

[naganohara-yoshino]

Looks like you made a mistake with rebasing. Try git fetch upstream && git rebase da80ed0708a09dc096c184345d6eb42cbcd50a1e --onto upstream/main (assuming you named rust-lang/rust "upstream") to fix it. Afterwards git log should show that the last commit before your own is on the upstream/main branch.

Thanks for your reminder. I think I have fixed it now.

[rustbot]

This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.