ty-aware delayed AST -> HIR lowering

[aerooneqq]

View all comments

This PR implements a prototype of ty-aware delayed AST -> HIR lowering. Part of #118212.

r? @petrochenkov

Motivation

When lowering delegation in perfect scenario we would like to access the ty-level information, in particular, queries like generics_of, type_of, fn_sig for proper HIR generation with less hacks. For example, we do not want to generate more lifetimes than needed, because without ty-level queries we do not know which delegee's lifetimes are late-bound. Next, consider recursive delegations, for their proper support without ty we would have to either duplicate generics inheritance code in AST -> HIR lowering or create stubs for parts of the HIR and materialize them later. We already use those queries when interacting with external crates, however when dealing with compilation of a local crate we should use resolver for similar purposes. Finally, access to ty-level queries is expected to help supporting delegation to inherent impls, as we can not resolve such calls during AST -> HIR stage.

Benefits

We eliminate almost all code that uses resolver in delegation lowering:

• Attributes inheritance is done without copying attributes from AST at resolve stage
• Fn signatures are obtained from tcx.fn_sig
• Param counts are also obtained from tcx.fn_sig
• is_method function now uses tcx.associated_item instead of resolver
• Generics are now inherited through get_external_generics that uses tcx.generics_of. Generics for recursive delegations should also work
• DelegationIds that stored paths for recursive delegations is removed, we now use only delegee_id
• Structs that were used for storing delegation-related information in resolver are almost fully removed
• ast_index is no more used

Next steps

• Remove creating generic params through AST cloning, proper const types propagation
• Inherent impls

High level design overview

Queries

We store ids of delayed items to lower in Crate struct. During first stage of lowering, owners that correspond to delayed items are filled with MaybeOwner::Phantom.

Next, we define two new queries: lower_delayed_owner, delayed_owner and a function force_delayed_owners_lowering.

The first query is used when lowering known (which is in delayed_ids) delayed owner.

The second is fed with children that were obtained during lowering of a delayed owner (note that the result of lowering of a single LocalDefId is not a single MaybeOwner, its a list of (LocalDefId, MaybeOwner) where the first MaybeOwner corresponds to delayed LocalDefId and others to children that were obtained during lowering (i.e. generic params)). By default delayed_owner returns MaybeOwner::Phantom. As we do not want to predict the whole list of children which will be obtained after lowering of a single delayed item, we need to store those children somewhere. There are several options:

• Make the return type of lower_delayed_owner to be FxIndexMap<LocalDefId, MaybeOwner> and search children here. Search will be either linear or we can introduce a map somewhere which will track parent-child relations between a single delayed LocalDefId and its children.
• Try to make query that will lower all delayed items in a loop and return a complete map of all delayed LocalDefIds and their children. In this case there will be problems with delayed items that require information about other delayed items.

By using proposed queries we handle the second concern, and in case of acyclic dependencies between delayed ids it automatically works, moreover we use queries as cache for delayed MaybeOwners. The only invariant here is that queries which are invoked during delayed AST -> HIR lowering of some LocalDefId should not in any way access children of other, yet unlowered, delayed LocalDefId, they should firstly materialize it.

The force_delayed_owners_lowering forces lowering of all delayed items and now integrated in hir_crate_items query.

Resolver for lowering

Currently the resolver_for_lowering is stolen in lower_to_hir function, however we want to prolong its life until all delayed LocalDefIds are materialized. For this purpose we borrow resolver_for_lowering in lower_to_hir and drop it after forcing materialization of all delayed LocalDefId in rustc_interface::run_required_analyses.

We split resolver for lowering into two parts: the first part is a readonly part that came to us from resolve, the second part is a mutable part that can be used to add or overwrite values in the readonly part. Such splitted resolver is used during delayed lowering, as we can't steal it.

AST index

Lowering uses an AST index. It is created in lower_to_hir function and it references parts of AST. We want to avoid reindexing AST on every delayed LocalDefId lowering, however now it is not clear how to properly do it. As delayed HIR lowering is used only for delegation unstable feature it should not affect other use-cases of the compiler. But it will be reworked sooner or later.

[petrochenkov]

cc @oli-obk you may be interested in this, because this likely intersects with incremental AST lowering.

[petrochenkov]

@bors try @rust-timer queue

[rust-bors]

☀️ Try build successful (CI)
Build commit: a77c1ae (a77c1ae1722badff1c427c67e07dc1e08989756f, parent: f82485388963286763f8c8a9261ac227627f6a18)

[rust-timer]

Finished benchmarking commit (a77c1ae): comparison URL.

Overall result: ❌✅ regressions and improvements - please read the text below

Benchmarking this pull request means it may be perf-sensitive – we'll automatically label it not fit for rolling up. You can override this, but we strongly advise not to, due to possible changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please do so in sufficient writing along with @rustbot label: +perf-regression-triaged. If not, please fix the regressions and do another perf run. If its results are neutral or positive, the label will be automatically removed.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	0.7%	[0.1%, 3.5%]	36
Regressions ❌ (secondary)	0.7%	[0.0%, 2.7%]	103
Improvements ✅ (primary)	-0.6%	[-1.4%, -0.2%]	9
Improvements ✅ (secondary)	-0.5%	[-0.8%, -0.2%]	18
All ❌✅ (primary)	0.4%	[-1.4%, 3.5%]	45

Max RSS (memory usage)

Results (primary 4.3%, secondary 3.2%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	4.3%	[0.4%, 27.0%]	57
Regressions ❌ (secondary)	3.4%	[1.0%, 24.6%]	83
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-3.7%	[-4.2%, -3.2%]	2
All ❌✅ (primary)	4.3%	[0.4%, 27.0%]	57

Cycles

Results (primary 2.7%, secondary 3.4%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	2.7%	[2.1%, 3.2%]	2
Regressions ❌ (secondary)	3.9%	[1.3%, 8.5%]	11
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-2.2%	[-2.2%, -2.2%]	1
All ❌✅ (primary)	2.7%	[2.1%, 3.2%]	2

Binary size

Results (primary 0.0%, secondary 0.1%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	0.0%	[0.0%, 0.1%]	11
Regressions ❌ (secondary)	0.1%	[0.0%, 0.2%]	16
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.0%	[0.0%, 0.1%]	11

Bootstrap: 480.442s -> 479.934s (-0.11%)
Artifact size: 397.10 MiB -> 397.38 MiB (0.07%)

[petrochenkov]

@bors try @rust-timer queue

[aerooneqq]

@rustbot ready

[petrochenkov]

r=me after squashing commits.
@rustbot author

[rustbot]

Reminder, once the PR becomes ready for a review, use @rustbot ready.

[aerooneqq]

@rustbot ready

[petrochenkov]

@bors r+

[rust-bors]

📌 Commit 5c441e6 has been approved by petrochenkov

It is now in the queue for this repository.

[rust-bors]

☀️ Test successful - CI
Approved by: petrochenkov
Duration: 3h 22m 11s
Pushing 461e973 to main...

[github-actions]

What is this?

This is an experimental post-merge analysis report that shows differences in test outcomes between the merged PR and its parent PR.

Comparing 86c839f (parent) -> 461e973 (this PR)

Test differences

Show 222 test diffs

222 doctest diffs were found. These are ignored, as they are noisy.

Test dashboard

Run

cargo run --manifest-path src/ci/citool/Cargo.toml -- \
    test-dashboard 461e9738a47e313e4457957fa95ff6a19a4b88d4 --output-dir test-dashboard

And then open test-dashboard/index.html in your browser to see an overview of all executed tests.

Job duration changes

1. dist-x86_64-apple: 2h 8m -> 1h 46m (-17.0%)
2. dist-aarch64-llvm-mingw: 1h 33m -> 1h 49m (+16.6%)
3. x86_64-gnu: 2h 8m -> 2h 28m (+14.8%)
4. x86_64-gnu-llvm-21-2: 1h 42m -> 1h 28m (-13.8%)
5. dist-aarch64-apple: 1h 55m -> 1h 40m (-12.7%)
6. x86_64-msvc-ext1: 2h 3m -> 2h 17m (+11.3%)
7. dist-aarch64-msvc: 1h 49m -> 1h 36m (-11.2%)
8. x86_64-gnu-aux: 2h 8m -> 2h 21m (+10.2%)
9. i686-gnu-nopt-1: 2h 25m -> 2h 11m (-9.7%)
10. i686-msvc-1: 3h 1m -> 2h 47m (-8.0%)

How to interpret the job duration changes?

Job durations can vary a lot, based on the actual runner instance
that executed the job, system noise, invalidated caches, etc. The table above is provided
mostly for t-infra members, for simpler debugging of potential CI slow-downs.

[rust-timer]

Finished benchmarking commit (461e973): comparison URL.

Overall result: ❌ regressions - no action needed

@rustbot label: -perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	0.2%	[0.2%, 0.2%]	1
Regressions ❌ (secondary)	0.1%	[0.1%, 0.1%]	1
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.2%	[0.2%, 0.2%]	1

Max RSS (memory usage)

Results (primary 0.8%, secondary 1.5%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	0.8%	[0.5%, 1.0%]	3
Regressions ❌ (secondary)	1.5%	[1.0%, 1.9%]	2
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.8%	[0.5%, 1.0%]	3

Cycles

Results (secondary 1.6%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	1.6%	[1.6%, 1.6%]	1
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-	-	0

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 491.123s -> 488.876s (-0.46%)
Artifact size: 396.86 MiB -> 396.98 MiB (0.03%)