various fixes for scalable vectors by davidtwco · Pull Request #153286 · rust-lang/rust

davidtwco · 2026-03-02T12:58:23Z

These are a handful of patches fixing bugs with the current #[rustc_scalable_vector] infrastructure so that we can upstream the intrinsics to stdarch:

Instead of just using regular struct lowering for tuples of scalable vectors, which results in an incorrect ABI (e.g. returning indirectly), we change to using BackendRepr::ScalableVector which will lower to the correct type and be passed in registers.
Clang changed from representing tuples of scalable vectors as structs rather than as wide vectors (that is, scalable vector types where the N part of the <vscale x N x ty> type was multiplied by the number of vectors). rustc mirrored this in the initial implementation of scalable vectors that didn't land.

When our early patches used the wide vector representation, our intrinsic patches used the legacy llvm.aarch64.sve.tuple.{create,get,set}{2,3,4} intrinsics for creating these tuples/getting/setting the vectors, which were only supported due to LLVM's AutoUpgrade pass converting these intrinsics into llvm.vector.insert. AutoUpgrade only supports these legacy intrinsics with the wide vector representation.

With the current struct representation, Clang has special handling in codegen for generating insertvalue/extractvalue instructions for these operations, which must be replicated by rustc's codegen for our intrinsics to use.

We add a new core::intrinsics::scalable module (never intended to be stable, just used by the stdarch intrinsics, gated by core_intrinsics) and add new intrinsics which rustc lowers to the appropriate insertvalue/extractvalue instructions.
Add generation of debuginfo for scalable vectors, following the DWARF that Clang generates.
Some intrinsics need something like simd_cast, which will work for scalable vectors too, so this implements a new sve_cast intrinsic that uses the same internals as simd_cast. This seemed better than permitting scalable vectors to be used with the simd_* intrinsics more generally as I can't guarantee this would work for all of them.

This is a relatively large patch but most of it is tests, and each commit should be relatively standalone. It's a little bit easier to upstream them together to avoid needing to stack them. It's possible that some more compiler fixes will be forthcoming but it's looking like this might be all at the moment.

Depends on #153653

r? @workingjubilee (discussed on Zulip)

rustbot · 2026-03-02T12:58:26Z

Some changes occurred to the intrinsics. Make sure the CTFE / Miri interpreter
gets adapted for the changes, if necessary.

cc @rust-lang/miri, @RalfJung, @oli-obk, @lcnr

This PR changes rustc_public

cc @oli-obk, @celinval, @ouz-a, @makai410

Some changes occurred in compiler/rustc_codegen_gcc

cc @antoyo, @GuillaumeGomez

rustbot · 2026-03-02T12:58:28Z

workingjubilee is currently at their maximum review capacity.
They may take a while to respond.

library/core/src/intrinsics/scalable.rs

library/core/src/intrinsics/simd/scalable.rs

compiler/rustc_abi/src/layout.rs

compiler/rustc_codegen_llvm/src/debuginfo/metadata.rs

compiler/rustc_abi/src/lib.rs

compiler/rustc_codegen_llvm/src/intrinsic.rs

tests/codegen-llvm/scalable-vectors/sve-undef.rs

rustbot · 2026-03-02T16:06:09Z

triagebot.toml has been modified, there may have been changes to the review queue.

cc @wesleywiser

rustbot · 2026-03-02T17:33:11Z

Some changes occurred to the CTFE machinery

cc @RalfJung, @oli-obk, @lcnr

Some changes occurred to MIR optimizations

cc @rust-lang/wg-mir-opt

Some changes occurred to the CTFE / Miri interpreter

cc @rust-lang/miri

lqd · 2026-03-10T11:33:47Z

Just to hopefully make this easier to review and help Jubilee, could you, for a start, extract the first three commits into a different PR and assign it to me?

davidtwco · 2026-03-10T11:40:12Z

Just to hopefully make this easier to review and help Jubilee, could you, for a start, extract the first three commits into a different PR and assign it to me?

Split into #153653

@lqd

scalable vector: type renames and simple checks Split out from rust-lang#153286 Per rust-lang#153286 (comment), renames `VectorKind` type to `SimdVectorKind` and `BackendRepr::ScalableVector` to `BackendRepr::SimdScalableVector`. Also adds a new check in `rustc_hir_analysis` enforcing that tuples of scalable vectors should only up to eight vectors. r? @lqd

@lqd

scalable vector: type renames and simple checks Split out from rust-lang#153286 Per rust-lang#153286 (comment), renames `VectorKind` type to `SimdVectorKind` and `BackendRepr::ScalableVector` to `BackendRepr::SimdScalableVector`. Also adds a new check in `rustc_hir_analysis` enforcing that tuples of scalable vectors should only up to eight vectors. r? @lqd

@lqd

scalable vector: type renames and simple checks Split out from rust-lang#153286 Per rust-lang#153286 (comment), renames `VectorKind` type to `SimdVectorKind` and `BackendRepr::ScalableVector` to `BackendRepr::SimdScalableVector`. Also adds a new check in `rustc_hir_analysis` enforcing that tuples of scalable vectors should only up to eight vectors. r? @lqd

@lqd

scalable vector: type renames and simple checks Split out from rust-lang#153286 Per rust-lang#153286 (comment), renames `VectorKind` type to `SimdVectorKind` and `BackendRepr::ScalableVector` to `BackendRepr::SimdScalableVector`. Also adds a new check in `rustc_hir_analysis` enforcing that tuples of scalable vectors should only up to eight vectors. r? @lqd

@lqd

Rollup merge of #153653 - davidtwco:sve-cleanups, r=lqd scalable vector: type renames and simple checks Split out from #153286 Per #153286 (comment), renames `VectorKind` type to `SimdVectorKind` and `BackendRepr::ScalableVector` to `BackendRepr::SimdScalableVector`. Also adds a new check in `rustc_hir_analysis` enforcing that tuples of scalable vectors should only up to eight vectors. r? @lqd

@lqd

scalable vector: type renames and simple checks Split out from rust-lang/rust#153286 Per rust-lang/rust#153286 (comment), renames `VectorKind` type to `SimdVectorKind` and `BackendRepr::ScalableVector` to `BackendRepr::SimdScalableVector`. Also adds a new check in `rustc_hir_analysis` enforcing that tuples of scalable vectors should only up to eight vectors. r? @lqd

workingjubilee

I have a weird nit... not that noting a typo is weird, but rather that it is on a commit message.

The PR looks correct otherwise. I'm going to doublecheck the debuginfo part after a nap and approve the PR then assuming I don't find anything worth noting past that commit message nit. Feel free to rebase.

View changes since this review

compiler/rustc_codegen_llvm/src/intrinsic.rs

compiler/rustc_codegen_llvm/src/type_of.rs

compiler/rustc_codegen_llvm/src/intrinsic.rs

Instead of just using regular struct lowering for these types, which results in an incorrect ABI (e.g. returning indirectly), use `BackendRepr::ScalableVector` which will lower to the correct type and be passed in registers. This also enables some simplifications for generating alloca of scalable vectors and greater re-use of `scalable_vector_parts`. A LLVM codegen test demonstrating the changed IR this generates is included in the next commit alongside some intrinsics that make these tuples usable.

rustbot · 2026-03-17T10:27:06Z

This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

workingjubilee

Ah, I wound up having a question about the intrinsic as presented in the standard library. I had missed a detail on my first scan.

View changes since this review

compiler/rustc_codegen_llvm/src/debuginfo/metadata.rs

workingjubilee · 2026-03-18T07:41:02Z

compiler/rustc_codegen_llvm/src/debuginfo/metadata.rs

+
+    let number_of_elements: u64 = (element_count as u64) * (number_of_vectors.0 as u64);
+    let number_of_elements_per_vg = number_of_elements / 2;
+    let mut expr = smallvec::SmallVec::<[u64; 9]>::new();


Hmm. Are we really saving much overhead with such a large "SmallVec"?

I'm happy to change it if it isn't worth it

workingjubilee · 2026-03-18T07:44:22Z

compiler/rustc_codegen_llvm/src/debuginfo/metadata.rs

+        let vector_ty = llvm::LLVMRustDICreateVectorType(
+            di_builder,
+            /* Size */ 0,
+            layout.align.bits() as u32,
+            element_di_node,
+            subscripts,
+            bitstride,
+        );


maybe we should define this as an argument struct if we have to comment... I'll open an issue.

I take it the given size is "0" because it is indeterminate, and we're just trusting the debugger to figure that out from the subrange expression being there? Or does the argument go unused...?

The debuginfo change is largely porting what Clang generates for its scalable vectors, which I've presumed matches what the debuggers expect to find so that they work correctly - I presume the 0 indicates the vector is scalable and that the debugger should use the subrange with the expression to determine the size dynamically.

workingjubilee · 2026-03-18T07:56:14Z

library/core/src/intrinsics/simd/scalable.rs

+#[cfg(target_arch = "aarch64")]
+#[rustc_intrinsic]
+#[rustc_nounwind]
+#[target_feature(enable = "sve")]
+pub unsafe fn sve_cast<T, U>(x: T) -> U;


Is this target_feature annotation necessary? Most of the intrinsics in the supermodule are not THAT kind of intrinsic... they are expected to generate code based on the surrounding features correctly without requiring repeating specifications for each architecture, because they address a pretty fundamental concept usually.

I don't even think the concept here for tuples are specific in a meaningful way to AArch64 SVE, RISCV has its weird register group thingies that are Basically Also Homogenous Tuples.

I added them just to try and limit the chance that they're used incorrectly, figuring it was easy to remove the annotation if we wanted to. I don't know enough about RVV to know if these intrinsics would be useful for them or if they'd need changes (I think they might still use the wide vector representation, so the tuple intrinsics might not be useful to them, and whether they use sve_cast would depend on their intrinsic definitions probably),

library/core/src/intrinsics/simd/scalable.rs

Clang changed to representing tuples of scalable vectors as structs rather than as wide vectors (that is, scalable vector types where the `N` part of the `<vscale x N x ty>` type was multiplied by the number of vectors). rustc mirrored this in the initial implementation of scalable vectors. Earlier versions of our patches used the wide vector representation and our intrinsic patches used the legacy `llvm.aarch64.sve.tuple.{create,get,set}{2,3,4}` intrinsics for creating these tuples/getting/setting the vectors, which were only supported due to LLVM's `AutoUpgrade` pass converting these intrinsics into `llvm.vector.insert`. `AutoUpgrade` only supports these legacy intrinsics with the wide vector representation. With the current struct representation, Clang has special handling in codegen for generating `insertvalue`/`extractvalue` instructions for these operations, which must be replicated by rustc's codegen for our intrinsics to use. This patch implements new intrinsics in `core::intrinsics::scalable` (mirroring the structure of `core::intrinsics::simd`) which rustc lowers to the appropriate `insertvalue`/`extractvalue` instructions.

Generate debuginfo for scalable vectors, following the structure that Clang generates for scalable vectors.

Abstract over the existing `simd_cast` intrinsic to implement a new `sve_cast` intrinsic - this is better than allowing scalable vectors to be used with all of the generic `simd_*` intrinsics.

@lqd

scalable vector: type renames and simple checks Split out from rust-lang/rust#153286 Per rust-lang/rust#153286 (comment), renames `VectorKind` type to `SimdVectorKind` and `BackendRepr::ScalableVector` to `BackendRepr::SimdScalableVector`. Also adds a new check in `rustc_hir_analysis` enforcing that tuples of scalable vectors should only up to eight vectors. r? @lqd

@lqd

scalable vector: type renames and simple checks Split out from rust-lang/rust#153286 Per rust-lang/rust#153286 (comment), renames `VectorKind` type to `SimdVectorKind` and `BackendRepr::ScalableVector` to `BackendRepr::SimdScalableVector`. Also adds a new check in `rustc_hir_analysis` enforcing that tuples of scalable vectors should only up to eight vectors. r? @lqd

davidtwco added the F-scalable-vectors `#[rustc_scalable_vector]` label Mar 2, 2026

rustbot added the A-LLVM Area: Code generation parts specific to LLVM. Both correctness bugs and optimization-related issues. label Mar 2, 2026

rustbot assigned workingjubilee Mar 2, 2026

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. T-libs Relevant to the library team, which will review and decide on the PR/issue. labels Mar 2, 2026

This comment has been minimized.

Sign in to view

davidtwco force-pushed the sve-intrinsics branch from ed74abe to dc9af49 Compare March 2, 2026 14:01

This comment has been minimized.

Sign in to view

RalfJung reviewed Mar 2, 2026

View reviewed changes

library/core/src/intrinsics/scalable.rs Outdated Show resolved Hide resolved

RalfJung reviewed Mar 2, 2026

View reviewed changes

library/core/src/intrinsics/simd/scalable.rs Show resolved Hide resolved

RalfJung reviewed Mar 2, 2026

View reviewed changes

library/core/src/intrinsics/simd/scalable.rs Show resolved Hide resolved

RalfJung reviewed Mar 2, 2026

View reviewed changes

compiler/rustc_abi/src/layout.rs Outdated Show resolved Hide resolved

Amanieu reviewed Mar 2, 2026

View reviewed changes

davidtwco force-pushed the sve-intrinsics branch from dc9af49 to 06b46c4 Compare March 2, 2026 15:40

This comment was marked as resolved.

Sign in to view

davidtwco force-pushed the sve-intrinsics branch from 06b46c4 to 4329945 Compare March 2, 2026 16:06

rustbot added the A-meta Area: Issues & PRs about the rust-lang/rust repository itself label Mar 2, 2026

This comment was marked as resolved.

Sign in to view

davidtwco force-pushed the sve-intrinsics branch from 4329945 to ab6b50b Compare March 2, 2026 17:33

davidtwco force-pushed the sve-intrinsics branch from ab6b50b to 5fc0060 Compare March 2, 2026 17:49

This comment was marked as resolved.

Sign in to view

davidtwco force-pushed the sve-intrinsics branch from 5fc0060 to f8b4bc8 Compare March 2, 2026 18:15

This comment has been minimized.

Sign in to view

davidtwco force-pushed the sve-intrinsics branch from f8b4bc8 to cfb2032 Compare March 2, 2026 18:42

This comment has been minimized.

Sign in to view

davidtwco force-pushed the sve-intrinsics branch from b904ac4 to f323831 Compare March 9, 2026 10:07

This comment was marked as resolved.

Sign in to view

davidtwco force-pushed the sve-intrinsics branch from f323831 to 3c65b93 Compare March 10, 2026 10:39

This comment has been minimized.

Sign in to view

davidtwco mentioned this pull request Mar 10, 2026

scalable vector: type renames and simple checks #153653

Merged

This comment was marked as resolved.

Sign in to view

davidtwco force-pushed the sve-intrinsics branch from 3c65b93 to 345b28e Compare March 11, 2026 10:15

This comment has been minimized.

Sign in to view

workingjubilee reviewed Mar 17, 2026

View reviewed changes

compiler/rustc_codegen_llvm/src/intrinsic.rs Show resolved Hide resolved

compiler/rustc_codegen_llvm/src/type_of.rs Show resolved Hide resolved

compiler/rustc_codegen_llvm/src/intrinsic.rs Show resolved Hide resolved

compiler/rustc_codegen_llvm/src/intrinsic.rs Show resolved Hide resolved

davidtwco mentioned this pull request Mar 17, 2026

SVE and SME on AArch64 rust-lang/rust-project-goals#270

Open

27 tasks

davidtwco force-pushed the sve-intrinsics branch from 345b28e to b0b5ef1 Compare March 17, 2026 10:27

workingjubilee reviewed Mar 18, 2026

View reviewed changes

davidtwco added 3 commits March 18, 2026 10:59

cg_llvm/debuginfo: scalable vectors

b1f5b46

Generate debuginfo for scalable vectors, following the structure that Clang generates for scalable vectors.

cg_llvm: sve_cast intrinsic

003827f

Abstract over the existing `simd_cast` intrinsic to implement a new `sve_cast` intrinsic - this is better than allowing scalable vectors to be used with all of the generic `simd_*` intrinsics.

davidtwco force-pushed the sve-intrinsics branch from b0b5ef1 to 003827f Compare March 18, 2026 11:21

Uh oh!

Conversation

davidtwco commented Mar 2, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rustbot commented Mar 2, 2026

Uh oh!

rustbot commented Mar 2, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

rustbot commented Mar 2, 2026

Uh oh!

This comment was marked as resolved.

rustbot commented Mar 2, 2026

Uh oh!

This comment was marked as resolved.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment was marked as resolved.

This comment has been minimized.

lqd commented Mar 10, 2026

Uh oh!

davidtwco commented Mar 10, 2026

Uh oh!

This comment was marked as resolved.

This comment has been minimized.

workingjubilee left a comment • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rustbot commented Mar 17, 2026

Uh oh!

workingjubilee left a comment • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

workingjubilee Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

davidtwco Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

workingjubilee Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

davidtwco Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

workingjubilee Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

workingjubilee Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

davidtwco Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

davidtwco commented Mar 2, 2026 •

edited by rustbot

Loading

workingjubilee left a comment •

edited by rustbot

Loading

workingjubilee left a comment •

edited by rustbot

Loading