Auto merge of #153489 - aerooneqq:two-phase-hir, r=petrochenkov

ty-aware delayed AST -> HIR lowering



This PR implements a prototype of ty-aware delayed AST -> HIR lowering. Part of #118212.

r? @petrochenkov

# Motivation

When lowering delegation in perfect scenario we would like to access the ty-level information, in particular, queries like `generics_of`, `type_of`, `fn_sig` for proper HIR generation with less hacks. For example, we do not want to generate more lifetimes than needed, because without ty-level queries we do not know which delegee's lifetimes are late-bound. Next, consider recursive delegations, for their proper support without ty we would have to either duplicate generics inheritance code in AST -> HIR lowering or create stubs for parts of the HIR and materialize them later. We already use those queries when interacting with external crates, however when dealing with compilation of a local crate we should use resolver for similar purposes. Finally, access to ty-level queries is expected to help supporting delegation to inherent impls, as we can not resolve such calls during AST -> HIR stage.

# Benefits

We eliminate almost all code that uses resolver in delegation lowering:
- Attributes inheritance is done without copying attributes from AST at resolve stage
- Fn signatures are obtained from `tcx.fn_sig`
- Param counts are also obtained from `tcx.fn_sig`
- `is_method` function now uses `tcx.associated_item` instead of resolver
- Generics are now inherited through `get_external_generics` that uses `tcx.generics_of`. Generics for recursive delegations should also work
- `DelegationIds` that stored paths for recursive delegations is removed, we now use only `delegee_id`
- Structs that were used for storing delegation-related information in resolver are almost fully removed
- `ast_index` is no more used

# Next steps
- Remove creating generic params through AST cloning, proper const types propagation
- Inherent impls

# High level design overview

## Queries
We store ids of delayed items to lower in `Crate` struct. During first stage of lowering, owners that correspond to delayed items are filled with `MaybeOwner::Phantom`. 

Next, we define two new queries: `lower_delayed_owner`, `delayed_owner` and a function `force_delayed_owners_lowering`.

The first query is used when lowering known (which is in `delayed_ids`) delayed owner. 

The second is fed with children that were obtained during lowering of a delayed owner (note that the result of lowering of a single `LocalDefId` is not a single `MaybeOwner`, its a list of `(LocalDefId, MaybeOwner)` where the first `MaybeOwner` corresponds to delayed `LocalDefId` and others to children that were obtained during lowering (i.e. generic params)). By default `delayed_owner` returns `MaybeOwner::Phantom`. As we do not want to predict the whole list of children which will be obtained after lowering of a single delayed item, we need to store those children somewhere. There are several options:
- Make the return type of `lower_delayed_owner` to be `FxIndexMap<LocalDefId, MaybeOwner>` and search children here. Search will be either linear or we can introduce a map somewhere which will track parent-child relations between a single delayed `LocalDefId` and its children. 
- Try to make query that will lower all delayed items in a loop and return a complete map of all delayed `LocalDefIds` and their children. In this case there will be problems with delayed items that require information about other delayed items.

By using proposed queries we handle the second concern, and in case of acyclic dependencies between delayed ids it automatically works, moreover we use queries as cache for delayed `MaybeOwners`. The only invariant here is that queries which are invoked during delayed AST -> HIR lowering of some `LocalDefId` should not in any way access children of other, yet unlowered, delayed `LocalDefId`, they should firstly materialize it.

The `force_delayed_owners_lowering` forces lowering of all delayed items and now integrated in `hir_crate_items` query. 

## Resolver for lowering

> ~Currently the `resolver_for_lowering` is stolen in `lower_to_hir` function, however we want to prolong its life until all delayed `LocalDefIds` are materialized. For this purpose we borrow `resolver_for_lowering` in `lower_to_hir` and drop it after forcing materialization of all delayed `LocalDefId` in `rustc_interface::run_required_analyses`.~

We split resolver for lowering into two parts: the first part is a readonly part that came to us from resolve, the second part is a mutable part that can be used to add or overwrite values in the readonly part. Such splitted resolver is used during delayed lowering, as we can't steal it.

## AST index

Lowering uses an AST index. It is created in `lower_to_hir` function and it references parts of AST. We want to avoid reindexing AST on every delayed `LocalDefId` lowering, however now it is not clear how to properly do it. As delayed HIR lowering is used only for delegation unstable feature it should not affect other use-cases of the compiler. But it will be reworked sooner or later.

Author: bors

compiler/rustc_ast_lowering/src/delegation.rs

@@ -45,16 +45,15 @@ use hir::{BodyId, HirId};
 use rustc_abi::ExternAbi;
 use rustc_ast as ast;
 use rustc_ast::*;
-use rustc_attr_parsing::{AttributeParser, ShouldEmit};
 use rustc_data_structures::fx::FxHashSet;
 use rustc_errors::ErrorGuaranteed;
 use rustc_hir as hir;
 use rustc_hir::attrs::{AttributeKind, InlineAttr};
-use rustc_hir::def_id::{DefId, LocalDefId};
+use rustc_hir::def_id::DefId;
 use rustc_middle::span_bug;
-use rustc_middle::ty::{Asyncness, DelegationAttrs, DelegationFnSigAttrs};
+use rustc_middle::ty::Asyncness;
 use rustc_span::symbol::kw;
-use rustc_span::{DUMMY_SP, Ident, Span, Symbol};
+use rustc_span::{Ident, Span, Symbol};
 use smallvec::SmallVec;
 
 use crate::delegation::generics::{GenericsGenerationResult, GenericsGenerationResults};
@@ -80,7 +79,7 @@ struct AttrAdditionInfo {
 
 enum AttrAdditionKind {
     Default { factory: fn(Span) -> hir::Attribute },
-    Inherit { flag: DelegationFnSigAttrs, factory: fn(Span, &hir::Attribute) -> hir::Attribute },
+    Inherit { factory: fn(Span, &hir::Attribute) -> hir::Attribute },
 }
 
 const PARENT_ID: hir::ItemLocalId = hir::ItemLocalId::ZERO;
@@ -97,7 +96,6 @@ static ATTRS_ADDITIONS: &[AttrAdditionInfo] = &[
 
                 hir::Attribute::Parsed(AttributeKind::MustUse { span, reason })
             },
-            flag: DelegationFnSigAttrs::MUST_USE,
         },
     },
     AttrAdditionInfo {
@@ -108,43 +106,11 @@ static ATTRS_ADDITIONS: &[AttrAdditionInfo] = &[
     },
 ];
 
-type DelegationIdsVec = SmallVec<[DefId; 1]>;
-
-// As delegations can now refer to another delegation, we have a delegation path
-// of the following type: reuse (current delegation) <- reuse (delegee_id) <- ... <- reuse <- function (root_function_id).
-// In its most basic and widely used form: reuse (current delegation) <- function (delegee_id, root_function_id)
-struct DelegationIds {
-    path: DelegationIdsVec,
-}
-
-impl DelegationIds {
-    fn new(path: DelegationIdsVec) -> Self {
-        assert!(!path.is_empty());
-        Self { path }
-    }
-
-    // Id of the first function in (non)local crate that is being reused
-    fn root_function_id(&self) -> DefId {
-        *self.path.last().expect("Ids vector can't be empty")
-    }
-
-    // Id of the first definition which is being reused,
-    // can be either function, in this case `root_id == delegee_id`, or other delegation
-    fn delegee_id(&self) -> DefId {
-        *self.path.first().expect("Ids vector can't be empty")
-    }
-}
-
 impl<'hir, R: ResolverAstLoweringExt<'hir>> LoweringContext<'_, 'hir, R> {
     fn is_method(&self, def_id: DefId, span: Span) -> bool {
         match self.tcx.def_kind(def_id) {
             DefKind::Fn => false,
-            DefKind::AssocFn => match def_id.as_local() {
-                Some(local_def_id) => {
-                    self.resolver.delegation_fn_sig(local_def_id).is_some_and(|sig| sig.has_self)
-                }
-                None => self.tcx.associated_item(def_id).is_method(),
-            },
+            DefKind::AssocFn => self.tcx.associated_item(def_id).is_method(),
             _ => span_bug!(span, "unexpected DefKind for delegation item"),
         }
     }
@@ -157,10 +123,10 @@ impl<'hir, R: ResolverAstLoweringExt<'hir>> LoweringContext<'_, 'hir, R> {
         let span = self.lower_span(delegation.path.segments.last().unwrap().ident.span);
 
         // Delegation can be unresolved in illegal places such as function bodies in extern blocks (see #151356)
-        let ids = if let Some(delegation_info) =
+        let sig_id = if let Some(delegation_info) =
             self.resolver.delegation_info(self.local_def_id(item_id))
         {
-            self.get_delegation_ids(delegation_info.resolution_node, span)
+            self.get_sig_id(delegation_info.resolution_node, span)
         } else {
             return self.generate_delegation_error(
                 self.dcx().span_delayed_bug(
@@ -172,28 +138,16 @@ impl<'hir, R: ResolverAstLoweringExt<'hir>> LoweringContext<'_, 'hir, R> {
             );
         };
 
-        match ids {
-            Ok(ids) => {
-                self.add_attrs_if_needed(span, &ids);
-
-                let delegee_id = ids.delegee_id();
-                let root_function_id = ids.root_function_id();
+        match sig_id {
+            Ok(sig_id) => {
+                self.add_attrs_if_needed(span, sig_id);
 
-                // `is_method` is used to choose the name of the first parameter (`self` or `arg0`),
-                // if the original function is not a method (without `self`), then it can not be added
-                // during chain of reuses, so we use `root_function_id` here
-                let is_method = self.is_method(root_function_id, span);
+                let is_method = self.is_method(sig_id, span);
 
-                // Here we use `root_function_id` as we can not get params information out of potential delegation reuse,
-                // we need a function to extract this information
-                let (param_count, c_variadic) = self.param_count(root_function_id);
+                let (param_count, c_variadic) = self.param_count(sig_id);
 
-                let mut generics = self.lower_delegation_generics(
-                    delegation,
-                    ids.root_function_id(),
-                    item_id,
-                    span,
-                );
+                let mut generics =
+                    self.lower_delegation_generics(delegation, sig_id, item_id, span);
 
                 let body_id = self.lower_delegation_body(
                     delegation,
@@ -204,20 +158,10 @@ impl<'hir, R: ResolverAstLoweringExt<'hir>> LoweringContext<'_, 'hir, R> {
                     span,
                 );
 
-                // Here we use `delegee_id`, as this id will then be used to calculate parent for generics
-                // inheritance, and we want this id to point on a delegee, not on the original
-                // function (see https://github.com/rust-lang/rust/issues/150152#issuecomment-3674834654)
-                let decl = self.lower_delegation_decl(
-                    delegee_id,
-                    param_count,
-                    c_variadic,
-                    span,
-                    &generics,
-                );
+                let decl =
+                    self.lower_delegation_decl(sig_id, param_count, c_variadic, span, &generics);
 
-                // Here we pass `root_function_id` as we want to inherit signature (including consts, async)
-                // from the root function that started delegation
-                let sig = self.lower_delegation_sig(root_function_id, decl, span);
+                let sig = self.lower_delegation_sig(sig_id, decl, span);
                 let ident = self.lower_ident(delegation.ident);
 
                 let generics = self.arena.alloc(hir::Generics {
@@ -236,9 +180,9 @@ impl<'hir, R: ResolverAstLoweringExt<'hir>> LoweringContext<'_, 'hir, R> {
         }
     }
 
-    fn add_attrs_if_needed(&mut self, span: Span, ids: &DelegationIds) {
+    fn add_attrs_if_needed(&mut self, span: Span, sig_id: DefId) {
         let new_attrs =
-            self.create_new_attrs(ATTRS_ADDITIONS, span, ids, self.attrs.get(&PARENT_ID));
+            self.create_new_attrs(ATTRS_ADDITIONS, span, sig_id, self.attrs.get(&PARENT_ID));
 
         if new_attrs.is_empty() {
             return;
@@ -258,15 +202,9 @@ impl<'hir, R: ResolverAstLoweringExt<'hir>> LoweringContext<'_, 'hir, R> {
         &self,
         candidate_additions: &[AttrAdditionInfo],
         span: Span,
-        ids: &DelegationIds,
+        sig_id: DefId,
         existing_attrs: Option<&&[hir::Attribute]>,
     ) -> Vec<hir::Attribute> {
-        let defs_orig_attrs = ids
-            .path
-            .iter()
-            .map(|def_id| (*def_id, self.parse_local_original_attrs(*def_id)))
-            .collect::<Vec<_>>();
-
         candidate_additions
             .iter()
             .filter_map(|addition_info| {
@@ -280,83 +218,22 @@ impl<'hir, R: ResolverAstLoweringExt<'hir>> LoweringContext<'_, 'hir, R> {
 
                 match addition_info.kind {
                     AttrAdditionKind::Default { factory } => Some(factory(span)),
-                    AttrAdditionKind::Inherit { flag, factory } => {
-                        for (def_id, orig_attrs) in &defs_orig_attrs {
-                            let original_attr = match def_id.as_local() {
-                                Some(local_id) => self
-                                    .get_attrs(local_id)
-                                    .flags
-                                    .contains(flag)
-                                    .then(|| {
-                                        orig_attrs
-                                            .as_ref()
-                                            .map(|attrs| {
-                                                attrs.iter().find(|base_attr| {
-                                                    (addition_info.equals)(base_attr)
-                                                })
-                                            })
-                                            .flatten()
-                                    })
-                                    .flatten(),
-                                None =>
-                                {
-                                    #[allow(deprecated)]
-                                    self.tcx
-                                        .get_all_attrs(*def_id)
-                                        .iter()
-                                        .find(|base_attr| (addition_info.equals)(base_attr))
-                                }
-                            };
-
-                            if let Some(original_attr) = original_attr {
-                                return Some(factory(span, original_attr));
-                            }
-                        }
-
-                        None
+                    AttrAdditionKind::Inherit { factory, .. } =>
+                    {
+                        #[allow(deprecated)]
+                        self.tcx
+                            .get_all_attrs(sig_id)
+                            .iter()
+                            .find_map(|a| (addition_info.equals)(a).then(|| factory(span, a)))
                     }
                 }
             })
             .collect::<Vec<_>>()
     }
 
-    fn parse_local_original_attrs(&self, def_id: DefId) -> Option<Vec<hir::Attribute>> {
-        if let Some(local_id) = def_id.as_local() {
-            let attrs = &self.get_attrs(local_id).to_inherit;
-
-            if !attrs.is_empty() {
-                return Some(AttributeParser::parse_limited_all(
-                    self.tcx.sess,
-                    attrs,
-                    None,
-                    hir::Target::Fn,
-                    DUMMY_SP,
-                    DUMMY_NODE_ID,
-                    Some(self.tcx.features()),
-                    ShouldEmit::Nothing,
-                ));
-            }
-        }
-
-        None
-    }
-
-    fn get_attrs(&self, local_id: LocalDefId) -> &DelegationAttrs {
-        // local_id can correspond either to a function or other delegation
-        if let Some(fn_sig) = self.resolver.delegation_fn_sig(local_id) {
-            &fn_sig.attrs
-        } else {
-            &self.resolver.delegation_info(local_id).expect("processing delegation").attrs
-        }
-    }
-
-    fn get_delegation_ids(
-        &self,
-        mut node_id: NodeId,
-        span: Span,
-    ) -> Result<DelegationIds, ErrorGuaranteed> {
+    fn get_sig_id(&self, mut node_id: NodeId, span: Span) -> Result<DefId, ErrorGuaranteed> {
         let mut visited: FxHashSet<NodeId> = Default::default();
-        let mut path: DelegationIdsVec = Default::default();
+        let mut path: SmallVec<[DefId; 1]> = Default::default();
 
         loop {
             visited.insert(node_id);
@@ -389,7 +266,7 @@ impl<'hir, R: ResolverAstLoweringExt<'hir>> LoweringContext<'_, 'hir, R> {
                     });
                 }
             } else {
-                return Ok(DelegationIds::new(path));
+                return Ok(path[0]);
             }
         }
     }
@@ -400,15 +277,8 @@ impl<'hir, R: ResolverAstLoweringExt<'hir>> LoweringContext<'_, 'hir, R> {
 
     // Function parameter count, including C variadic `...` if present.
     fn param_count(&self, def_id: DefId) -> (usize, bool /*c_variadic*/) {
-        if let Some(local_sig_id) = def_id.as_local() {
-            match self.resolver.delegation_fn_sig(local_sig_id) {
-                Some(sig) => (sig.param_count, sig.c_variadic),
-                None => (0, false),
-            }
-        } else {
-            let sig = self.tcx.fn_sig(def_id).skip_binder().skip_binder();
-            (sig.inputs().len() + usize::from(sig.c_variadic), sig.c_variadic)
-        }
+        let sig = self.tcx.fn_sig(def_id).skip_binder().skip_binder();
+        (sig.inputs().len() + usize::from(sig.c_variadic), sig.c_variadic)
     }
 
     fn lower_delegation_decl(
@@ -455,41 +325,21 @@ impl<'hir, R: ResolverAstLoweringExt<'hir>> LoweringContext<'_, 'hir, R> {
         decl: &'hir hir::FnDecl<'hir>,
         span: Span,
     ) -> hir::FnSig<'hir> {
-        let header = if let Some(local_sig_id) = sig_id.as_local() {
-            match self.resolver.delegation_fn_sig(local_sig_id) {
-                Some(sig) => {
-                    let parent = self.tcx.parent(sig_id);
-                    // HACK: we override the default safety instead of generating attributes from the ether.
-                    // We are not forwarding the attributes, as the delegation fn sigs are collected on the ast,
-                    // and here we need the hir attributes.
-                    let default_safety =
-                        if sig.attrs.flags.contains(DelegationFnSigAttrs::TARGET_FEATURE)
-                            || self.tcx.def_kind(parent) == DefKind::ForeignMod
-                        {
-                            hir::Safety::Unsafe
-                        } else {
-                            hir::Safety::Safe
-                        };
-                    self.lower_fn_header(sig.header, default_safety, &[])
-                }
-                None => self.generate_header_error(),
-            }
-        } else {
-            let sig = self.tcx.fn_sig(sig_id).skip_binder().skip_binder();
-            let asyncness = match self.tcx.asyncness(sig_id) {
-                Asyncness::Yes => hir::IsAsync::Async(span),
-                Asyncness::No => hir::IsAsync::NotAsync,
-            };
-            hir::FnHeader {
-                safety: if self.tcx.codegen_fn_attrs(sig_id).safe_target_features {
-                    hir::HeaderSafety::SafeTargetFeatures
-                } else {
-                    hir::HeaderSafety::Normal(sig.safety)
-                },
-                constness: self.tcx.constness(sig_id),
-                asyncness,
-                abi: sig.abi,
-            }
+        let sig = self.tcx.fn_sig(sig_id).skip_binder().skip_binder();
+        let asyncness = match self.tcx.asyncness(sig_id) {
+            Asyncness::Yes => hir::IsAsync::Async(span),
+            Asyncness::No => hir::IsAsync::NotAsync,
+        };
+
+        let header = hir::FnHeader {
+            safety: if self.tcx.codegen_fn_attrs(sig_id).safe_target_features {
+                hir::HeaderSafety::SafeTargetFeatures
+            } else {
+                hir::HeaderSafety::Normal(sig.safety)
+            },
+            constness: self.tcx.constness(sig_id),
+            asyncness,
+            abi: sig.abi,
         };
 
         hir::FnSig { decl, header, span }