Stabilize #[my_macro] mod foo; (part of proc_macro_hygiene)

[TimNN]

Stabilization report

Summary

Allow mod foo; (an "outlined" module) in the input of attribute and derive (proc) macros.

Tracking:

• Tracking issue for procedural macros and "hygiene 2.0" #54727
  • The proc_macro_hygiene feature covers multiple related features. This PR proposes only a partial stabilization

Reference PRs:

• Document modules in attribute & derive macro input reference#2284

What is stabilized

Describe each behavior being stabilized and give a short example of code that will now be accepted.

Currently, mod foo; (an outlined / non-inline) module is explicitly rejected in the input of derive and attribute (proc) macros.

This PR removes the check, allowing mod foo;.

The module is provided to the macro as it is written in the source code, that is in its "not-yet-loaded" form without a body.

// The following was previously rejected and will now compile.

#[my_proc_macro]
mod foo;

#[my_proc_macro]
fn bar() {
    #[path = "foo.rs"]
    mod foo;
}

#[derive(MyTrait)]
struct Foo([u8; {
    mod foo;
    0
]);

What isn't stabilized

Describe any parts of the feature not being stabilized. Talk about what we might want to do later and what doors are being left open for that. If what we're not stabilizing might lead to surprises for users, talk about that in particular.

The remaining parts of proc_macro_hygiene are not stabilized (specifically, attribute macros in places where they aren't currently allowed).

Design

Reference

What updates are needed to the Reference? Link to each PR. If the Reference is missing content needed for describing this feature, discuss that.

rust-lang/reference#2284

RFC history

What RFCs have been accepted for this feature?

This feature has not been explicitly specified by any RFC.

• Ahead of the proc_macro stabilization, #[my_macro] mod foo; specifically was feature gated.
• As a side effect of Track whether module declarations are inline (fixes #12590) #52319, the (unstable) behavior changed:
  • Before that PR, mod foo; was actually provided to macros as the loaded mod foo { ... }
  • After that PR, mod foo; was provided to macros as-is.
• In 2019 (after the proc_macro stabilization, expand: Feature gate out-of-line modules in proc macro input #66078 made mod foo anywhere in the input to derive or attribute proc macros unstable.
  • Before that PR, only proc macros directly applied to mod foo; were unstable.

Answers to unresolved questions

What questions were left unresolved by the RFC? How have they been answered? Link to any relevant lang decisions.

N/A

Post-RFC changes

What other user-visible changes have occurred since the RFC was accepted? Describe both changes that the lang team accepted (and link to those decisions) as well as changes that are being presented to the team for the first time in this stabilization report.

See the RFC history above.

Key points

What decisions have been most difficult and what behaviors to be stabilized have proved most contentious? Summarize the major arguments on all sides and link to earlier documents and discussions.

The main design decisio is whether outlined modules should be passed to proc macros with or without their body.

Arguments in favor of "without their body":

• (To the best of my knowledge) the current compiler architecture makes "without their body" the natural choice.
  • To quote @petrochenkov, "it's clear that the current behavior of #[my_macro] mod foo; (the "not-yet-loaded" version of the module is passed to the macro) is natural to the current setup, stable and is not going to change".
• At least I would be somewhat surprised to see a macro (which is operating on the AST) able to see or affect code outside the current file. I would assume by default that they get as input exactly what I have written.
• Function-like procedural macros operate on arbitrary token trees. mod foo; in their input is not even recognized by the compiler and thus not expanded. Attribute and derive proc macros also receiving the unexpanded mod foo; in their input makes the two consistent.
• Other syntactic constructs, e.g. other macros like inlcude!("foo.rs"), are also not expanded before being passed to proc macros. Providing modules without their body would be consistent with that.
• Providing mod foo; to proc macros with its body would (as far as I know) be the only case where a proc macro would not get exactly what is written in the source code as its input.

Arguments in favor of "with their body":

• #[my_macro] mod foo; and #[my_macro] mod foo {} would have different behavior, which means that in the presence of proc macros, migrating an inline module to an outlined one would no longer be a purely syntactic change, but could affect the behavior of the proc macro.
• #[my_macro] mod foo; and #![my_macro] in foo.rs (currently unstable) would have different behavior.
  • Other attributes already have different behaviors depending on whether they are applied as inner vs outer attributes. For example #[cfg(false)] on a module as an outer attribute means the corresponding file can be missig or have syntax errors, while using it as an inner attribute requires the file to be syntactically valid.
• Some macros may need access to (or change) the module body to function properly.

Nightly extensions

Are there extensions to this feature that remain unstable? How do we know that we are not accidentally committing to those?

No.

Doors closed

What doors does this stabilization close for later changes to the language? E.g., does this stabilization make any other RFCs, lang experiments, or known in-flight proposals more difficult or impossible to do later?

None.

If, in the future, there is a desire to make the module body of outlined modules available to proc macros, there are multiple ways to add that in a backward-compatible way (e.g. an explicit opt-in during the proc macro registration, or an API in the proc_macro crate).

Feedback

Call for testing

Has a "call for testing" been done? If so, what feedback was received?

No.

Nightly use

Do any known nightly users use this feature? Counting instances of #![feature(FEATURE_NAME)] on GitHub with grep might be informative.

Searching Github produces 4.4k files. Reviewing the first page of results shows many repositories that haven't been updated in multiple years, so it is unclear which part of the proc_macro_hygiene feature they actually use (other parts of the feature were previously stabilized).

Implementation

Major parts

Summarize the major parts of the implementation and provide links into the code and to relevant PRs.

The restriction is currently implemented as an explicit visitor of the proc macro input, rejecting outlined modules. The visitor is removed in this PR.

Coverage

Summarize the test coverage of this feature.

See the test changes attached to this PR. The added tests explicitly verify that proc macros receive mod foo; as an input without its body, to ensure that the behavior doesn't unintentionally change in the future.

Outstanding bugs

What outstanding bugs involve this feature? List them. Should any block the stabilization? Discuss why or why not.

None.

Outstanding FIXMEs

What FIXMEs are still in the code for that feature and why is it OK to leave them there?

None.

Tool changes

What changes must be made to our other tools to support this feature. Has this work been done? Link to any relevant PRs and issues.

Generally none. The feature is a minor addition to the inputs that (proc) macros accept, and should not require special handling in any of the tools.

rust-analyzer has some existing limitations around handling modules transformed by proc macros (e.g. (on stable) auto-complete does not work if a proc macro transforms mod foo {} to mod foo;). It works fine for simple case where the the proc macro leaves the mod foo; mostly unmodified.

Breaking changes

If this stabilization represents a known breaking change, link to the crater report, the analysis of the crater report, and to all PRs we've made to ecosystem projects affected by this breakage. Discuss any limitations of what we're able to know about or to fix.

N/A: The feature is not expected to cause any breakage.

Type system, opsem

N/A: This feature only affects the AST, not type checking.

Common interactions

Temporaries

Does this feature introduce new expressions that can produce temporaries? What are the scopes of those temporaries?

No.

Drop order

Does this feature raise questions about the order in which we should drop values? Talk about the decisions made here and how they're consistent with our earlier decisions.

No.

Pre-expansion / post-expansion

Does this feature raise questions about what should be accepted pre-expansion (e.g. in code covered by #[cfg(false)]) versus what should be accepted post-expansion? What decisions were made about this?

No. (At least nothing that doesn't equally apply to existing stable proc macros).

Edition hygiene

If this feature is gated on an edition, how do we decide, in the context of the edition hygiene of tokens, whether to accept or reject code. E.g., what token do we use to decide?

N/A: Not gated on an edition.

SemVer implications

Does this feature create any new ways in which library authors must take care to prevent breaking downstreams when making minor-version releases? Describe these. Are these new hazards "major" or "minor" according to RFC 1105?

No.

Exposing other features

Are there any other unstable features whose behavior may be exposed by this feature in any way? What features present the highest risk of that?

No.

History

List issues and PRs that are important for understanding how we got here.

See the RFC History section above.

Acknowledgments

Summarize contributors to the feature by name for recognition and so that those people are notified about the stabilization. Does anyone who worked on this not think it should be stabilized right now? We'd like to hear about that if so.

• @alexcrichton for the work towards the initial proc-macro expansion.
• @petrochenkov for significant work on hygiene and expansion since then.

Open items

List any known items that have not yet been completed and that should be before this is stabilized.

None.

---

r? @petrochenkov for an initial review (feel free to edit the stabilization report if you like)

[rustbot]

stdarch is developed in its own repository. If possible, consider making this change to rust-lang/stdarch instead.

cc @Amanieu, @folkertdev, @sayantn

[TimNN]

stdarch is developed in its own repository. If possible, consider making this change to rust-lang/stdarch instead.

Removed the stdarch changes from this PR.