Pay Per Crawl: Add documentation and changelog for Terms URL feature

[CameronWhiteside]

Summary

Added documentation for the new optional Terms URL feature in Pay Per Crawl (private beta). Site owners can now configure a custom terms URL in the AI Crawl Control Settings tab, which will be included in the Link header of HTTP 200 and 402 responses served to crawlers.

Changes:

• Added new section for Site Owners explaining how the Terms URL works and when it's included in responses
• Added new section for AI Owners explaining to check for the presence of the Terms URL before using paid content
• Created changelog entry at 2025-12-04-pay-per-crawl-custom-terms.mdx announcing the feature, with the prefix that PPC is in private beta

Documentation checklist

• Is there a changelog entry (guidelines)? If you don't add one for something awesome and new (however small) — how will our customers find out? Changelogs are automatically posted to RSS feeds, the Discord, and X.
• The change adheres to the documentation style guide.