Pay Per Crawl: Add documentation and changelog for Terms URL feature

src/content/changelog/ai-crawl-control/2025-12-05-pay-per-crawl-custom-terms.mdx

@@ -0,0 +1,17 @@
+---
+title: Custom terms URL for Pay Per Crawl
+description: Site owners can now provide a custom terms URL that will be included in HTTP responses to crawlers.
+date: 2025-12-05
+---
+
+:::note[Pay per crawl closed beta]
+Pay per crawl is currently in closed beta.
+
+To find out how to join the beta program, reach out to us at [Pay per crawl signup](http://www.cloudflare.com/paypercrawl-signup/), or contact your account executive if you are an existing Enterprise customer.
+
+To learn more about pay per crawl, refer to Cloudflare blog: [Introducing pay per crawl: enabling content owners to charge AI crawlers for access](https://blog.cloudflare.com/introducing-pay-per-crawl/).
+:::
+
+Site owners using [Pay Per Crawl](/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/) can now optionally configure a **Terms URL** in **AI Crawl Control** > **Settings**. The URL is included in the `Link` header of HTTP 200 and 402 responses served to crawlers, allowing you to provide custom terms and conditions for crawler access to your paid content.
+
+Learn more about [setting a pay per crawl Terms URL](/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/set-a-pay-per-crawl-price/).

src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/crawl-pages.mdx

@@ -23,7 +23,6 @@ When an AI crawler makes a request to a page protected by pay per crawl, the ser
 
 ```txt
 HTTP/2 402
-date: Fri, 06 Jun 2025 08:42:38 GMT
 crawler-price: USD 0.01
 ```
 
@@ -56,7 +55,6 @@ The value of the `crawler-charged` header indicates the exact amount that will b
 
 ```txt
 HTTP/2 200
-date: Fri, 06 Jun 2025 08:42:38 GMT
 crawler-charged: USD 0.01
 ```
 
@@ -66,11 +64,16 @@ If the request is unsuccessful due to an incorrect price header, you will receiv
 
 ```txt
 HTTP/2 402
-date: Fri, 06 Jun 2025 08:42:38 GMT
 content-type: text/plain; charset=utf-8
 crawler-price: USD 0.01
 ```
 
+### 2.4. Check for custom terms
+
+Before using paid content, check if the response includes a `Link` header. If present, the site owner has provided custom terms and conditions that supersede the default Pay Per Crawl agreement. Review these terms before proceeding with payment.
+
+This header may be present in both HTTP 200 and 402 responses.
+
 ## 3. Track your spending
 
 When you successfully access pay per crawl content (response with HTTP status code 200), the response will include `crawler-charged`. For example:

src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/set-a-pay-per-crawl-price.mdx

@@ -21,22 +21,34 @@ click D "/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owne
 click E "/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/manage-payouts/"
 ```
 
-Once your domain's visibility is set to **Visible** in Account Settings, you can set a pay per crawl price and enable pay per crawl for that domain.
+Once your domain's visibility is set to **Visible** in Account Settings, you can set a pay per crawl price, configure a Terms URL, and enable pay per crawl for that domain.
 
 {/* prettier-ignore */}
 <Steps>
 1. Go to **AI Crawl Control**.
 
-   <DashButton url="/?to=/:account/:zone/ai" />
+    <DashButton url="/?to=/:account/:zone/ai" />
 
 2. Go to the **Settings** tab.
 3. In the **Pay Per Crawl** card, select **Enable**.
 4. Set your per crawl price - this is the amount charged for each successful content retrieval (HTTP 200 response) by an AI crawler.
-5. Select **Save**.
+5. (Optional) Set a **Terms URL** - a valid URL to custom terms and conditions that will be included in the `Link` header of HTTP 200 and 402 responses to crawlers.
+6. Select **Save**.
 </Steps>
 
-After enabling and setting a price, the domain's status in Account Settings will change to **Enabled**.
+## Crawl price
+
+The crawl price is the amount charged for each successful content retrieval (HTTP 200 response) by an AI crawler that is set to **Charge**.
 
 :::note[Pricing considerations]
 The minimum price is $0.01 USD per crawl. Consider your content value and expected crawler volume when setting your price.
 :::
+
+After setting a crawl price the first time, the domain's status in Account Settings will update to **Enabled**.
+
+## Terms URL (optional)
+
+You can optionally provide a **Terms URL** that points to custom terms and conditions for crawlers paying to access to your content. When configured:
+
+- The URL is included in the `Link` header of HTTP 200 and 402 responses served to crawlers by Pay Per Crawl.
+- Crawler operators are responsible for reviewing your custom terms before using your content.