diff --git a/src/content/changelog/ai-crawl-control/2025-12-05-pay-per-crawl-custom-terms.mdx b/src/content/changelog/ai-crawl-control/2025-12-05-pay-per-crawl-custom-terms.mdx new file mode 100644 index 000000000000000..1703c1ff6d2ccbc --- /dev/null +++ b/src/content/changelog/ai-crawl-control/2025-12-05-pay-per-crawl-custom-terms.mdx @@ -0,0 +1,17 @@ +--- +title: Custom terms URL for Pay Per Crawl +description: Site owners can now provide a custom terms URL that will be included in HTTP responses to crawlers. +date: 2025-12-05 +--- + +:::note[Pay per crawl closed beta] +Pay per crawl is currently in closed beta. + +To find out how to join the beta program, reach out to us at [Pay per crawl signup](http://www.cloudflare.com/paypercrawl-signup/), or contact your account executive if you are an existing Enterprise customer. + +To learn more about pay per crawl, refer to Cloudflare blog: [Introducing pay per crawl: enabling content owners to charge AI crawlers for access](https://blog.cloudflare.com/introducing-pay-per-crawl/). +::: + +Site owners using [Pay Per Crawl](/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/) can now optionally configure a **Terms URL** in **AI Crawl Control** > **Settings**. The URL is included in the `Link` header of HTTP 200 and 402 responses served to crawlers, allowing you to provide custom terms and conditions for crawler access to your paid content. + +Learn more about [setting a pay per crawl Terms URL](/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/set-a-pay-per-crawl-price/). diff --git a/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/crawl-pages.mdx b/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/crawl-pages.mdx index 0ff55e47a27f00a..d450dbbcd47885e 100644 --- a/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/crawl-pages.mdx +++ b/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/crawl-pages.mdx @@ -23,7 +23,6 @@ When an AI crawler makes a request to a page protected by pay per crawl, the ser ```txt HTTP/2 402 -date: Fri, 06 Jun 2025 08:42:38 GMT crawler-price: USD 0.01 ``` @@ -56,7 +55,6 @@ The value of the `crawler-charged` header indicates the exact amount that will b ```txt HTTP/2 200 -date: Fri, 06 Jun 2025 08:42:38 GMT crawler-charged: USD 0.01 ``` @@ -66,11 +64,16 @@ If the request is unsuccessful due to an incorrect price header, you will receiv ```txt HTTP/2 402 -date: Fri, 06 Jun 2025 08:42:38 GMT content-type: text/plain; charset=utf-8 crawler-price: USD 0.01 ``` +### 2.4. Check for custom terms + +Before using paid content, check if the response includes a `Link` header. If present, the site owner has provided custom terms and conditions that supersede the default Pay Per Crawl agreement. Review these terms before proceeding with payment. + +This header may be present in both HTTP 200 and 402 responses. + ## 3. Track your spending When you successfully access pay per crawl content (response with HTTP status code 200), the response will include `crawler-charged`. For example: diff --git a/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/set-a-pay-per-crawl-price.mdx b/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/set-a-pay-per-crawl-price.mdx index fcaab09910860bd..17634f74a9e6df8 100644 --- a/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/set-a-pay-per-crawl-price.mdx +++ b/src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/set-a-pay-per-crawl-price.mdx @@ -21,22 +21,34 @@ click D "/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owne click E "/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/manage-payouts/" ``` -Once your domain's visibility is set to **Visible** in Account Settings, you can set a pay per crawl price and enable pay per crawl for that domain. +Once your domain's visibility is set to **Visible** in Account Settings, you can set a pay per crawl price, configure a Terms URL, and enable pay per crawl for that domain. {/* prettier-ignore */} 1. Go to **AI Crawl Control**. - + 2. Go to the **Settings** tab. 3. In the **Pay Per Crawl** card, select **Enable**. 4. Set your per crawl price - this is the amount charged for each successful content retrieval (HTTP 200 response) by an AI crawler. -5. Select **Save**. +5. (Optional) Set a **Terms URL** - a valid URL to custom terms and conditions that will be included in the `Link` header of HTTP 200 and 402 responses to crawlers. +6. Select **Save**. -After enabling and setting a price, the domain's status in Account Settings will change to **Enabled**. +## Crawl price + +The crawl price is the amount charged for each successful content retrieval (HTTP 200 response) by an AI crawler that is set to **Charge**. :::note[Pricing considerations] The minimum price is $0.01 USD per crawl. Consider your content value and expected crawler volume when setting your price. ::: + +After setting a crawl price the first time, the domain's status in Account Settings will update to **Enabled**. + +## Terms URL (optional) + +You can optionally provide a **Terms URL** that points to custom terms and conditions for crawlers paying to access to your content. When configured: + +- The URL is included in the `Link` header of HTTP 200 and 402 responses served to crawlers by Pay Per Crawl. +- Crawler operators are responsible for reviewing your custom terms before using your content.