Export logs through the Cloudflare Logpull API

[jan-dh]

With the rise of AI and the impact on SEO there's an emerging strategy: Server-side Query Fan-Out Tracking.
The idea is to look at your Cloudflare log files to gain insights on when your website was visited by AI bots (in a search) and to gain some insights on how strong your content is for certain topics etc. Interesting read in the details: https://conversem.com/the-query-fan-out-session/

There's already a script available that does a lot of the analysis but it needs the Cloudflare logs; the script can fetch them through the Cloudflare Logpull API. I'm not sure if this would be possible in Craft Cloud?

[timkelty]

Ohh interesting idea.

We plan on providing traffic metrics soon, so this could be a good companion. Putting it in the backlog!

[timkelty]

@jan-dh for your use-case, are you thinking core request details would be adequet (eg path, headres/user-agent) or are you thinking of the Cloudflare hueristic data (eg bot score, ja3 fingerprint)?

[jan-dh]

This is an example of the data we'd need:

{
    "EdgeStartTimestamp": "2025-12-13T00:00:36.625637+00:00",
    "ClientRequestURI": "/docs/latest/integrations",
    "ClientRequestHost": "docs.example.com",
    "ClientRequestUserAgent": "Mozilla/5.0 (compatible; Google-Extended; +https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers)",
    "BotScore": 4,
    "BotScoreSrc": "Machine Learning",
    "VerifiedBot": true,
    "BotTags": [
        "llm"
    ],
    "ClientIP": "10.58.92.172",
    "ClientCountry": "US",
    "EdgeResponseStatus": 200
}

[timkelty]

@jan-dh if you need this before we officially support it: An easy workaround might be to use a (normally free) Cloudflare zone for your DNS, so you can ingest request logs before it proxies on to Craft Cloud.

[jan-dh]

Thanks, I'll pick this up with our ops team to see if this would be an option 🙏

[jamesmacwhite]

I'm going to add another use case for something similar, Microsoft Clarity has AI bot analytics, but requires analysing server logs, I assume for the User-Agent data:

https://clarity.microsoft.com/blog/ai-bot-activity-in-clarity/

Cloudflare Enterprise is listed, but obviously we don't have access to this layer directly. This is using LogPush, it would be great to be able to either allow us to send logs to other destinations in some form via Craft Console or in app.php

[timkelty]

Thanks for the detail, @jamesmacwhite, that helps.