v4 - `AdaptivePlaywrightCrawler` doesn't process any requests

[barjin]

There seems to be a regression in v4, where AdaptivePlaywrightCrawler doesn't process the provided requests.

import { AdaptivePlaywrightCrawler } from './packages/playwright-crawler/dist/internals/adaptive-playwright-crawler.js';

const crawler = new AdaptivePlaywrightCrawler({
    requestHandler: ({ request, log }) => {
        log.info(`Processing ${request.loadedUrl}...`);
    },
});

await crawler.run(['https://example.com']);

Expected behaviour: the crawler prints "Processing example.com".

Current behaviour:

[harryautomazione]

Hi everyone,

I was looking into this issue and tried to reproduce it locally. I wanted to share my findings in case they can help resolve this regression.

🔬 Diagnosis

It seems there is a small logic conflict in the AdaptivePlaywrightCrawler early-return mechanism when running in HTTP-only (static) mode.

Inside runRequestHandlerWithPlainHTTP, if a request is skipped (for example due to an EnqueueStrategy mismatch during redirects), it sets request.state = RequestState.SKIPPED and returns undefined:

if (!this.requestMatchesEnqueueStrategy(crawlingContext.request)) {
    request.state = RequestState.SKIPPED;
    return; // returns undefined
}

Then, in _runRequestHandler, the early-return check evaluates if it can skip the browser fallback:

const plainHTTPRun = await this.runRequestHandlerWithPlainHTTP(crawlingContext);

if (plainHTTPRun.ok && this.resultChecker(plainHTTPRun.result)) {
    return;
}

By default, without a custom resultChecker, it falls back to () => true. Because of this, this.resultChecker(undefined) returns true. The crawler treats the HTTP run as successful, increments statistics, and skips falling back to the browser, so the requestHandler is never executed.

---

🛠️ Suggested Fix

To avoid this silent skip, I experimented locally with updates to packages/playwright-crawler/src/internals/adaptive-playwright-crawler.ts:

1. Exclude SKIPPED requests from the early-return validation.
2. Reset request.state to RequestState.UNPROCESSED before falling back to the browser, so the skipped state does not block downstream layers.

Proposed Diff:

@@ -362,7 +362,7 @@
 
                 const plainHTTPRun = await this.runRequestHandlerWithPlainHTTP(crawlingContext);
 
-                if (plainHTTPRun.ok && this.resultChecker(plainHTTPRun.result)) {
+                if (plainHTTPRun.ok && crawlingContext.request.state !== RequestState.SKIPPED && this.resultChecker(plainHTTPRun.result)) {
                     crawlingContext.log.debug(`HTTP-only request handler succeeded for ${crawlingContext.request.url}`);
@@ -378,6 +378,11 @@
                 }
             }
 
+            if (crawlingContext.request.state === RequestState.SKIPPED) {
+                crawlingContext.request.state = RequestState.UNPROCESSED;
+                crawlingContext.request.noRetry = false;
+            }
+
             crawlingContext.log.debug(`Running browser request handler for ${crawlingContext.request.url}`);

With these changes, a small local reproduction script that previously resulted in a skipped request correctly falls back to the browser and executes the user handler.

Let me know if this diagnosis makes sense or if I missed anything!