Puppeteer doesn't care what it renders. page.goto() resolves, page.pdf() produces bytes, your job reports success. If you screenshot, archive, monitor or scrape pages at any volume, some of those pages are an interstitial from Cloudflare, Akamai, Imperva or the site's own nginx — and nothing in the default pipeline notices.

This is the detector we ended up with, and more usefully, the cases that slipped past each earlier version.

Rule 1: an HTTP status alone proves nothing

The obvious first version is if (response.status() >= 400) fail(). It is wrong in both directions:

• Plenty of real pages are served under 403 or 429 with full content behind them. Fail those and you refuse captures that would have been fine.
• Many block pages come back as 200. Imperva/Incapsula, for one, answers with a 200 page whose only content is a line inside an iframe.

So the rule we settled on: a status must be corroborated by the page actually looking like an interstitial. Signatures are matched against the interstitial's wording, never the site's:

const BLOCK_SIGNATURES = [
  /sorry,?\s+you\s+have\s+been\s+blocked/i,
  /attention\s+required!?\s*\|\s*cloudflare/i,
  /just\s+a\s+moment\s*(\.\.\.|…)/i,            // Cloudflare managed challenge
  /checking\s+your\s+browser\s+before\s+accessing/i,
  /enable\s+javascript\s+and\s+cookies\s+to\s+continue/i,
  /verify\s+you\s+are\s+(a\s+)?human/i,
  /access\s+denied.{0,40}reference\s*#/i,        // Akamai
  /request\s+blocked.{0,40}request\s*id/i,       // AWS WAF
  /pardon\s+(the\s+)?interruption/i,             // PerimeterX / HUMAN
  /vercel\s+security\s+checkpoint/i,
  /we'?re\s+verifying\s+your\s+browser/i,        // Vercel, see below
  /request\s+unsuccessful\.?\s+incapsula\s+incident/i,
  /_Incapsula_Resource/i,
];

const BLOCK_STATUSES = [401, 403, 406, 429, 451, 503];

We match against document.title plus the first 4,000 characters of document.body.innerText — the rendered text, not the HTML. Interstitials hide their wording in scripts and iframes often enough that the source is a worse signal than what a human would actually see.

Rule 2: a challenge is not a block — wait it out

Cloudflare's "Just a moment…" is a managed challenge: it is designed to clear by itself once the browser passes the check. Our first version refused at domcontentloaded, which reported perfectly capturable sites as blocked.

We measured it: on a Cloudflare-protected page, modern headless (headless: true) cleared the challenge in under 5 seconds and rendered the real document. The old headless: 'shell' mode never did — still "Just a moment…" at 10 s, blank at 15 s. If you are still on the shell mode and seeing challenges, that alone may be your problem. (So is a default HeadlessChrome/… user agent: without a real one, the same URL gave a hard 403 instead of the softer challenge.)

So the detector returns two kinds of verdict, and only one of them is final:

const CHALLENGE_RE = /just\s+a\s+moment|checking\s+your\s+browser|we'?re\s+verifying\s+your\s+browser|vercel\s+security\s+checkpoint|verify\s+you\s+are\s+(a\s+)?human/i;

async function waitOutChallenge(page, verdict, { timeoutMs = 15000, stepMs = 1000 } = {}) {
  const deadline = Date.now() + timeoutMs;
  while (Date.now() < deadline) {
    await new Promise(r => setTimeout(r, stepMs));
    let state;
    try { state = await readPageState(page); } catch { return verdict; }

    // No new response object here, so judge on content alone.
    const again = detectBlock({ status: 200, title: state.title, text: state.text });
    if (!again) {
      // Clearing a challenge navigates; for a moment there is no document.body.
      await page.waitForFunction(
        () => document.body && document.readyState !== 'loading',
        { timeout: 10000 }
      ).catch(() => {});
      return null;                      // through — the real page is up
    }
    if (again.kind === 'blocked') return again;   // escalated, stop waiting
  }
  return verdict;
}

The waitForFunction at the end matters more than it looks. The moment the challenge clears, the browser navigates to the real page, and for a beat there is no document.body. Every step after this — removing cookie banners, measuring height, screenshotting — reaches straight into it and throws.

Rule 3: catch Chrome's own error page first

This one surprised us. When a server answers with an error status and no usable body, Chrome paints its own page: "This page isn't working — HTTP ERROR 406." Puppeteer happily screenshots that too.

Unlike a CDN interstitial, there is no site content on it at all, so it needs no corroboration beyond the status:

const BROWSER_ERROR_RE = /this\s+page\s+is\s*n.?t\s+working|this\s+site\s+can.?.?t\s+be\s+reached|http\s+error\s+\d{3}|\bERR_[A-Z_]{3,}/i;

It is gated on status >= 400 so an article that merely discusses HTTP errors still gets captured. We found it on scribd.com, which answers document URLs from a datacenter IP with a bare 406.

Next to it sits the empty refusal — a refusal status with almost nothing behind it:

if (BLOCK_STATUSES.includes(status) && blob.replace(/\s+/g, '').length < 40) {
  return { kind: 'blocked', message: `The site refused the request (HTTP ${status}) and sent no page.` };
}

Rule 4: rate limits are their own case

web.archive.org answers our server, when it has had enough, with a bare nginx "429 Too Many Requests" page. It matched none of the CDN signatures, so a user got a one-page PDF of it.

A rate limit deserves a different message: the site works, it is refusing you, for now. Telling the user "this site blocks automated capture" is false; "try again later" is true.

const RATE_LIMIT_RE = /429\s+too\s+many\s+requests|too\s+many\s+requests|rate\s+limit(ed)?\s+exceeded/i;

if (status === 429 || (status >= 400 && RATE_LIMIT_RE.test(blob))) {
  return { kind: 'blocked', rateLimited: true, message: '…usually temporary — please try again later.' };
}

A small confession about that regex: for a while it read 429s+toos+manys+…. The backslashes were lost in an edit, the pattern never matched anything, and only a literal 429 status was caught. A regex that silently matches nothing raises no error. Put your signature list under a test with one real sample per entry.

The order, assembled

function detectBlock({ status = 0, title = '', text = '' }) {
  const blob = `${title}\n${text.slice(0, 4000)}`;

  if (status >= 400 && BROWSER_ERROR_RE.test(blob))                 return blocked('no page');
  if (status === 429 || (status >= 400 && RATE_LIMIT_RE.test(blob))) return blocked('rate limited');
  if (BLOCK_STATUSES.includes(status) && blob.replace(/\s+/g, '').length < 40) return blocked('empty refusal');

  if (!BLOCK_SIGNATURES.some(re => re.test(blob))) return null;  // a real page, even under 403
  if (CHALLENGE_RE.test(blob))                    return challenge();   // wait it out
  return blocked('interstitial');
}

And at the call site, right after navigation and before anything that costs the user something:

const response = await page.goto(url, { waitUntil: 'domcontentloaded', timeout: 30000 });
let verdict = detectBlock(await readPageState(page, response));
if (verdict?.kind === 'challenge') verdict = await waitOutChallenge(page, verdict);
if (verdict) throw new BlockedError(verdict.message);

We also run the same function in the crawler that discovers a site's pages. Otherwise an Imperva page gets listed as "the site's one page," and the user is offered a one-page archive of an error.

The one-word miss

The case that convinced us the list will never be finished: a docs site behind Vercel answered our server with 403 and the line "We're verifying your browser." We had Cloudflare's "checking your browser" and Vercel's "Security Checkpoint" title, but not this wording — one word off. The user got a blank 26 KB archive of the checkpoint.

So treat the signature list as a living log, not a finished design:

1. Log every verdict and every "success" under an error status. Our misses all showed up as captures that worked with a 4xx, or as a tiny output file.
2. Each new signature gets a comment saying where it was found. Six months later, that comment is the only thing stopping someone from "cleaning up" a pattern that looks redundant.
3. Prefer a clear error to a clever bypass. Some sites block automated access on purpose. No amount of tuning changes that, and a useless PDF delivered as a success is worse than an honest "this site refused the request."

I build Site2PDF, which archives web pages and whole sites as PDF, PNG or ZIP. The detector above runs before every capture there.