Solve hCaptcha in Scrapy.

Scrapy has no captcha story of its own: when a target throws an hCaptcha interstitial, your spider just parses gate pages. The fix is small. Read data-sitekey off the gated response, POST it to NoneCap, and resend the request with the returned token as h-captcha-response. This page wires that in two ways: directly in a spider callback, and as a reusable downloader middleware.

Recognizing the gate from a callback

An hCaptcha interstitial is easy to fingerprint in the HTML Scrapy already gives you: a div with class h-captcha carrying a data-sitekey attribute (something like f5ab1c2d-7e8f-4a9b-b1c2-d3e4f5a6b7c8), plus a script tag loading hcaptcha.com/1/api.js. Some sites render the widget without the class, so fall back to any element with data-sitekey:

def hcaptcha_sitekey(response):
    """Return the sitekey if this response is an hCaptcha gate, else None."""
    if b"hcaptcha" not in response.body:
        return None  # cheap pre-check before touching the selector engine
    return (
        response.css(".h-captcha::attr(data-sitekey)").get()
        or response.xpath("//*[@data-sitekey]/@data-sitekey").get()
    )

Two Scrapy-specific traps. First, gates often ship with a 403 or 503 status, and the HttpError spider middleware silently drops non-2xx responses, so your callback never sees them; set handle_httpstatus_list on the spider (or the handle_httpstatus_list key in Request.meta) to let them through. Second, never hard-code the sitekey. Sites rotate them, so read the value fresh from each gated response. The sitekey guide covers the places it hides.

The straightforward pattern: solve it in the spider

Scrapy callbacks are plain sync functions, which makes the official nonecap PyPI package (sync client) a natural fit: detect the gate, call nc.solve() with the sitekey and the page URL, and yield a FormRequest that carries the token. NoneCap holds the connection until the solve is terminal, so there is no polling loop to write.

pip install scrapy nonecap

import os

import scrapy
from nonecap import NoneCap

nc = NoneCap(api_key=os.environ["NONECAP_KEY"])


class ListingsSpider(scrapy.Spider):
    name = "listings"
    start_urls = ["https://target.example/listings"]
    # Gates often arrive with a 403 status. Without this, the HttpError
    # spider middleware drops the response before parse() ever runs.
    handle_httpstatus_list = [403]

    def parse(self, response):
        sitekey = response.css(".h-captcha::attr(data-sitekey)").get()
        if sitekey is None:
            yield from self.parse_listings(response)  # no gate, business as usual
            return

        # Sync call: the crawl pauses here until the token arrives.
        # Fine for a single-session crawl; the middleware below scales.
        solve = nc.solve(type="hcaptcha", sitekey=sitekey, url=response.url)

        yield scrapy.FormRequest(
            response.url,
            formdata={
                "h-captcha-response": solve.token,
                # hCaptcha's reCAPTCHA-compat field name, same token
                "g-recaptcha-response": solve.token,
            },
            callback=self.parse,
            dont_filter=True,  # same URL again, skip the dupefilter
        )

    def parse_listings(self, response):
        for row in response.css(".listing"):
            yield {
                "title": row.css("h3::text").get(),
                "price": row.css(".price::text").get(),
            }

The retry yields back into parse(), which now sees the unlocked page and falls through to the real parsing. dont_filter=True matters: the retry targets a URL the dupefilter has already seen.

Two caveats. nc.solve() is a blocking call, and Scrapy runs everything in one thread, so the whole crawl pauses while it waits; for a single-session spider that is usually fine, and the middleware section below removes the stall. And Scrapy 2.16 (May 2026) deprecated scrapy.FormRequest in favour of the form2request package. It still works with a warning, and the no-dependency equivalent is a plain scrapy.Request with method="POST", a urlencoded body, and a Content-Type: application/x-www-form-urlencoded header, which is exactly what the middleware below builds.

The reusable pattern: a downloader middleware

Once two spiders need the same gate handling, move it into a downloader middleware. process_response() inspects every response on its way back to the spider; when it spots a gate it solves, then returns a new Request, which Scrapy reschedules through the full middleware chain while the spider stays captcha-blind. Since Scrapy 2.13 the asyncio reactor is the default, and downloader middleware methods can be coroutines, so the async client fits without blocking anything:

# middlewares.py
import logging
from urllib.parse import urlencode

from scrapy.exceptions import NotConfigured
from scrapy.http import TextResponse
from nonecap import AsyncNoneCap

logger = logging.getLogger(__name__)


class HcaptchaSolverMiddleware:
    """Detect hCaptcha gates and retry the request with a real token."""

    def __init__(self, crawler):
        api_key = crawler.settings.get("NONECAP_API_KEY")
        if not api_key:
            raise NotConfigured("NONECAP_API_KEY is not set")
        self.crawler = crawler
        self.nc = AsyncNoneCap(api_key=api_key)
        self.max_solves = crawler.settings.getint("NONECAP_MAX_SOLVES", 2)

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler)

    # Scrapy 2.13 and earlier also pass a trailing `spider` argument;
    # add it to the signature there. 2.14+ deprecated it.
    async def process_response(self, request, response):
        if not isinstance(response, TextResponse):
            return response  # binary body, nothing to inspect

        sitekey = response.css(".h-captcha::attr(data-sitekey)").get()
        if sitekey is None:
            return response  # not gated, pass straight through

        solves = request.meta.get("nonecap_solves", 0)
        if solves >= self.max_solves:
            logger.warning("still gated after %d solves: %s", solves, response.url)
            return response  # give up, let the spider decide

        solve = await self.nc.solve(
            type="hcaptcha", sitekey=sitekey, url=response.url
        )

        retry = request.replace(
            method="POST",
            body=urlencode({
                "h-captcha-response": solve.token,
                "g-recaptcha-response": solve.token,
            }),
            dont_filter=True,  # retries must survive the dupefilter
        )
        retry.headers["Content-Type"] = "application/x-www-form-urlencoded"
        retry.meta["nonecap_solves"] = solves + 1
        return retry  # the engine reschedules it; callback stays the same

# settings.py
import os

NONECAP_API_KEY = os.environ["NONECAP_KEY"]

DOWNLOADER_MIDDLEWARES = {
    # 560 > RetryMiddleware's 550: process_response() runs in
    # DECREASING priority order, so this middleware sees a gated 503
    # before RetryMiddleware burns its retries on it.
    "myproject.middlewares.HcaptchaSolverMiddleware": 560,
}

# Default since Scrapy 2.13. Set it explicitly on older versions so
# async middleware methods and async callbacks run on asyncio.
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

The priority number is doing real work. Scrapy calls process_response() in decreasing priority order, and the built-in RetryMiddleware sits at 550 with 503 in its default RETRY_HTTP_CODES. Registered at 560, the solver sees a 503-wrapped gate first and answers it with a token; left at a typical 543, the gate would eat RETRY_TIMES blind retries before your code ever ran. The nonecap_solves counter in meta caps how many tokens a stubborn URL can burn, and request.replace() keeps the original callback, cb_kwargs, and meta on the retry.

The blocking-call tradeoff

Scrapy is built on Twisted, and the reactor runs in a single thread. A blocking HTTP call inside a plain def process_response() halts that thread, which means it halts the whole process: every in-flight download and every scheduled callback waits, possibly the full 90 seconds a slow solve can take. With one gated URL per thousand pages you might never notice. On a heavily gated target the crawl crawls.

Scrapy gives you two documented ways out. The first is what the middleware above does: declare the method async def (supported for process_request, process_response, and process_exception) and await a non-blocking client, here AsyncNoneCap. The second helps when you want to keep sync code:

# Stuck with the sync client (or plain requests)? Keep the method
# async and push the blocking call onto a thread, so the reactor
# keeps scheduling every other download while the solve runs.
import asyncio

async def process_response(self, request, response):
    ...
    solve = await asyncio.to_thread(
        self.nc.solve, type="hcaptcha", sitekey=sitekey, url=response.url
    )

Both need the asyncio reactor, which is the default since Scrapy 2.13; on older versions set TWISTED_REACTOR as shown in the settings above. What you should not do is raise CONCURRENT_REQUESTS to paper over a blocking solver. Concurrency settings cannot help a reactor that is not running.

Keep the session consistent

A token is accepted in context. Three rules keep that context intact:

• Same URL. Pass the gated page’s own URL (response.url) as the url field of the solve, and submit the token back to that same page. Don’t solve against the site root.
• Same cookies. Scrapy’s CookiesMiddleware keeps one jar per spider by default, so the retry naturally shares the session that received the gate. If you shard sessions with the cookiejar meta key, make sure the retry carries the same key; request.replace() preserves meta, a hand-built FormRequest does not unless you pass it.
• Same IP. If a rotating-proxy middleware assigns a fresh exit per request, pin the retry to the proxy that fetched the gate (request.meta["proxy"] survives replace(); check that your rotation middleware doesn’t overwrite it on the rescheduled pass).

Timing matters too: the token is single-use and valid for roughly 120 seconds, so solve on demand when a gate appears. Don’t mint tokens ahead of time and queue them.

Scope: hCaptcha gates only

NoneCap solves hCaptcha only: regular, invisible, and enterprise (rqdata). If your Scrapy target sits behind reCAPTCHA or Cloudflare Turnstile, this middleware cannot unlock it; those are on the roadmap, not supported today. Check the gate first: the h-captcha div and the hcaptcha.com/1/api.js script are the tell.

For the broader no-browser pipeline this page builds on (detect, solve, inject, continue), read hCaptcha solving for web scraping. For the full Python client surface, including asyncio fan-out and typed errors, see the Python guide. Request and response shapes for every endpoint are in the API reference.