Web scraping with residential proxies: a practical Python setup

A scraper that works through a residential proxy is not the one with the cleverest headers. It is the one that treats each exit as a visitor, asks for the right kind of session for each job, retries only the failures a retry can fix, and knows how many gigabytes it will burn before it starts. This is a complete setup in Python with requests on GhostConnect residential proxies. The host, the ports and every price below are read from the configuration the order form uses.

What the proxy changes, and what it does not

A residential proxy changes one thing about a request: the address it arrives from. That address is a real household connection in the country, state or city you ask for, drawn from a pool in 195+ countries. A site that refuses hosting ranges on sight no longer refuses you by range. Everything else is still yours to get right: the proxy does not run JavaScript, does not touch your headers or cookies, and does not alter what a site's terms allow you to do.

The product is billed per gigabyte, $2.99 per GB, and the bandwidth never expires. You can buy 1 to 1,000 GB in one order and top up the same plan later. One set of credentials covers every country and both kinds of session, because targeting and the session are appended to the password. The gateway is residential.ghostconnect.com, port 1080 for HTTP (1111 as a second port) and 1081 for SOCKS5 (2222). This post uses requests; the configuration cookbook has the same strings for curl, Node, Scrapy and Playwright.

Two cases where residential is the wrong purchase. If the target does not care about hosting ranges, a datacenter proxy at $3.99 per IP per 31 days with unlimited bandwidth is cheaper for any crawl of size. If the target reaches over IPv6 and you want thousands of addresses rather than a believable one, rotating IPv6 at $0.012 per thread per day is priced by capacity, not by data. Residential is for sites that check where you come from, and it is metered, so every design choice below is also a bandwidth choice.

One Session per logical visitor

A requests.Session holds a cookie jar, default headers, a connection pool and a proxy setting. Treat one Session as one visitor. A visitor has one Accept-Language, one User-Agent, one exit policy, and a cookie jar that was filled by that exit. Two exits sharing one cookie jar is how a scrape starts to look like a stolen session: the server set a cookie for a visitor in one city and sees it come back from another.

The helper below builds the proxy URL from your credentials and whatever targeting the visitor needs, and visitor() wraps it in a Session. The tags follow the gateway's rules: country in capitals, state and city lowercase with spaces as dots, and a sticky session as a lifetime in minutes plus a session id of 6 to 15 letters, digits or dashes. A session id outside that rule makes the gateway drop the connection without an error, which looks like the proxy being down.

import secrets
import string
from urllib.parse import quote

import requests

HOST = "residential.ghostconnect.com"
PORT = 1080          # HTTP. SOCKS5 is on 1081 and needs requests[socks] and a socks5h:// URL.
USER = "rJxYKRe2U"        # the documentation credentials from the API reference: use your own
PASS = "Yxa5WGCUkvP9km"


def session_id(length=8):
    """6 to 15 letters, digits or dashes. Anything else makes the gateway drop the connection."""
    alphabet = string.ascii_lowercase + string.digits
    return "".join(secrets.choice(alphabet) for _ in range(length))


def proxy_url(country=None, state=None, city=None, sticky_minutes=None, sid=None):
    """Targeting and the session ride in the password. The username stays as issued."""
    password = PASS
    if country:
        password += "_country-" + country.upper()
    if state:
        password += "_state-" + state.lower().replace(" ", ".")
    if city:
        password += "_city-" + city.lower().replace(" ", ".")
    if sticky_minutes:
        password += "_lifetime-%d_session-%s" % (sticky_minutes, sid or session_id())
    return "http://%s:%s@%s:%d" % (quote(USER, safe=""), quote(password, safe=""), HOST, PORT)


def visitor(country=None, sticky_minutes=None, **targeting):
    """One Session is one visitor: one cookie jar, one set of headers, one exit policy."""
    s = requests.Session()
    url = proxy_url(country=country, sticky_minutes=sticky_minutes, **targeting)
    s.proxies = {"http": url, "https": url}
    s.headers.update({"User-Agent": "your-client-name/1.0", "Accept-Language": "de-DE,de;q=0.8"})
    return s

Keep the language header consistent with the country tag. A visitor from Germany asking for English pages is not wrong, but it is a detail a site can notice, and it changes what you get back on sites that pick a locale from the header rather than the address.

Rotating for independent pages, sticky for paginated flows

The rule is about dependency. If you can write down the URL of page N without having visited page N minus one, the pages are independent, and a rotating visitor is right: every request leaves from a new household, no single address accumulates a request count, and nothing needs to be kept between fetches. Product pages by id, sitemap entries, listings with a predictable page parameter, anything you can enumerate up front.

If page N only exists because of page N minus one, use a sticky visitor. Search results whose cursor lives in a server-side session, a filter applied by form post that is remembered in a cookie, a detail page that checks the referring listing, any flow where the server ties state to the address. A sticky visitor keeps one exit for the lifetime you set, 1 to 1440 minutes, and every request it makes under that session id leaves from the same household. In the code, visitor(country="DE") is rotating and visitor(country="DE", sticky_minutes=15) is sticky for a quarter of an hour.

Size the lifetime to the flow and no more. A twenty-page walk at one request a second needs a few minutes, not a day. Nothing holds an exit past its lifetime, and a flow that outlives its session changes address in the middle, which is the one thing a sticky session was bought to avoid. The sticky-session guide covers the rest of the mechanics; the short version is that lifetime and session always go together, and a different session id is a different exit.

Country, state and city tags

Targeting is the second reason to use residential rather than any static address: prices, stock, language and even the existence of a page differ by region, and the only way to see what a visitor in Bavaria sees is to arrive from Bavaria. _country-DE alone gives you a new German household per request. Add _state-bayern and _city-munich to narrow it, remembering that a state or city without a country is refused. Narrower targeting means a smaller pool to draw from, so ask for the narrowest geography the job needs rather than the narrowest you can type.

Check the exit before trusting a crawl's output. Any page that echoes the caller's address will do:

curl -x http://residential.ghostconnect.com:1080 -U 'rJxYKRe2U:Yxa5WGCUkvP9km_country-DE_state-bayern_city-munich' https://ifconfig.me

Run it twice: a rotating string returns two addresses, a sticky one returns the same address both times. If it fails outright rather than answering, the usual cause is the tags on the username instead of the password, which the gateway answers with a 407.

Retries with backoff

Retry what a retry can fix: a connection that could not be opened, a timeout, and a 429 or a 5xx from the target. Do not retry a 404, think before retrying a 403, and never retry a 407 from the proxy, because that is your credentials or your tag placement and the fifth attempt will fail like the first. On an https URL a 407 surfaces in requests as a ProxyError whose message contains the status, so the code checks for it and raises instead of looping.

On a rotating visitor a retry is also a new exit, which is usually what you want. On a sticky visitor a retry stays on the same exit, which is what you want for a flow, until it becomes clear that the exit itself is the problem. Then start a new session id and redo the flow from its first page, not from the page that failed. The server's state belongs to the old visitor.

import random
import time

RETRY_STATUS = {429, 500, 502, 503, 504}


def fetch(s, url, attempts=4, timeout=(10, 30)):
    """GET with exponential backoff and jitter. Retries network errors and RETRY_STATUS, nothing else."""
    for attempt in range(attempts):
        try:
            r = s.get(url, timeout=timeout)
        except requests.exceptions.ProxyError as exc:
            if "407" in str(exc):
                raise        # wrong credentials, or tags on the username: a retry cannot fix it
            r = None
        except (requests.exceptions.ConnectionError, requests.exceptions.Timeout):
            r = None
        if r is not None and r.status_code not in RETRY_STATUS:
            return r
        if attempt == attempts - 1:
            break
        wait = min(60, 2 ** attempt) + random.uniform(0, 1)
        retry_after = r.headers.get("Retry-After", "") if r is not None else ""
        if retry_after.isdigit():
            wait = max(wait, int(retry_after))
        time.sleep(wait)
    raise RuntimeError("gave up on " + url)

Backoff is exponential with jitter and a cap, and a numeric Retry-After from the target overrides the computed wait when it is longer. Cap the attempts: every retry is metered bandwidth, and four attempts at a page that is down is four times the cost of a page that was up.

Per-domain concurrency, robots.txt and rate limits

Concurrency is a property of the target, not of your machine. A proxy pool lets you open many connections; it does not change what one site can bear, and the site's answer to too many is the same 429 or 503 it would give anyone. Limit in-flight requests per host and space the starts. The limiter below allows two connections per host and one request start per second by default; those are the example's starting values, to be tuned per target and lowered at the first 429.

Read robots.txt and honour it. A disallowed path is disallowed from every address, and a residential exit is not an argument against it. urllib.robotparser handles the parsing and gives you any Crawl-delay the site publishes, which the code feeds to the limiter.

import collections
import threading
from urllib import robotparser
from urllib.parse import urlsplit


class HostLimiter:
    """A few connections in flight per host and a pause between request starts. Tune per target."""

    def __init__(self, in_flight=2, delay=1.0):
        self.delay = delay
        self.sem = collections.defaultdict(lambda: threading.BoundedSemaphore(in_flight))
        self.next_start = collections.defaultdict(float)
        self.lock = threading.Lock()

    def get(self, s, url, delay=None):
        host = urlsplit(url).hostname
        with self.lock:
            sem = self.sem[host]
            start = max(time.monotonic(), self.next_start[host])
            self.next_start[host] = start + (delay or self.delay)
        time.sleep(max(0.0, start - time.monotonic()))
        with sem:
            return fetch(s, url)


def robots(s, host):
    """robots.txt for one host. A missing file allows everything; a disallow is a disallow through any proxy."""
    rp = robotparser.RobotFileParser()
    r = fetch(s, "https://%s/robots.txt" % host)
    rp.parse(r.text.splitlines() if r.status_code == 200 else [])
    return rp

If you run the crawler across threads, the limiter is safe to share. A rotating visitor can also be shared for independent pages, since nothing in its cookie jar matters, but keep each sticky flow in its own Session on its own thread: the flow is the thing you are keeping intact.

The bandwidth budget

Everything the gateway carries is metered: the HTML, the redirects before it, the TLS handshake around it, the robots.txt fetch, the exit check, and every retry. What requests does not fetch is images, scripts and stylesheets, which is most of the saving against a browser and the reason a requests-based scraper is the cheap way to use a residential plan.

A worked budget, with the assumptions stated. Say the job is 40,000 pages and each page transfers about 350 KB, compressed HTML plus the connection overhead around it. That is 14.0 GB of pages. Add a 1.15x margin for retries and redirects and round up to a whole gigabyte, because orders are in whole gigabytes: 17 GB, which costs $50.83 at $2.99 per GB. Per thousand pages, at that page size, the data costs $1.047.

Measure your own page size before buying: fetch fifty pages through the proxy, sum len(r.content), add a tenth for headers and handshakes, and scale. Then watch the plan while it runs. The dashboard shows what each hour used, and the API returns the same series, hourly for the last day:

curl -H "Authorization: Bearer $GT_KEY" \
  "https://ghostconnect.com/api/v1/proxies/residential/obcl4126/usage?window=24h"

If the budget was short, a top-up adds gigabytes to the same plan at the same price without a new order. Unused bandwidth never expires, so buying a little more than the estimate costs nothing but the float; terminating a plan early, on the other hand, does not refund what is left.

The whole thing

Put the four blocks above in one file and this is the entry point. A rotating visitor walks a list of independent pages under the robots rules and the host limiter; a sticky visitor walks a paginated search, with the lifetime sized to the walk.

if __name__ == "__main__":
    ua = "your-client-name/1.0"
    limiter = HostLimiter(in_flight=2, delay=1.0)

    # Independent pages: one rotating visitor is already a pool of exits.
    crawler = visitor(country="DE")
    rp = robots(crawler, "www.example.com")
    delay = rp.crawl_delay(ua) or 1.0
    for url in ["https://www.example.com/p/1001", "https://www.example.com/p/1002"]:
        if not rp.can_fetch(ua, url):
            continue
        r = limiter.get(crawler, url, delay=delay)
        print(r.status_code, len(r.content), url)

    # A paginated flow: one sticky visitor for the whole walk, lifetime sized to the flow.
    walker = visitor(country="DE", sticky_minutes=15)
    page = "https://www.example.com/search?q=lamp"
    while page:
        r = limiter.get(walker, page, delay=delay)
        page = next_page_url(r)      # your parser: the href of the next link, or None at the end

What this setup leaves behind on our side is small. The network keeps the destination and the response status code of each request it carries and nothing else; the address you connect from is never logged, and the content is not looked at. The privacy page lists everything the site stores. The account itself is a key with no name or email on it; get one, buy the gigabytes the budget says, and start with a single page through the exit check before you start the crawl.