Proxy configuration cookbook: curl, Python, Node, Scrapy and Playwright

Every tool wants the same four facts, a host, a port, a protocol and a way to prove who you are, and every tool asks for them differently. This is the lookup table: a working proxy setting for curl, Python requests and httpx, Node 18 fetch, Scrapy and Playwright, each over HTTP and over SOCKS5, each with a username and password or with an allowlisted IP and no password. The values are GhostConnect's residential gateway and the documentation credentials from the API reference, so nothing here connects until you swap in your own.

The values every recipe uses

The residential gateway is residential.ghostconnect.com. HTTP is on port 1080, SOCKS5 on port 1081; a second port of each kind, 1111 and 2222, behaves identically and exists for networks that block the first. The username below is rJxYKRe2U and the password Yxa5WGCUkvP9km; yours are on the proxy's dashboard page and in the username and password fields the API returns.

There are two ways to authenticate. With a username and password, the gateway accepts you from any address, and targeting and sticky sessions ride in the password as tags: Yxa5WGCUkvP9km_country-US pins the exit to the United States, Yxa5WGCUkvP9km_country-US_lifetime-30_session-k2p9x7vq keeps one American exit for 30 minutes under that session name. Pass the whole thing as the password, underscores included; the sticky-session guide covers the tags. With an allowlisted IP, you add the public address you connect from on the proxy's page or with POST /proxies/{type}/{id}/allowlist, up to 10 per proxy, and connect with no credentials at all. That is simpler for a server, and it is also the one place an address of yours is stored, as the privacy page notes. Since the tags live in the password, an allowlisted connection that sends no password has nowhere to put a country or a session; use the password form whenever you need targeting.

The same recipes work for the other products with their own values. A datacenter or ISP proxy gives you an IP, an HTTP port and a SOCKS5 port per address; an IPv6 datacenter address goes in square brackets inside a URL. A rotating IPv6 gateway is HTTP only, so skip the SOCKS5 lines. ISP proxies have no allowlist, so skip those too.

One protocol note before the code. An HTTP proxy always receives the hostname you asked for, so name resolution happens at the exit. SOCKS5 does not: a client may resolve the name itself and hand the proxy an IP, which leaks the lookup to your own resolver and can return an address meant for your region rather than the exit's. Where a tool offers socks5h, the h means the hostname is sent to the proxy, and that is the one to use.

curl

-x (or --proxy) takes the proxy as a URL whose scheme picks the protocol, and -U (--proxy-user) takes the credentials. The target is https://ifconfig.me throughout, because it echoes the address it saw, which is the exit's.

# HTTP, username and password, pinned to the United States
curl -x http://residential.ghostconnect.com:1080 -U 'rJxYKRe2U:Yxa5WGCUkvP9km_country-US' https://ifconfig.me

# HTTP, allowlisted IP, no credentials
curl -x http://residential.ghostconnect.com:1080 https://ifconfig.me

# SOCKS5 with remote DNS, username and password, one sticky exit for 30 minutes
curl -x socks5h://residential.ghostconnect.com:1081 -U 'rJxYKRe2U:Yxa5WGCUkvP9km_country-US_lifetime-30_session-k2p9x7vq' https://ifconfig.me

# SOCKS5 with remote DNS, allowlisted IP
curl -x socks5h://residential.ghostconnect.com:1081 https://ifconfig.me

Quote the credentials: an unquoted password with an underscore is fine in most shells, but a habit of quoting saves you when a password contains a character the shell cares about. If the first port is blocked where you are, change 1080 to 1111 or 1081 to 2222 and nothing else. curl -v shows the proxy conversation, including the CONNECT and any 407, which is the quickest way to tell a proxy problem from a target problem. The options are in the curl manual.

Python: requests and httpx

Requests takes a proxies dictionary keyed by the scheme of the target URL, so set both http and https to the same proxy. Credentials go in the proxy URL. SOCKS needs the optional dependency, pip install requests[socks], and the socks5h scheme for remote resolution; plain socks5 resolves on your machine.

import requests

# HTTP, username and password
proxy = "http://rJxYKRe2U:Yxa5WGCUkvP9km_country-US@residential.ghostconnect.com:1080"
proxies = {"http": proxy, "https": proxy}
print(requests.get("https://ifconfig.me", proxies=proxies, timeout=(10, 30)).text)

# HTTP, allowlisted IP
proxies = {"http": "http://residential.ghostconnect.com:1080", "https": "http://residential.ghostconnect.com:1080"}

# SOCKS5 with remote DNS, username and password (pip install requests[socks])
proxy = "socks5h://rJxYKRe2U:Yxa5WGCUkvP9km_country-US_lifetime-30_session-k2p9x7vq@residential.ghostconnect.com:1081"
proxies = {"http": proxy, "https": proxy}

# SOCKS5 with remote DNS, allowlisted IP
proxies = {"http": "socks5h://residential.ghostconnect.com:1081", "https": "socks5h://residential.ghostconnect.com:1081"}

s = requests.Session()
s.proxies.update(proxies)        # every request on the session uses the proxy

A password inside a URL must be percent-encoded if it contains @, :, / or %. The documentation password and its tags use only letters, digits, underscores, dashes and dots, so they can be pasted as they are. Use urllib.parse.quote(password, safe="") if yours ever differs. A session that should keep one exit needs one sticky string for its whole life; a new session name is a new exit. The requests documentation has the rest.

httpx takes a single proxy argument on the client, with the same URL forms. SOCKS needs pip install httpx[socks]; socks5:// is accepted, and recent versions accept socks5h:// for remote resolution too.

import httpx

# HTTP, username and password
with httpx.Client(proxy="http://rJxYKRe2U:Yxa5WGCUkvP9km_country-US@residential.ghostconnect.com:1080", timeout=30) as client:
    print(client.get("https://ifconfig.me").text)

# HTTP, allowlisted IP
client = httpx.Client(proxy="http://residential.ghostconnect.com:1080")

# SOCKS5, username and password (pip install httpx[socks])
client = httpx.Client(proxy="socks5://rJxYKRe2U:Yxa5WGCUkvP9km_country-US_lifetime-30_session-k2p9x7vq@residential.ghostconnect.com:1081")

Node 18: fetch through undici's ProxyAgent

Node 18's built-in fetch is undici, and undici routes through a proxy with a ProxyAgent dispatcher. Install the undici package and import both fetch and ProxyAgent from it, so the agent and the fetch are the same version; mixing the installed agent with the global fetch is where version mismatches bite. Credentials go in a token, a ready-made Proxy-Authorization value, or inside the uri.

// npm install undici
import { fetch, ProxyAgent, setGlobalDispatcher } from "undici";

// HTTP, username and password
const token = "Basic " + Buffer.from("rJxYKRe2U:Yxa5WGCUkvP9km_country-US").toString("base64");
const dispatcher = new ProxyAgent({ uri: "http://residential.ghostconnect.com:1080", token });
const res = await fetch("https://ifconfig.me", { dispatcher });
console.log(await res.text());

// HTTP, allowlisted IP: no token
setGlobalDispatcher(new ProxyAgent("http://residential.ghostconnect.com:1080"));   // every fetch from here on

undici does not speak SOCKS5. Use the HTTP port, or, if the job really needs SOCKS5, the socks-proxy-agent package with Node's https module rather than fetch. Node 18's fetch also ignores the proxy environment variables, so the dispatcher is the only way to route it. The undici repository documents ProxyAgent.

Scrapy

Scrapy's HttpProxyMiddleware is enabled by default at order 750. It reads request.meta["proxy"] as a proxy URL, moves any credentials in it into a Proxy-Authorization header, and tunnels HTTPS through CONNECT. The smallest change is a meta on each request; the tidier one is a middleware that sets it for every request, placed before 750 so the built-in one sees the value.

# settings.py
GHOSTCONNECT_PROXY = "http://rJxYKRe2U:Yxa5WGCUkvP9km_country-US@residential.ghostconnect.com:1080"
# allowlisted IP instead: GHOSTCONNECT_PROXY = "http://residential.ghostconnect.com:1080"

DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.GhostConnectProxyMiddleware": 740,
}

# middlewares.py
class GhostConnectProxyMiddleware:
    def __init__(self, proxy):
        self.proxy = proxy

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.settings.get("GHOSTCONNECT_PROXY"))

    def process_request(self, request, spider):
        request.meta["proxy"] = self.proxy

# or per request, inside a spider
yield scrapy.Request(url, meta={"proxy": "http://rJxYKRe2U:Yxa5WGCUkvP9km_country-US@residential.ghostconnect.com:1080"})

Scrapy has no SOCKS5 support of its own; its downloader speaks HTTP proxies only, so use port 1080. The middleware above is also the place to hand different requests different sticky sessions, by building the password with a different _session- tag per item: Scrapy sends whatever URL you set. The middleware documentation covers the details, including that it honours http_proxy and https_proxy from the environment when no meta is set.

Playwright

A browser loads dozens of resources per page, so route a browser through a sticky session: with a rotating string every image and script would leave from a different household, which no real visitor does. Pass the proxy at launch, with the server as a URL and the credentials as separate fields.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    # HTTP, username and password, one exit for 30 minutes
    browser = p.chromium.launch(proxy={
        "server": "http://residential.ghostconnect.com:1080",
        "username": "rJxYKRe2U",
        "password": "Yxa5WGCUkvP9km_country-US_lifetime-30_session-k2p9x7vq",
    })
    page = browser.new_page()
    page.goto("https://ifconfig.me")
    print(page.text_content("body"))
    browser.close()

    # HTTP, allowlisted IP
    browser = p.chromium.launch(proxy={"server": "http://residential.ghostconnect.com:1080"})

    # SOCKS5, allowlisted IP only: Chromium does not send credentials to a SOCKS5 proxy
    browser = p.chromium.launch(proxy={"server": "socks5://residential.ghostconnect.com:1081"})

// Node
const { chromium } = require("playwright");
const browser = await chromium.launch({
  proxy: { server: "http://residential.ghostconnect.com:1080", username: "rJxYKRe2U", password: "Yxa5WGCUkvP9km_country-US_lifetime-30_session-k2p9x7vq" },
});

Chromium authenticates to HTTP proxies and not to SOCKS5 ones, so the SOCKS5 route in Playwright is for an allowlisted IP; with a password, use HTTP. Chromium resolves names at a SOCKS5 proxy, so there is no h variant to choose. The same proxy object is accepted by new_context() for a per-context proxy; the Playwright network documentation lists the browser-specific caveats.

Environment variables

Most command-line tools and libraries read a proxy from the environment, which configures a whole shell session at once. The variables are http_proxy, https_proxy, all_proxy and no_proxy, named for the scheme of the target, not of the proxy.

# HTTP, username and password
export http_proxy="http://rJxYKRe2U:Yxa5WGCUkvP9km_country-US@residential.ghostconnect.com:1080"
export https_proxy="$http_proxy"
export no_proxy="localhost,127.0.0.1"

# allowlisted IP
export http_proxy="http://residential.ghostconnect.com:1080"
export https_proxy="$http_proxy"

# SOCKS5 with remote DNS, for tools that honour all_proxy (curl does)
export all_proxy="socks5h://rJxYKRe2U:Yxa5WGCUkvP9km_country-US_lifetime-30_session-k2p9x7vq@residential.ghostconnect.com:1081"

Two traps. curl reads http_proxy in lowercase only, by design, because an upper-case HTTP_PROXY can be set from the outside in CGI environments; https_proxy and all_proxy are read in either case. And a variable set in one terminal is not set in a cron job or a systemd unit, so a script that works at the keyboard and fails on a schedule has usually lost its environment. requests, httpx and Scrapy honour these variables; Node 18's fetch and Playwright do not.

What each tool can and cannot do

Whichever row you are on, test it the same way: fetch https://ifconfig.me through the proxy and read the address back. A residential exit changes between calls on a rotating string and stays put on a sticky one. If the fetch fails rather than returning a surprising address, check the port against the protocol first (HTTP on 1080, SOCKS5 on 1081), then the credential field, then the tags.

Testing is cheap. Residential bandwidth is $2.99 per GB, so the smallest order, one gigabyte for $2.99, covers every recipe on this page many thousands of times over, on the assumption that an echo response is under a kilobyte. If you want to try a tool's proxy support with no meter at all, a single datacenter IP for a week costs $0.90 with unlimited bandwidth, and one ISP IP for a month costs $5.00. The order flow for all of them is the same, on the website or, as the API walkthrough shows, from code: a key, a balance, and the residential, datacenter, ISP or rotating IPv6 product that fits.