What are proxies, and why does a scraper need them?

What it is, in plain words

Think of sending letters to a company that has decided it gets too many letters from your street. After a while, the mailroom starts returning anything with your return address on it. A proxy is like asking friends in other towns to post your letters for you. The company sees their addresses, not yours.

On the internet, every request a scraper sends carries an IP address, which is roughly the return address of the computer that sent it. Websites watch those addresses. When one address asks for hundreds of pages in a few minutes, the site can slow it down, show a captcha or refuse it outright.

A proxy is a computer that sits in between. Your scraper sends its request to the proxy, the proxy fetches the page under its own address and passes the answer back. A proxy service rents you many such addresses, often rotating them so each request seems to come from somewhere else.

Words you will hear in a quote:

• Datacenter proxies: addresses that belong to server companies. Cheaper and fast, but easy for a site to recognise as not-a-person.
• Residential proxies: addresses that belong to home internet connections. Harder for a site to tell apart from ordinary visitors, and usually more expensive.
• Rotating proxies: a pool that hands out a different address per request or per few minutes.
• Billed per GB or per IP: the two common ways these services charge, by traffic used or by number of addresses.

A concrete example

A small shop wants a weekly spreadsheet of competitor prices from a few large online stores. The first test run, a few dozen pages, works fine from the freelancer’s own computer. The full run, several thousand pages every Monday, starts getting refusals halfway through. The freelancer comes back and says the job now needs a proxy service, and that it will cost money every month.

That is the usual shape of the conversation, and it is a fair one. But it is not the only answer.

One collector of mine pulls channel lists from a video platform. The platform offers a feed for exactly that, but the feed answered “too many requests” too often, so the collector reads the platform’s public channel pages instead. No proxies were involved: the fix was to ask in a way the site accepts, not to disguise who was asking.

Before agreeing to a proxy bill, ask the freelancer whether the same data can be had more cheaply:

• Fewer requests. Does the job really need every page every week, or only the pages that changed?
• A slower schedule. Spreading the same run over a whole night instead of a few minutes often stays under the site’s limits.
• An official route. Some sites offer a feed, an export or an API for the data you want. It may be limited, but it is the route the site has agreed to.

Why it matters when you hire someone

Proxies change three things about the job you are buying.

1. The price never stops. The scraper is a one-time build. A proxy subscription is a monthly bill for as long as the scraper runs, and it grows with the number of pages. The data scraping job page lists anti-bot protection among the things that raise a quote, and proxies are a large part of why.

2. The account question. If the proxy service is on the freelancer’s account, your scraper stops the day their subscription does. This is the same problem as API keys: the paid account should be yours, with access you can remove.

3. It can be a sign of a fight with the site. A proxy that spreads polite traffic over a few addresses is one thing. A pool of residential addresses used to get past a block the site put up on purpose is another. Before paying for that, read what robots.txt and the terms of service say about the site you want scraped. A site that is working hard to keep scripts out will keep changing its defences, and every change is a maintenance call.

What goes wrong when the buyer does not know this: the quote looks cheap, the first delivery works, and two weeks later the scraper stops with an error nobody understands, or a monthly charge appears that was never in the budget.

What to ask the freelancer

A good answer starts with “maybe not”: a slower schedule, fewer pages, or an official data feed if the site offers one. If proxies are still needed, the freelancer should name the type, give you a monthly estimate and set the service up on your account. For rate limits that slow a scraper down without blocking it, see what a rate limit is.