My scraper started getting blocked: what to ask the freelancer who built it

The short answer

A scraper that worked for weeks and now gets 403 Forbidden, 429 Too Many Requests or a captcha has not broken in the usual sense. The site changed how it treats automated visitors, or the scraper started asking too often. The code may be exactly as it was on delivery day.

So the first message to the freelancer is not “fix it”. It is a short report and six questions: what exactly the site now returns, why, what the cheapest way back is, what it will cost each month, whether it is allowed, and who pays. Those answers decide whether this is a small adjustment, a paid repair, or a sign to stop scraping that site.

What the error is telling you

The three signals look alike from the outside (no data arrived) but they mean different things.

What you see	What the site is saying	Usual first fix
429 Too Many Requests	“You are asking too often.”	Slow down, spread runs out, wait properly after a refusal
403 Forbidden	“I know what you are, and the answer is no.”	Find out what changed: bot protection, login, a new page path
Captcha page	“Prove you are a person.”	Ask whether a script belongs here at all, then look at slower or official routes
200 OK, but empty rows	Nothing, which is the dangerous one	The page layout changed; the scraper reads the wrong place

The first row is one I know from my own collectors. One of them collects new videos from a video platform’s channels. The platform’s public feed answers “too many requests” often enough that the collector does not use it and reads each channel’s video page directly instead. The fix was not clever code. It was “ask less, or ask the way the site accepts”.

The fourth row is the one that does not announce itself. When a site redesigns a page or simply stops showing a field, nothing errors: the scraper keeps “succeeding” and hands you a spreadsheet with an empty column, or with numbers read from the wrong place. You only notice when someone uses the data. That is why the brief section below asks for an alert on low counts, not only on errors.

If you are not sure which row you are in, ask the freelancer for one raw response from the site: the status code and the first lines of the page it returned. That single sample usually settles the question, and it costs nothing to send. Keep it in the message thread. If the block comes back later, you can compare the new response with the old one instead of starting the diagnosis from zero.

Six questions to send the freelancer

Copy this, fill in the first two lines, and send it before anyone quotes:

Question 4 is where most surprise costs live. A proxy or captcha service is a monthly bill that lasts as long as the scraper does, on top of the running costs you already expect, and it should sit on your account, not the freelancer’s, for the same reasons as API keys.

Question 5 matters more than it sounds. If the site added a captcha to the exact pages you collect, the freelancer can often get past it, but that is a decision you are making on your business’s behalf. Read robots.txt and terms of service before you approve it.

Fix, new work, or stop

How you pay for this depends on what you agreed at the start.

• If you agreed maintenance terms (fixes included for a period, then a rate), a site change that stops the scraper is the textbook case of a fix: the scraper no longer does what the brief said, and nothing in your brief changed. The data scraping brief has a “When the site changes” field for exactly this.
• If you did not, expect a paid repair. That is fair: the original code did what it promised, and the site moved. Ask for a fixed price for the repair plus terms for the next time, the way maintenance in the contract describes.
• If your volume went up (more sites, more pages, more often), the block is a consequence of a change you made. Treat it as new work.
• If the site is clearly fighting scripts (captchas on every page, blocks returning days after each fix), the honest answer may be to stop. A data source that needs a new workaround every month costs more than it looks, and every workaround is a step further from what the site’s owner wants.

Whatever you decide, write it in the same message thread, so the next block starts from an agreement instead of a negotiation. If the freelancer is unreachable, who fixes a stopped automation covers the hand-over.

What to put in the next brief

Most blocks are predictable at brief time. Add these lines so the next scraper is built for them:

Pace. Requests spread out, with a wait after any refusal. No retries in a tight loop. Signals. The run log records the status code of every failed page. An alert reaches me on any 403, 429 or captcha, and on any run that returns zero rows or an unusually low count. Paid services. Any proxy, captcha or browser service is named in the quote, with monthly cost, on my account. Change response. Fix rate and response time when the site changes, for the first 60 days.

The alert on zero rows is the one people skip, and it is the one that catches the fourth row of the table. A scraper that quietly returns nothing is worse than one that fails loudly, because you keep trusting the spreadsheet. The acceptance checklist for unattended jobs has the tests to run before you accept the repaired version.

Wrap-up

A blocked scraper is a message from the site, not a bug report. Find out which message it is, ask the six questions before anyone quotes, and decide on the cost and the rules together. Then put pace, alerts and paid services into the next brief, so the next block is a short ticket instead of a surprise.