Hiring someone to build a scheduled web scraper: frequency, output, and the day the site changes

The short answer

A one-off scrape buys you a spreadsheet. A scheduled scraper buys you a small program that reads a website you do not control, every day or every week, for as long as you need it. That second thing needs a different brief.

Before you message anyone, decide five things: how often it runs, what comes out and where it lands, where it runs, how you hear about a bad run, and who fixes it when the site changes. The data scraping brief already has fields for source, columns and volume. This guide is about the five that make it keep working in month three.

Why a recurring scraper is a different job

The freelancer’s first delivery will almost always look fine. They run it, the file fills up, the sample rows match the site. The problems arrive later, from the source rather than the code:

• The page changes its layout. A field moves, a class name changes, the scraper reads an empty cell.
• The page stops showing something. A line of information you relied on simply disappears.
• The site starts refusing automated visits. Requests come back blocked or with a captcha.
• The page behaves differently than anyone assumed. It loads some items twice, or shows a different sample each time.

I have run into two of these in my own collectors. While checking marketplace seller pages for recency, the line I used as my main signal, the date of the seller’s last delivery, stopped being shown on the page. Nothing crashed; the field was just gone, and the rule built on it had to be replaced with a different signal (the date of the most recent review). And one collector of mine clicked a “show more” button that loaded some reviews twice, which only came to light when I extended the collection to more sellers (the full note).

None of those were coding mistakes. They are what a scheduled scraper lives with. The brief should assume they will happen.

Decide these five things first

1. Frequency. Pick the slowest schedule that still serves the decision the data feeds. If you look at the numbers on Monday mornings, a weekly run on Sunday night is enough. Every extra run is another chance to be blocked and another set of rows to de-duplicate. Write the schedule as a time and a time zone, not “regularly”. If cron schedules are new to you, that page explains what the freelancer will set up.

2. Output. Name the place and the shape. A new tab per run, one growing sheet, a CSV in a shared folder, a database table. Then decide what happens on the second run: does it append new rows, replace the whole set, or update rows that already exist? This one decision prevents most “why do I have three copies of every listing” conversations. Ask for a column with the run date on every row.

3. Where it runs. On a platform or server in your name, with the code in your repository. Scheduled jobs on a personal machine skip runs when it sleeps or restarts; one of mine missed a run entirely while the laptop slept. If the freelancer proposes their own server, ask what happens the day you stop paying them.

4. Failure signal. A scraper rarely fails with a clean error. It fails by returning nothing, or by returning rows with an empty column. Ask for an alert that goes to you when a run returns zero rows, far fewer rows than usual, or a key column that is suddenly empty. A separate check that looks at the output, not the scraper’s own report, is what catches these. Heartbeat monitoring is the name for that idea.

5. Change policy. Agree, in writing, what a fix costs when the site changes and how fast it happens. A layout change is not a bug in the delivery, and it is not new work either; it is maintenance. The maintenance guide has a message you can adapt.

The brief you can copy

Line 8 matters more than it looks. A scraper that requests pages as fast as it can is the one that gets blocked first. A good freelancer will explain what a rate limit means for your schedule before you ask.

Questions that separate the quotes

Send the brief to two or three sellers and ask the same four questions. The answers tell you more than the prices.

• “What is hard about this site?” A seller who opened your sample URLs will name something specific: infinite scroll, a login, a search that caps results. “No problem” is not an answer.
• “What does the second run do to the first run’s data?” Listen for append, replace or update, and a key to match rows on.
• “How will I know if it returns nothing?” You want an alert to you, not a log on their machine.
• “What happens when the layout changes?” You want a rate and a response time, not “just message me”.

On the data scraping page, BigiByte is listed as the pick for a recurring scraper on a schedule; an agency is useful when the job has to keep working after the first delivery. For the scheduling and delivery side, a workflow automation seller can also wrap a scraper in a platform you already use.

Acceptance: test the schedule, not the script

The scraping acceptance lines cover the data itself: 20 random rows against the source, the row count against the site, missing values marked the agreed way. For a scheduled scraper, add the tests from the unattended acceptance checklist:

Test	How	Pass
Runs on its own	Let three scheduled runs fire without touching it	Three dated runs in the output
Second run behaves	Compare run 1 and run 2	No duplicated rows; changed items updated or appended as agreed
Empty run alerts	Point it at a page with no results, or break the selector once	You get an alert, not a silent empty file
You can run it	Follow the README yourself, or watch it run outside the seller’s computer	It runs from your account
Credentials are yours	Check where any login or proxy key lives	In an account you can rotate

Do not skip the empty-run test. A scheduled job that quietly writes an empty file every Monday is worse than no job, because you keep trusting it.

Where it goes wrong anyway

• The site adds bot protection. The fix may need a different method, such as a real browser session or a paid service, and that changes the running cost. Agree that the seller tells you before switching.
• The site’s terms change. Re-read them when you renew. The job page FAQ notes that personal data and logged-in areas are where problems start; this guide is not legal advice.
• Your needs change. A new column or a second site is a change, not a fix. Budget it as a small new order.

Wrap-up

Brief a scheduled scraper the way you would brief any unattended job: say how often, say where the result lands and what the second run does to it, put it on your account, and make it tell you when it returns nothing. Then agree, before the first layout change, who fixes it and at what rate. The scraping is the easy part. Staying right every week is what you are paying for.