What are robots.txt and a site's terms of service?

What they are, in plain words

Think of a shop. On the door there is a small sign: “Staff only beyond this point” on one corridor, “No deliveries through the front” on another. The sign does not lock anything. It tells polite visitors where the owner would rather they did not go. Behind the counter there is also a set of house rules you agreed to when you signed up for the loyalty card.

Robots.txt is the sign on the door. It is a plain text file at the root of almost every website (type the domain followed by /robots.txt). It speaks to automated visitors, called crawlers or bots, and lists the paths they are asked not to fetch. It can address all bots at once or name specific ones.

The terms of service are the house rules. They are a legal document for everyone who uses the site, and many of them say something about automated access, copying content or collecting data.

Two facts about robots.txt surprise most people:

• It is not enforced. Google’s own guide says the instructions “cannot enforce crawler behavior to your site; it’s up to the crawler to obey them.” The standard that defines the file, RFC 9309, says its rules “are not a form of access authorization.”
• It is public. Anyone can read it, and listing a path there actually tells the world the path exists.

So robots.txt tells you what the owner wants. The terms tell you what you agreed to. Neither is a technical barrier, and that is exactly why the person you hire has to choose to respect them.

A concrete example

Say a small furniture shop wants a weekly spreadsheet of competitor prices from three online stores. The freelancer’s first step should not be code. It should be reading three robots.txt files and three sets of terms.

Typical findings look like this:

• Store A’s robots.txt blocks /checkout/ and /account/ but leaves product pages open. The terms say nothing about automated access. This is the easy case.
• Store B’s robots.txt allows product pages, but its terms say no automated collection of prices or listings. The robots file is friendly; the house rules are not.
• Store C needs a login to see prices. Anything behind a login usually falls under the account terms the user accepted, which is where problems most often start.

Sites also have technical defences that are separate from both documents. When a script of mine requested pages on a large marketplace automatically, the site answered with a 403 refusal and a captcha. I postponed that check and did it later in my own browser. A refusal like that is the site telling you, in code, what its terms usually say in words.

From the other side: a monitoring job of mine checks the robots.txt of my own sites every few days, because one wrong line there can ask search engines to stop reading a whole site. Site owners do pay attention to this file.

Why it matters when you hire someone

The scraper runs on your behalf. If it collects data a site asked bots not to collect, or breaks terms you are bound by, the consequences land on you and your business, not on the freelancer who has moved on to the next order.

Four things go wrong when the buyer does not know about this:

1. The data is built on something that can be taken away. A store that does not want to be scraped can add blocks at any time. A weekly scraper against a hostile site is a maintenance bill waiting to happen.
2. Your own account is at risk. If the scraper logs in with your account, it is your account that breaks the terms and your account that gets suspended.
3. Speed becomes a problem. Some robots.txt files include a crawl delay. A scraper that ignores it sends requests fast enough to trigger rate limits and blocks.
4. Personal data changes everything. Names, emails and profiles carry privacy rules on top of the site’s terms. Public, non-personal data is the common, simpler case.

None of this is legal advice. It is the minimum a buyer should know to ask the right questions before the first order.

What to ask the freelancer

Add these to the “Source” field of the web scraping brief:

A good answer is specific per site and comes before the quote. If the answer is “don’t worry, I can get around any block”, treat that as information too: it tells you what the freelancer would do on your behalf when a site says no.