I published 437 reviews. There were 317: the page had loaded some of them twice

What I expected

For this site I wanted to know what buyers actually say about automation freelancers after an order: how much they paid, how long they waited, whether they came back. Star ratings alone say little, so I read the reviews themselves.

The collector was simple. It opened each seller’s gig page in a browser, clicked the page’s Show More Reviews button a few times to load more of them, then read every review on the page: rating, price band, duration and a label the page shows when a buyer has ordered before.

I ran it on the eleven sellers of the workflow automation shortlist and published the result on September 26: 437 reviews, 94% of them five stars, and “exactly half” from repeat clients. The collector ran without errors, and nothing in its output flagged a problem.

What actually happened

Later that day I extended the same collection to sellers in two other categories, and found that the same reviews could appear twice in what was collected. Clicking Show More had loaded some reviews a second time, and the collector had kept both copies. On one gig, half of what it collected were duplicates: 40 reviews that were really 20.

While looking at duplicates, I found a second counting error. The repeat-client label is shown on every review a returning buyer leaves. One buyer with many orders leaves many labelled reviews. I had counted labels, not buyers. On one chatbot gig, 35 reviews came from only 14 buyers, and one of them had left 15.

Figure	First published	After recount
Reviews read	437	317
Repeat buyers	“exactly half” (219 labelled reviews)	42 of 254 different buyers (17%), who left 33% of the reviews
Share of 5-star reviews	94%	94%

The one headline number that held was the share of five-star reviews. The per-seller table changed the most, because repeat-buyer shares had been computed from labels.

Why

Nothing in the code crashed, and nothing in the page was hidden. Two assumptions were wrong:

1. “Show More adds new reviews.” It does not always. The page can hand back reviews it has already shown. A collector that appends whatever appears will count them again.
2. “One label means one repeat buyer.” The label belongs to the review, not to the person. Counting labels measures how chatty repeat buyers are, not how many there are.

Both are about how the source behaves, not about my code. That is also why neither was visible in a test run: the collector did exactly what I told it to do.

What I changed

• Duplicates removed by review text inside the collector, before anything is counted.
• Buyers counted as people. Each review is tied to its buyer, and repeat buyers are the ones with two or more reviews.
• A fixed sample rule, written down. Up to six clicks of Show More, at most 35 reviews per seller, in the order the site shows them, stated as not random.
• Fresh page for every run. Running the collector again on a page that was already expanded gave a different sample size, so each run now starts from a newly loaded page.
• All 26 sellers recollected across the three categories, and the automation page rewritten with the new numbers.

The correction went out the same night, with a note at the end of the article saying what the first version claimed, what was wrong and when it was corrected. I left the note visible. Readers who saw the old number deserve to know it changed.

What this has to do with hiring someone

If you hire someone to collect data, whether reviews, leads, prices or listings, this is the failure you are least likely to catch at delivery. The file arrives, the row count looks plausible, and the numbers are wrong in a way nobody sees until they are compared with something outside the data set.

Three questions catch most of it before you rely on the numbers:

• How many duplicates did you remove, and by what rule? “None” is a possible answer. “I didn’t check” is not.
• Are you counting people or rows? Customers, buyers and users are people. Orders, reviews and labels are rows. Ask which one each figure in the report is.
• How was the sample taken? Which pages, in what order, how far down, and is it random or the site’s own ordering?

Ask for a small spot check in acceptance too: pick five rows and compare them with the source by hand. It is the same idea as the acceptance checklist for unattended automations: test the result against the world, not against the job’s own report. A data job that looks correct from the inside needs something outside it to check against. That is also why “the source behaved differently than we assumed” appears among the breakages in maintenance terms. Finding the cause takes investigation, not a quick edit.