One malformed field held back three batches of drafts

What I expected

My content pipeline has a scheduled job that writes drafts in batches. Each draft goes through a gate of fixed rules. After the gate, the job builds everything together, all the unpublished drafts at once, and only if that build succeeds does it pass the drafts on to a separate checking step and then to scheduling.

That combined build is there on purpose. It is the last point where a broken file can be caught before it goes anywhere. My expectation was simple: a draft that passes the gate moves on, and a draft that fails the gate is held back. One bad draft should cost me one draft.

What actually happened

In September I ran several batches in a row. The drafts kept passing the gate. None of them moved on.

The cause was in three drafts. Each had written its list of sources as plain web addresses. The format the job expects is different: every source has to be an object, not a bare address string. To a person reading the file, both versions look fine. To the build, the plain address is the wrong shape, and the build that includes all drafts failed.

Because that build is the door to the next step, its failure held back everything behind it. The three batches after the bad drafts, about 15 drafts in total, passed the gate and then stopped. They were not checked and they were not scheduled. They were not broken. They were simply built together with three drafts that were.

How I found it

Not through an alert. The same run was later killed partway through by low memory, which is a separate incident. Afterwards I read through the job log to see where it had stopped, and that is where the build failure was. The line the job had recorded was this:

draft build exit=-1073740791

On its own, that number tells you nothing about the cause. It says the build ended badly, not which file or which field.

That is the part I would change first if I were briefing someone else. The failure was real, it was holding back a meaningful amount of work, and the only place it existed was a log file that nobody was asked to read.

Why it got that far

The gate and the build were looking at different things. The gate checked each draft against its own rules. The format of the sources list was not one of them; it was left to the build. So a draft could pass the gate with a malformed field, and the only check that knew about the field was the one that runs on every draft at once.

That is a common shape for a pipeline: cheap checks first, one expensive full check at the end. It works until a single bad item meets a step that treats the whole batch as one unit.

What I changed

1. Fixed the three drafts. Each source is now written as an object, as the format expects.
2. Wrote the rule into the instructions. The instructions the writing step works from now say that every source must be an object and never a plain address. The rule sits at the point where the data is produced, not only at the point where it is rejected.

What I did not change: the build still includes all drafts, and a failed build still holds all of them. That protects everything downstream. The cost is the one I found: one bad item can delay many good ones, so the failure needs to be seen quickly.

What this has to do with hiring someone

Most business automations process things in batches: rows from a spreadsheet, orders from a shop, emails from an inbox, invoices from a folder. Sooner or later one item arrives in a slightly different shape. What happens to the other items at that moment is a design decision, and it is rarely discussed before the build.

There are two reasonable answers. The bad item is set aside, named, and the rest go through. Or the whole batch stops, because a partial result would be worse than none. Either can be right. What is not acceptable is the default nobody chose: the batch stops and the only trace is a line in a log.

Three questions are worth asking a freelancer before the build:

• If one record is malformed, what happens to the others? Agree the behaviour in writing.
• Who hears about it, and how? A failure that holds back work should reach a person. The error alerts brief covers what a useful alert says.
• Where is the format enforced? The earlier a bad record is caught, the fewer good records wait behind it.

You can test this at acceptance by feeding the automation one deliberately broken record alongside good ones and watching what happens to the good ones. It fits next to the forced-failure test in the acceptance checklist for unattended automations. And if the job runs on a schedule with nobody watching, a heartbeat check is the outside view that notices when the expected result never shows up.