My batch job was killed halfway by low memory, and it picked up where it stopped

What I expected

One of my scheduled jobs writes drafts in batches. It reads a queue file, takes the next items marked as waiting, works on them, and records each item’s state back in the queue.

Normally it runs once a night, at 02, on its own. In September I started it by hand instead, during the day, and lined up six batches to run back to back, five items each. I expected them to run one after another and finish.

What actually happened

Four batches ran. The fifth was killed partway through, because the PC ran low on memory. The sixth never started.

I did not record which process was using the memory at that moment, so I will not guess at it here. What I know is the outcome: the run ended from the outside, in the middle of its work.

Why that matters more than the cause

A process that is killed from the outside gets no chance to clean up, finish what it was doing, or send a message about it. So whatever the job does when it “fails” is irrelevant in this case. What matters is what state it leaves behind.

This job keeps its progress in the queue file. Each item’s state is written there, in a file on disk, not held only in the job’s memory until the end of a batch. When the process died, the file did not. Every item that had been marked before the kill was still marked afterwards.

So restarting did not require remembering anything. The job read the queue and continued from where it stood.

If the job had kept its progress in memory and written it only at the end of a batch, the restart would have had two bad options: redo work that was already done, or guess what had been finished.

What I changed

1. Resumed from the queue. The job picked up from the state recorded in the queue file.
2. Smaller batches. The batch size went from five items to three. A smaller batch holds less in flight at any moment, and a kill in the middle of it costs less.

What I did not add is anything that predicts memory trouble. The fix was to make a crash cheap, not to forecast it.

One more thing came out of the same day. Reading the job log after the kill was also how I found a separate problem that had been holding drafts back, a malformed field in three drafts that had stopped later batches from moving on. A killed run is a good reason to read the whole log, not just the last line.

What this has to do with hiring someone

Any automation that works through a list will sometimes be stopped halfway: a memory limit, a restart for updates, a laptop lid closed, a hosting provider that ends long runs. A demo never shows it, because demos run short lists on a quiet machine. A computer that goes to sleep is the same problem in another form, covered in what sleep mode does to scheduled jobs.

Three questions to put to a freelancer before you accept a batch job:

• Where is progress written down, and when? The answer you want is “after each item, in a file or table you can see”, not “at the end”.
• What happens if I stop it halfway and start it again? Then do it during acceptance, and check the results: nothing missing, nothing doubled. The duplicate-run check in the acceptance checklist for unattended automations is the same test from a different angle.
• Who notices a run that started but never finished? The job itself cannot. That is the role of a heartbeat check: a small outside watcher that expects a “finished” signal and complains when it does not arrive.

These belong in the failure-behaviour part of the brief. They are cheap to build in at the start and expensive to add after the first lost batch.