My scheduled posts stopped for two days, and every job reported success

What I expected

At the time, my social-media publishing job ran twice a day, at 12 and 21. Each run prepares the next post and publishes it. To do that, it first uploads the images and the video to a code-hosting service, because the social platform needs a public web address to fetch them from. That upload step uses a login token.

Around the job, I had already built two safety nets after earlier incidents:

• A make-up step. After each slot, it reads the log. If the slot came up empty because of a temporary problem, such as a network drop, it schedules one more attempt 15 minutes later.
• A watchdog. Every morning it checks that each slot actually ran after its scheduled time, and messages me only when something looks wrong.

So I expected that a failed slot would either be retried or reported.

What actually happened

Starting on the evening of September 21, no posts went out. Every layer said it was fine:

Slot	Posts published	Scheduled run result	Make-up scheduled	Watchdog
Sep 21, 21	0	success	no	no alert
Sep 22, 12	0	success	no	no alert
Sep 22, 21	0	success	no	no alert
Sep 23, 12	0	success	no	no alert
Sep 23, 21	0	success	no	no alert

The upload step was being refused with HTTP 401, meaning the service no longer accepted the token. The same token was used by a separate job that backs up my files to the same service. That one had stopped too, and four backups were waiting to be pushed.

Why every layer stayed green

Each safety net did exactly what it was designed to do. That was the problem.

The scheduler only sees how the job exits. The wrapper script around each slot finished normally, so the scheduler recorded success.

The make-up step separates temporary errors from permanent ones on purpose. A network drop is worth retrying in 15 minutes. A 401 is not: it will be the same 15 minutes later, and retrying it just uses up time before the next slot. I had designed it that way in August, after a slot was lost to a network error that retried for only ten seconds. So for a 401 it correctly scheduled nothing. At that point, the only alert it could send was when it had already used up its two make-up attempts for the day, and a permanent error never got that far.

The watchdog checks whether each slot ran after its scheduled time. Each slot did run. It just published nothing.

So the one error that could not fix itself was the one that no layer escalated. The information was in the log the whole time; nobody reads that log unless something tells them to.

What I changed

On September 23 I added two things:

1. An alert on every permanent failure. When a slot ends with an error that will not go away on its own, such as a 401 or a permissions error, the make-up step now sends a message to my phone straight away, for every slot, including the command that fixes it, so the message is useful away from the computer, not just “check the log”.
2. A direct login check. The morning watchdog now runs the service’s own login status check every day at 08. It no longer relies only on the slots to reveal a bad token.

The retry rules themselves did not change. Separating temporary from permanent errors is still right. What was missing was the second half of the rule: a permanent error is a person’s job, so a person must hear about it.

What this has to do with hiring someone

If a freelancer had delivered this job to me, it would have passed a normal demo, and even a week of watching, as long as the token stayed valid. The weakness only shows when a login stops working, and logins always stop working eventually.

That is why “run once with an expired or revoked token” is one of the checks in the acceptance checklist for unattended automations, and why credentials with an expiry belong in the maintenance terms. When you brief a workflow automation, ask two questions in writing: what happens when a login stops working, and who hears about it, and how fast? A good answer names the alert, the person who receives it and the time. A job that retries quietly and exits with success is not an answer, even if every piece of it works as designed.