Keeping a nightly posting batch alive long enough to replace it

The batch that posts branch transactions to the general ledger was written in COBOL sometime before anyone still employed at the bank, and the developer who understood it retired with three months' notice. It ran every night, mostly worked, and occasionally overran its window in ways nobody could predict — and a replacement program was 'two years away', which everyone involved understood to be a hopeful number. Someone had to keep it alive in the meantime, and I volunteered before I fully understood what I was volunteering for.

Officially, senior backend developer on the core systems team. In practice, the batch custodian: the person paged when it overran, the reviewer of any change that touched its inputs, and the one who sat with the retiring developer for his last three months transcribing thirty years of context into a wiki. It was a stewardship role more than a building role, which took some adjusting to.

I owned the runbook, which did not exist when I started; the monitoring, which previously consisted of an operator noticing the next job hadn't started; and every one of the eleven changes we made to the batch in two years, each of which I can still recite because each one cost me sleep. I also owned saying no, which was most of the job — the safest change to a system like that is the one you talk someone out of.

The central decision was to instrument around the batch rather than inside it: parsing its job logs and file outputs to build step-level timing and record counts in a dashboard, instead of modifying COBOL to emit metrics. That gave us overrun prediction by 1am rather than discovery at 6am, at zero risk to the thing itself. I turned down two separate proposals to 'quickly rewrite' individual steps in Java, on the grounds that a hybrid batch would double the failure modes while the full replacement was supposedly coming. I did allow one genuine change of substance — a restart-from-step capability using checkpointed file snapshots — because recovering by rerunning from the top was the single biggest cause of missed windows.

In two years the batch missed its window twice, both in the first six months, both recovered before branches opened using the restart capability. Overruns went from a 6am surprise to a 1am prediction with a decision procedure attached. The wiki grew to a hundred-odd pages and onboarded my successor in three weeks instead of three months — because yes, the replacement program was still 'two years away' when I handed the pager on, and for all I know it still is.

The retiring developer, most of all — the wiki is really his knowledge with my formatting. The night operations team fed me every observation about the batch's moods, and one of them caught a bad input file by eye before my monitoring did, which I have never lived down. My successor contributed the best correction: several wiki pages, he pointed out, documented what we believed rather than what the code did.