The production failures nobody flagged
Backups that had stopped, logins that could never arrive, PDFs that only worked locally. None of them raised an alert. What OES found on two live insurance platforms, and three checks for your own.
, updated , filed under production, operations, reliability, delivery, aws
Most founders running a live product have a dashboard that’s green and a nagging doubt about what it isn’t showing them. That doubt is usually right. The failures that hurt are the quiet ones: nothing crashes, no alert goes off, and the thing that was meant to happen simply doesn’t.
OES took on two live insurance platforms this year, running releases on one and preparing the other for launch. On both, it found failures that had been there before OES arrived, that nothing was reporting, and that nobody had flagged. Here are three of them, and what each one says about where to look.
Off-site backups had stopped
A UK rental deposit-replacement insurer had off-site backups in place. On paper, the platform was covered. In practice, the backups had stopped running, and the off-site store they were meant to land in was empty. Nothing had failed loudly. The job just wasn’t doing its job.
OES got the backups running again with two small scheduled functions and, the part that matters, restored a real dump to prove they could be used, rather than trusting that the job had run.
A backup nobody has restored is a hope, not a backup.
Every magic-link login was failing
On an insurance quoting platform, users sign in with a link sent by email. While setting up production for user acceptance testing, OES found that the production email configuration didn’t match what the app expected. Every magic-link login would fail, silently. The sign-in form accepted the address, said a link was on its way, and the link never arrived.
It was caught before the testers depended on it. Left alone, the first sign of it would have been users who couldn’t get in and assumed the product was broken.
PDF generation broke in production
On the deposit-replacement platform, policy documents were generated as PDFs by running a headless browser. It worked on every developer’s machine and failed in production, because the production containers run with a read-only filesystem for security, and the browser needed to write to disk.
OES replaced it with a library that builds the PDFs directly. Production no longer needs a browser at all, and the security setting stays as it should be.
The pattern
Every one of these looked like success from the outside. A backup job that ran. A login form that said the email was sent. A document feature that worked in development. The failure was always in the step after the one anybody was watching.
None of them needed a big investigation. They needed someone to check the outcome rather than the activity.
Three checks for your own product
- Restore a backup. Not “check the job ran”: take a recent backup and restore it somewhere. If nobody has done that, you don’t know you have one.
- Follow a real user journey in production, end to end. Sign up, sign in, buy, and check every email and document that should arrive.
- Treat “it reported success” as a claim to verify. Jobs, emails, generated files: check what actually landed.
If you’ve got a live product and a nagging feeling about what nobody’s checking, let’s talk.