Backups haven't gone away

A news story flashed by in my feed and grabbed my attention. A dev returns to work to find the production database gone, and after a little prodding, their AI coyly admits to deleting it. Well, it tries to hide it first before ‘fessing up. If you want all of the gory details PCGamer has them.

I’m not going to talk about the AI, except to congratulate it for apparently achieving a nepo-intern level of incompetence and evasiveness. The dev was vibe coding with the AI so I’m really not sure who to point the finger at.

An an operations guy, my concerns were in the fallout. Apparently, restoring their database was a big, big issue. They couldn’t just roll back the commands, the AI dropped the tables. Erm, call me old school, but is rolling back the commands your recovery strategy there?

And your code-assistance AI had production database access?

The entire point of dev environments is so that the intern can annhiliate everything in sight and no real harm is done. Your code can fork-bomb or go postal or write its favourite log line over and over and over and nothing of consequence is broken.

The most troubling aspect of this is the direct production access. Yes, devs should be trusted with production if they act as support. Separating them entirely simply gives you the burden of training support people in code-level detail. Naah, don’t to that. Hire devs you can trust with production access but teach them the rules of prod. Authorised changes and incident handling only. Document everything. You know the drill.

As AI has burst onto the code scene, and every scene, it’s starting to look like a lot of the old discipline is being sidelined because ‘AI will fix it’. But unless you give your AI enough access to do exactly what Replit did, you still need to do your own backups. And load testing. And failover testing. And disaster recovery drills.

You need to train your system operators to do these tasks, and to understand how to treat a production environment such that it would never occur to them to give an AI that access.

But even so, accidents happen. Don’t wait for one to happen to realise, yeah, we really should have backed up that database.

There, I let AI draw something

If you want to understand better how to take care of prod, do not hesitate to get in touch.