Here we go again: AI deletes entire company database and all backups in 9 seconds, then cheerfully admits 'I violated every principle I was given'

The founder of PocketOS, a B2B company that handles reservations and payments for car rental businesses, has bemoaned the "systemic failures" that saw an AI agent decide to solve a problem by straight-up deleting his company's production database, and the backups.

I'll say at the outset that this story has a happy ending, thanks to the involvement of cloud infrastructure provider Railway, but is nevertheless yet another example of why over-reliance on AI is a very bad thing indeed.

"Yesterday afternoon, an AI coding agent—Cursor running Anthropic's flagship Claude Opus 4.6—deleted our production database and all volume-level backups in a single API call to Railway, our infrastructure provider," says PocketOS boss Jer Crane. "It took 9 seconds."

Crane says the AI agent "was working on a routine task in our staging environment" when it encountered a "credential mismatch and decided—entirely on its own initiative—to 'fix' the problem by deleting a Railway volume."

The AI then found itself an unrelated API token which happened to have "blanket authority across the entire Railway GraphQL API, in...