Make it Hard to Screw Up

It was Friday evening. I was upstairs at my father’s house in rural Pennsylvania, (barely) connected to the internet with my Verizon hotspot. An ancient window AC wheezed a trickle of coolish air into the steamy August night. One more task and I could go down and get my wheelchair-bound father out to his favorite restaurant.

I had a database explorer utility open in a few tabs - one pointed at the Production data, another at a sandbox where I had been debugging a process that was ingesting and processing data.

The OOPS!

With the process finalized, I swapped back to the utility and hit the Execute Query button to delete the data. Except, it was the wrong window! What I deleted was 1/3rd of the Production database, not the sandbox.

Flop sweat.

Panic.

Difficulty breathing.

I called the CTO & founder, and the only other developer at our 4-man startup. After I explained the problem, he reminded me that we could process all the raw data again and restore the missing data overnight. Since it was Friday, the customer wouldn't really miss it since it was the weekend. (Best boss ever - he did it with no yelling, no recriminations, and no panic)

Stress relieved

The Lessons

Two big lessons out of this.

First, always have repeatable, non-destructive, non-duplicative repair processes ready to go. Build and test those as you go, not as a panic-driven add-on. We had done that, and it saved my weekend (and fended off a stomach ulcer...)

Second, make it hard to mess up like this. My penchant for having a lot of browser tabs open, combined with the database utility not being able to boldly display precisely which datastore I was referencing, helped me mess up. So ...

• Never leave a command to destroy data sitting there, queued up and ready to kill.

• Better yet, script all data deletions, and have every execution of it recorded in a log.

• Put Production data access in a different browser with a different browser color scheme

• Make special users for deleting data. Let the usual id you log in with explore, update, and create, but not delete.

With these safeguards in place, you'll enjoy an uninterrupted evening with your family, instead of a sweat-filled, ulcer-inducing crawl through your data fixing a bad mistake.