Damn I cannot even imagine the stress that admin was feeling after he realised he deleted DB1. He must have aged twenty years.

"You think it's expensive to hire a professional? Wait till you hire an amateur" - some old wise businessman.

The fact they live streamed while trying to restore the data is a truly epic move.

I think the biggest problem (seemingly addressed at 6:21) is the fact they could delete an employee account by spam reporting it.

So you're telling me a platform as big as GitLab went down because one engineer picked the wrong SSH session? Damn that makes me feel way better about my mistakes lol

Given the trouble they were in after the deletion, a recovery time of 24h and a recovery point of 6h is actually pretty heroic. Especially considering the stress they would have been under. 😰

This is why problems like this are actually sometimes good. Of course extremely stressful, but they found sooo many issues and fixed them all. Amazing.

This reminds me of Toy Story, and how like a month before release the entire animation was accidentally deleted, causing absolute panic and hell at Disney. Luckily, one employee had the whole thing on a hard drive that they were taking home to work on. Her initials are on one of the number plates of one of the cars in the film. Always make a backup. Edit: She was a project manager who had to work from home, and the numberplate was actually "Rm Rf" in reference to the notorious line of code that did it.

_Luckily team 1 took a snapshot 6 hours before..._ This happened to me. I copied a clients database to my development environment about 2 hours before they accidently wiped it. They called our company explaining what happened and it got around that I had a copy. Our company looked like a hero that day, and I got a bunch of credit for good luck.

Something my first boss taught me (when I broke something big in production in my first few weeks) is that post mortems are to identify problems in a system and how to prevent them, avoiding blame to individuals. This is huge. Making sure to identify why it was even possible for something like this to happen and how to prevent it in the future is a great way to handle a post mortem like this. Good on the GitLab team.

When I was still a junior developer at some startup company, I was working on a specific PHP online store. Every time we would upgrade the site, we would first do it on Staging, then copy it over to Production. The whole process was kinda annoying as there was no streamlined upgrade flow yet and no documentation anywhere - it was a relatively new project we took over. I have upgraded it before so I knew what to do, and I just did the thing I always did. I was close to finishing it up and we had an office meeting coming up soon and lunch afterwards, so I wanted to be done with this before that - so I rushed a bit. And when I was copying files to Production, I overlooked something - I had also copied the staging config file (that contained database access info etc) to the production location and overwrote the production config file. After the copying had finished, thinking I was finally done, I relaxed and prepared myself for the meeting. As I was closing everything, I also tried refreshing the production site, just to see if it works. And then I realized... Articles weren't appearing, images weren't loading, errors everywhere. Initially I didn't believe this was production at all, probably just localhost or something, RIGHT?? However after re-refreshing it and confirming I had actually broke production, panic set in. Instead of informing anyone, I quietly moved closer to my computer, completely quiet, and started looking at what is wrong - with 100% focus, I don't think I was ever as focused as I was then - I didn't have time to inform anyone, it would only cause unnecessary delays. I had to restore this site ASAP. I remember sweating... the meeting was starting and I remember colleagues asking me "if I am coming" - and I just blurted "ye ye, just checking some things..." completely "calmly" as I was PANICKING to fix the site as soon as possible. Luckily I quickly found the source of the mistake within a minute and had to find a backup config file - and then after recovering the config file, everything was fixed. Followed by a huge sigh of relief. The site must have been down for only around 2 minutes. No one actually noticed what I had done - and I just joined the meeting as if nothing had happened - even though I was sweating and breathing quickly to calm myself down, I hid it pretty well. And this was a long time ago - and still to this day, I still remember that panic very well. Now I always make sure I have quick recovery options available at all times in case something goes wrong - and if possible always automate the upgrade process to minimize human errors

Ugh, felt that "he slammed CTRL+C harder than he ever had before" (3:55). The only thing worse than deleting your own data is deleting everyone else's. In this case the poor guy kinda did both. Great story arc.

The rule I apply for backups is that no one should connect to both a backup server and a primary at the same time, two people should be working together. The employee that was logged on both DBs should have been really two physically separated employees

Imagine for a moment, that you're that guy. That feeling of pure dread and the adrenaline rush immediately after the realization of what you've just done. We've all felt it at some point.

There is an awful lot that could be learned from this. 1) You should "soft delete" i.e. use mv to either rename the data e.g. renaming MyData to something like MyData_old or MyData_backup, or just mv it out of the way so you can restore it later if needed. Don't just rm -rf it from orbit 2) Script all your changes. Everything you need to do should be wrapped in a peer-reviewed script and you just run the script, so that the pre-agreed actions are all that gets done. Do not go off piste, do not just SSH into prod boxes and start flinging arbitrary commands around 3) Change Control - as above 4) If you have Server A and Server B, you should NOT have both shell sessions open on the same machine. Either use a separate machine entirely or - better still - get a buddy to log onto Server A from their end and you get on Server B from yours. Total separation 5) Do not ever just su into root. You use sudo, or some kind of carefully managed solution such as CyberArk to get the root creds when needed

The best practice is to rename the directory or file to something else. Idk how the developers are so calm when using deletion commands

A helpful hack is to set production terminal to red and test terminal to blue or something like that. Just a small helper to avoid human f’ups if you need to run manual commands in sensitive systems.

I once accidentally ran a chmod -R 0777 /var because i've missed a dot before the slash (in a web project with a /var folder), which (as i've now learned) may make a unix system totally unresponsive. I can very well understand how it feels, the moment you realize what you have just done. That did cost us a few hundred euros and kept 2 technicians busy for an afternoon on the weekend. Lessons learned, today we can laugh about it.

One of my most stressful moments as a software designer was when I accidentally broke a test environment right before a meeting with our client; I managed to have the project running at a 2nd test environment but that really taught me the importance of backups and telling the rest of staff about a problem ASAP.

If you are working with multiple shells, VMs, remote sessions or the like - make sure they are color coded based on the machine you are running against! It can be as simple as picking a different color scheme in windows. But it is just too easy to mess up when all the visual difference is a single number, somewhere in the header.

imagine flagging messing with some employee and managing to bring down the entire site by proxy

"rm -rf" is one of those commands I have huge respect for cause it reminds me of looking down the barrel of a gun (or any similar example of your choosing): Best case, you do it a) seldom, b) after a lot of strict and practiced checks, and c) if there's no alternative; unfortunately, the worst case is when you _think_ you're in that best case scenario.

I'm glad they didn't fire the engineer. It goes to show the differences in mindsets from some organizations that care about it being a learning experience (albeit an expensive one). Many corporations would have fired the engineer as soon as the issue was resolved without hesitation. Thanks to those orgs who care about their team members and being more concerned with lessons learned.

Ultimate workplace comeback: "At least I've never nuked the entire database"

I once accidentally deleted 2000 rows in one of my companies production databases, everything was restored 5 minutes later but it felt so bad, can't imagine what deleting an entire database would feel like

For this reason, all our servers have color-coded prompts. Dev/Testing servers are green. Staging is yellow. Prod is bright red. When you enter a shell, you immediately see if you are on a server that is "safe" to mess around with, or not. The advantage to doing this in addition to naming your server something like "am03pddb", is that you don't have to consciously read anything. Doesn't matter if you accidentally SSH into the wrong server. If you meant to SSH into a "safe" server, then the bright red prompt will alert you that you are on prod. And if you meant to SSH into a prod server, then you better take the time to read which server it actually is.

This video is awesome! The step by step analysis of what occurred during the outage coupled with the story telling format helped me learn some things I didn't know about database recovery procedures. Please make more videos in this format!

the realization of what you're doing before it finishes itself is so cruel and happens so often, thats why when you're doing a job you always do it slow but correctly

The real problem here is that you can delete any user data by simply mass reporting him

As an engineer for a large company you got me in the feels talking about asking for help or posting a pr and then seeing all the mistakes you made😊

Your editing is phenomenal. What an insane series of events 😂 Glad gitlab was able to get back to running, seeing all that public documentation was refreshing to see since it shows they were being transparent about their continued mistakes and their recovery process.

A long time ago we implemented a policy that absolutely nobody operates the production console alone. There always has to be someone else looking over your shoulder to point out oversights like the one in the video.

"Slams Ctrl+C harder than he ever had before" As a relatively new linux user, I felt that one.

Our prod server has no staging environment or anything like that. I've asked the DB admin if the data and schema is safe in case of someone accidentally deleting everything and they told me everything is backed up daily. Kinda scared that I don't know how or where this is happening except for a job.

Seen your video history and the evolution of your videos - this format is amazing and you're really good at it :D

The #1 thing I learned WAY EARLY on in my IT career (three decades): Never delete anything you can't _immediately_ put back. Never do anything you can't undo. Instead of deleting the data directory, _rename_ it. If you're on the wrong system, that can easily be fixed. (and on a live db server, that alone will be enough of a mess to clean up.) As for backups, if you aren't actively checking that (a) they've run, (b) they've completed successfully, and (c) they're actually usable... well, this is the shit you end up in. (The fact they're actively hiding ("lying") about this fiasco should be criminal.)

A few places i worked at as a linux admin or engineer, the shell prompts (PS1) were color coded. Green was dev, yellow was qa and red meant your in prod. Worked like a charm.

Awesome work on the video!! I love the editing being both funny and straight to the point, and your narration is easy to understand too. You seriously deserve more attention.

Oh God my heart started sinking when you said he noticed the shell he was running the command in.

When I was just starting in a company, I accidentally deleted all the ticket intervals from the database. Causing all the tickets to close immediately and make some massive spam to the admins. I was really terrified of the situation and didn't know what to do, we didn't have any backup as well. I apologized as much as I can and didn't make another mistake like this again in years, sometimes mistakes make you work harder and be more careful in life.

An old best practice that so many people these days seem to forget or never have heard about is that every week, you try to pull a random file from your backup system, whatever that is. (Or systems, in this case). You will learn SO MUCH about how horribly your backups are structured by doing this - so many people think they set up good backup systems but never continuously test them in any way, and then they get big surprises (like the GitLab team) when they do need to fall back on them.

Testing to verify backups, replication, failover and the like is absolutely critical. As new scenarios occur, having a feedback loop to update the plan is key. It's a continuous process that most shops have learned the hard way. It is boring and tedious but if you don't test you will experience catastrophic consequences.

your content is really good, please keep up making these mini documentaries about tech failures!

What's far more impressive about this whole situation is how calm the engineers were in handling the situation. That to me is far more valuable than having engineers that are too gun-shy to make prod db changes at 12AM and panic when something goes wrong.

As a former Amazonian (only QA for the now-ended Scout program, sadly), I read quite a few cautionary tales on the internal wiki about Wrong Window Syndrome. Sometimes, not even color-coded terminals and "break-glass protocols" (setting certain Very Spicy commands to only be usable if a second user grants the first user a time-limited permission via LDAP groups) is enough to save you from porking a prod database.

respect for not firing the guy, it was obviously just a small mistake, and it wasn't his fault that the backups didn't work. it shouldn't be possible for 1 command to completely delete everything in the first place. Good that they didn't just use him as a scapegoat :p

Lucky guy that he found the data from Manually Snapshot instead of backup. Can't imagine how the admin-1 feeling in that moment.

This reminds me of all the times I have been in the wrong ssh session just before doing something that would have been pretty bad. I setup custom PS2 prompts to tell me exactly what environment, cluster, etc I am in.. and even colorize them accordingly but the problem is.. you start to just ignore them after a while. Its also kinda dangerous when stuff becomes fairly routine that is manual and potentially damaging

Yesterday I was added to a support team because we are getting a lot of tickets from users not waiting long enough for a service to load and closing the connection early. I died laughing from this story.

all things aside, that wasn't that bad. Yeah, they weren't operational for 24h, but that made many other companies realize their fault management. For example, my uni professor told us about this incident and we could comprehend the importance of backups and testing

Even before I started working in one company, one IT specialist deleted the directories of the new CC-supporting system. This was shortly after its implementation into production. Worse still, it turned out that the backup process was not working properly. For a week, the team responsible for programming this system practically stayed at work, recreating the environment almost from scratch. :D

One idea to help prevent this is setting up the ssh sessions so each one has a different fore/background color, lets say the prod machine has green foreground and the backup is blue and make it standard for everyone working with the terminals, that way it'll be harder to get confused between the two. you can even have multiple ssh terminals and assign each one a different foreground color.

I can just imagine the relief that team felt when they find SOMETHING that they could use to restore files.

One of my first jobs in IT was working as a big data admin and this video allows me to re-live the spicy moments of that job but with none of the responsibility attached

One thing I learnt all this is never run delete command ever, and if you are paste the screenshot of command in your group before running it

Mistakes in the moment happen. I'm focusing more on the "we thought things were working as expected" parts. The backup process familiarity, backups not going to S3, Postgres version mismatches, insufficient WALs space, alert email failures, diligence on abuse deletes... These were all things that could have been and should have been caught way before the actual incident.

In my vocational school I had a subject simply called "Databases" and our teacher there once told us a story about how one of his co-workers lost his job. In essence he did everything right, created his backups and backup scripts and everything worked. At some point during the lifetime of the server this was running on someone replaced a harddrive for whatever reason, this lead to a change of the device UUID, which he had hard-coded into his backup script, when the main database failed a year or two later, they tried restoring from this backup only to find that there was none. Wasn't even really his fault, the only mistake he made was not implementing enough fail-saves. Nowadays we have it comparatively easy with all the automatic monitoring and notifications, but this was at least 30 years ago.

i cant even imagine the sheer terror Team Member 1 felt when he saw the db in which he ran the delete command

I know the exact feeling of terror the moment you realize the command you just ran has is about to cause havoc

Great video! Well produced content about software engineering war/horror stories are exactly what I’ve been looking for, keep it up!

As a dev for a large company who has been on a number of late night calls, I literally gasped at this. But good on the team to work through the issue, and good on management to keep these guys around

At my previous job anything done on production servers required a change request which takes about a week to get approved and complex commands had to be tested on Lab environment before they could be copy pasted to production server.

I must say, I heard many stories about this.. but that was a very nice summary of the nitty-gritty details, thank you. (:

A couple of jobs ago, I had a colleague who managed to do worse than this. I think they were playing about with learning Terraform and managed to delete the entire account. Prod servers, databases, the dev/qa servers, disk images, even the backups. Luckily it was a smaller account hosting a handful of tiny trivial legacy sites, but even so, we didn't see them for the rest of the week after that mishap

Having a live screenshare with team members watching might seem a little wasteful. But for critical procedures like this, it is well worth the added cost.

The story telling/edit is unmatched. Hands down best docu/short movie on youtube😂!

Nice to hear that they didn't fire him. He did the correct procedure, some of the steps were unknown like the lag caused by the command, which could have been avoided by having clear documentation about it. Also when people are tired late at night, mistakes do happen, which anyone can be the victim of.

This video made me say "Oh... my... God..." way too many times 😂😂. Felt like some Chernobyl documentary about a bad sequence of actions. Love it! This is very insightful as to what things can take place on these types of environments as well as what are some measures that can prevent major falis like that. It's also super interesting to see that, no matter how perfect a software system is, humans will still find a way to screw it up 😂

I didn’t realize I would like these videos, but you are a good storyteller for production issues and I hope to see more in the future I am gonna share this with some of my coworkers

OMG, we have all been there haven't we? That awful, dreadful realization after deleting something that you shouldn't have. Mine was back in the days of manual code backups, before ALM tools were ubiquitous like today. I thought I had taken the last three days of code changes and overwritten the old backups that were no longer needed. And then I realized that I had done the exact opposite, and just deleted three complete days of coding - and would now have to recreate them from scratch 😒😭

By far the most suspense I have felt during a dev story

Wow! This was great and so interesting. I'm so glad I found this channel. I would love to hear more in depth analysis of software engineering fails

This gives me good insight on why our tech team keeps breaking shit….

All my backup jobs have to report to an uptime service.

Linux actually can in certain circumstances "undo" this wild kind of situation. Having ZFS as the file system will allow you to revert to a previous image of the filesystem. it's like versioning but for the entire file system. of course it takes up quite a bit of space so it's not done that often, software install are automated "imaging" points for instance. but you can trigger one manually when you think you're about to do something you're unsure about. (since the selection of save states is at GRUB, yes an unbootable system is still recoverable if you still have GRUB)

I barely understand anything here, but all I can say is massive thanks to the team who have worked hard, advancing our computer tech to the current state we have!

Honestly the worst part of this was all the backup failures

The fact that running a raw rm command on the backup server was anywhere close to an expected procedure is also a problem, like you could make an alias for that specific rm command called "wipe_backup" or something, only on the backup server, so that trying to run the command in the wrong SSH window would just error out with "command not found". But anyway, good on them for not firing the guy and realizing such a catastrophic failure is never a single person's fault

This is my worst nightmare. I was on a release where a database administrator did something similar and he spent a good 10 minutes swearing. We had backups but it was a pain to find which one to use and get it setup.

I’m pretty sure this event only ended up affecting things like comments and issues, but not the actual git repositories themselves, which would have been a huge relief, I imagine. Still, this was one of the most interesting things I’ve ever followed and ended up motivating me to learn a ton about databases, cloud practices, devops, and everything-as-code culture. Thanks for providing such a great lesson, GL. And huge kudos to them for transparency

3:52 it was at this moment when the viewers collectively scream, transcending space-time and raising a cosmic choir of dread and regret.

This reminds me of when I was working at another company's site back in the late 80s. One of the company's employees had logged into a VMS system and then connected to a Unix system from there. At some point, the employee accidentally started a 'rm -rf /' command. He realized the mistake and did a ^C. Unfortunately, all that did was log him out of the VMS system. The command continued to run on the Unix system until it finally removed something important enough so as to crash the system. It took a few days to restore their computer. And that also resulted in the removal of Admin privileges to most employees. Funnily enough, I was able to keep my Admin privilege, not being an employee.

thats why you always use a shell that says "hey your rm -rf'ing something, you sure about that?"

I've always been paranoid when working in Prod. Always make it a point to have at least the Ops Lead on a screen-sharing session where I show what I'm doing while requesting affirmative acknowledgement of each step before proceeding. It's annoying. It's slow. But boy ohh boy does it make me feel safer.

This is so cool to know the inner workings of a team like this

I am working as a bank programmer and we have two important servers in production. One is in sync with the main one. If the main one is broken or something does not work properly we change it to the other one. Also they have many ways to backup like multiple storage units and maximum security of who has access to data. We had some issues on testing platform where a guy accidentally deleted the database but we had backup in less than 30 min made by our sys admin guy. We did not ever have any tragic issue on production.

lol, that's why for the production database CLI, I set the background color of my terminal to red to make sure I don't confuse it with the staging environment. But I also use Amazon RDS and let it manage backup for me, because I know setting up that stuff is a PITA, particularly because you'll need to test the replication, failover, the rollback logs, the snapshotting, to make sure all your config actually works.

One strict rule I always follow when connecting to prd servers via ssh or DB UI agent (pgadmin) is I always use different background colors, Red for prod Green for staging Black for test and local + double checking every command You can never be sure enough

I like how the terminal has the decoration of some linux-y windowmanager, but the message boxes are winXP xD

I deleted the main site from our backend in my first month as a full stack developer. Fortunately i figured out how to rebuild the apache server and clone the repository but i definitely worked well past my hours that day and the stress was crazy

I experienced something similar a couple years ago, it's the kind of thing that you think only can happen to others but yeah... I had to delete some specific data from the production database, I created the sql requests and executed them to the testing environment. The dataset between those databases is completely different, and the requests passed without any issue. But when I passed them to production they were taking way too long and then I realised. I almost had a panic attack. I reported the incident immediately and was mentally prepared to be fired. Fortunately we could retrieve most data from a backup and the lost ones were not that big of an issue. I still work in the same company :p

And yes... this is exactly the reason why I didn't study programming / engineering in college, and instead opted for graphic design / communication. if I write or design something wrong and it gets published, well, at worst the publication stays published as a reminder of my mistake, in programming all it takes is one finger mistake, misremembering something or just a simple distraction and you can absolutely wipe an entire company's network infrastructure out of existence.

We need more videos like this one. This was amazingly interesting

A long time ago, in our thriving software shop, on a corporate network of 50 or so SGI workstations and some heavier iron a script's rm -rf line accidentally picked up a space character after one of the leading '/'s of a file name. As all disks were remotely mounted, this became a corporate total deletion, after midnight, with the main server room locked tight.

Also one thing I used to scoff at when I was a newbie was assigning names as aliases to your servers. Like: actual words instead of numbers. It seemed a little asinine to me at first but even in this scenario: it's much easier to confuse db1 and db2 than, eg.: amelie and betrand.

Can't wait for part 2

Wow the amount of stuff i learned here is huge, please make more reviews like these i subscribed and turned on notifications please don't disappoint me

This genuinely made my day.

My only tech lead yet told me almost the exact same story after i got stressed out after breaking some layout in our production website. Instantly destressed and was a great intro to Software Engineering :D

absolute nightmare. loved every min of this