Dev Loses $440 Million in 28 minutes, Chaos Ensues

Рет қаралды 195,876

Күн бұрын

In this video, we take a deep dive into the disaster that occurred at Knight Capital Group, an American global financial services firm engaging in market making, electronic execution, and institutional sales and trading, on August 1st 2012. Whether you're a pen tester, security researcher, software engineer, or cyber security expert, having a solid foundation of test automation, DevOps, and automated deployment practices are critical.
0:00 - Prologue
1:06 - Overview
2:42 - Dark Pools
4:28 - RLP
5:14 - SEC Approval
5:43 - SMARS
WE HAVE A DISCORD NOW! / discord
Doug Seven's report:
dougseven.com/2014/04/17/knig...
SEC filing:
www.sec.gov/files/litigation/...
Henrico Dolfing's report:
www.henricodolfing.com/2019/0...
MUSIC CREDITS:
LEMMiNO - Cipher
• LEMMiNO - Cipher (BGM)
CC BY-SA 4.0
LEMMiNO - Firecracker
• LEMMiNO - Nocturnal (BGM)
CC BY-SA 4.0
LEMMiNO - Nocturnal
• LEMMiNO - Nocturnal (BGM)
CC BY-SA 4.0
LEMMiNO - Siberian
• LEMMiNO - Nocturnal (BGM)
CC BY-SA 4.0
#programming #software #softwareengineering #computerscience #code #programminglanguage #softwaredevelopment #hacking #hack #cybersecurity #exploit #tracking #softwareengineer #vulnerability #pentesting #privacy #spyware #malware #cyber #cyberattack #bugbounties #ethicalhacking #lowlevelsecurity #zeroday #zeroday #cybersecurityexplained #bugbounty #injection #breach #Zoom #knightcapital #marketmaking #devops #deployment #databreach #testautomation #QA #disasterrecovery

Пікірлер: 602

@DanielBoctor 5 ай бұрын

Thanks for watching! I thought I would try something slightly different with this video, focusing a bit more on telling a story. I had a lot of fun with it. Open to feedback from y'all, as well as suggestions for future videos (vulnerabilities, breaches, exploits, anything really). I'm doing a bit of travelling next week, so it might be a bit longer until my next upload. JOIN THE COMMUNITY ➤ discord.gg/WYqqp7DXbm Also, I think I finally fixed my intonations LOL Thank you for all of the support, I love all of you ♥ *EDIT* - At 2:23 those timestamps were meant to be 9:35am, my apologize for the mistake. I thought I fixed it, however I must have ended up uploading the wrong render. *EDIT #2* - 2:00 should have been "two and a half minutes", rather than seconds. Thanks to those who pointed this out!

@ImperialRoads 5 ай бұрын

I love pegging

@BillAnt 4 ай бұрын

Fascinating, love it. :)

@DanielBoctor 4 ай бұрын

@@BillAnt love you more

@renakunisaki 3 ай бұрын

I enjoyed it. Good pace and explanations.

@stockstreamtwitch 2 ай бұрын

Great work dude. Glad this video fell into my recommended.

@sanderbos4243 5 ай бұрын

Imagine the stress of the engineers trying to identify the problem, knowing their company is losing 2.5 MILLION dollars per second

@DanielBoctor 5 ай бұрын

At least they ordered pizza LOL

@BillAnt 4 ай бұрын

@@DanielBoctor- Give new meaning to "When the sh*t hits the server fan". Ha-Ha

@supersat 3 ай бұрын

I'm a little surprised NYSE doesn't have a mechanism to block trades when something is obviously going wrong

@DanielBoctor 3 ай бұрын

They actually do, however they were not that helpful for Knight as they were designed for price swings, not trading volume. Mary Schapiro (the SEC chairman at the time) did end up reversing 6 of Knight's transactions, as they reached the cancellation thresholds outlined below: The SEC required more specific conditions governing the cancellation of trades. For events involving between five and 20 stocks, trades could be cancelled if they were at least 10 percent away from the “reference price,” the last sale before pricing was disrupted; for events involving more than 20 stocks, trades could be cancelled if they deviated more than 30 percent from the reference price. You can read more about this at Henrico Dolfing's report linked in my description.

@cambrown5777 3 ай бұрын

It's actually $292,000 per second, if the title is correct (440M / (28*60)). Still absurd.

@zdrux 5 ай бұрын

At first I thought, how could anyone be this stupid?.. Then I got to the point in video where they were given a month to design and deploy a whole new piece of software, and everything made sense.

@DanielBoctor 5 ай бұрын

yep, that will do it

@ghutkamukesh 5 ай бұрын

@@DanielBoctor😂😂😂

@paraax 3 ай бұрын

Yes, getting the software wrong is understandable. Not knowing how to turn your software off is however not understandable. The new routines worked within their existing framework. Deploying and decommissioning should be literally one of the first things they learned. Turn it off, and don't let it out of test mode again until you are sure it works.

@exponentialcomplexity3051 3 ай бұрын

@@paraax I am still confused. Why couldn't they shut it down? Just pull the plug in worst case no?

@BAmalakas 3 ай бұрын

@@exponentialcomplexity3051distributed system

@orterves 3 ай бұрын

Blaming the devs for losing the money when the company pushed for the release in a month, through procedures that involved manual unverified deployments, classic.

@spacemanmat 3 ай бұрын

You know that Devs complained and weren’t listened to. Would be surprised if the company had a history of doing this. This time it the company paid the price.

@tr7zw 3 ай бұрын

I mean, it was a failure on all fronts. 1 Month to implement this, no kill switch, broken deploy scripts, at that point 7-year-old dead and dangerous legacy code in the codebase, being able to "reuse" a flag that causes "dead" code to revive, no plans in case of emergencies... There's a lot at fault here.

@snorman1911 3 ай бұрын

Similar to my company. Each rush job builds on the tech debt of the previous rush job, with the whole system getting worse each time. Then management demands to know why everything doesn't work optimally. Every objection is met with "If we don't get this out by next week, we'll miss the market!"

@Gideonrex1 3 ай бұрын

I’m a dev ops engineer and my employer made me delete our dev environment bc he didn’t see how it was needed and was costing money. So I could see a company literally just having prod and the devs have no say.

@voidspirit111 3 ай бұрын

@@tr7zwthere was a kill switch. There allways is. They just wantes to.keep operationa going. The handling of thw situation ia more of a management and risk management failiure. But in America managament is rarely blamed. Based on the story they wanted to recover while being online so they don't lose face as a MM. They always had a hardware kill switch. A server kill switch would.have been a nicer option. I say it's a crisis management issue, because they were doing live debugging and troubleshooting while losing so much money and apperently nobody in the chain of command said " Stop take it offline". Beaides the fact that they had to.ve contacted from outside... like.. nobody was supervising that???

@LordHonkInc 3 ай бұрын

If you're losing ~$150M a minute, there is a kill-switch. It's called the server room breaker

@ratulsaha9487 2 ай бұрын

Exactly. Don't know why they didn't shut down all operations until they figured it out. They wanted to continue business as usual and ended up losing the entire company. What a bunch of dim wits.

@eyeofthepyramid2596 2 ай бұрын

What does that do, never heard of it ?

@mortyrosenstein4211 2 ай бұрын

It’s the power circuit for the room. A light switch essentially. You just literally turn off the power for everything in the server room so the servers immediately stop melting the company down. The most basic and sure fire way to stop the problem. You just hit the big power button.

@astr0man573 2 ай бұрын

These days it would be in the cloud and no one would have the credentials to nuke the account 😂

@AccountInactive 2 ай бұрын

@@eyeofthepyramid2596Same as the breaker in your home. Turns off power to any given room or circuit (like laundry machines or stove)

@TheEnlightenedMalignancy 3 ай бұрын

Heavy engineer: “It costs thirty million dollars to run my code for twelve seconds”

@slomnim 3 ай бұрын

Put that on your resume xD

@TheNefastor 3 ай бұрын

Underrated comment ! 😂

@Fossil_Frank 3 ай бұрын

Well, on a good day it probably earns them more than that. It's part of the risk game, you accept having a robot making transactions for you and earning money at lightspeed, so you have to accept that sometimes it might lose money at lightspeed too.

@JohnSmith-ox3gy 2 ай бұрын

@@Fossil_Frank Just no. These are very low margin services. In 45 minutes they spent 4 years of profits of the entire thousand employee business. Your comment is off by a factor of over 1000.

@Fossil_Frank 2 ай бұрын

@@JohnSmith-ox3gy I don't know where you're from, but on my home turf any kind of stock transaction costs a hefty premium to perform. Granted, these kind of automated services probably discount it for the customers, but those customers are more than likely exclusively high rollers who bring in truck loads of money. I find it hard to belive they would charge them flat rates.

@KandyWrongIncognito 3 ай бұрын

The part where the engineers were engaged in live debugging on a production system made me cringe into the next dimension. That's like trying to perform open heart surgery on a marathon runner as they're running the race. What an absolute disaster. Great video.

@DanielBoctor 3 ай бұрын

well said. glad you liked it

@Entropy67 3 ай бұрын

it must have been torture, considering that there was no bug in their new code, it was a deployment issue 🤣

@gingeral253 3 ай бұрын

@@Entropy67The worst situation. Everything looks like it’s right and the problem turns out to be somewhere you never looked.

@thomquiri9860 3 ай бұрын

not gonna lie, that's a new fear unlocked for me as a future software engineer

@LesserAndrew 2 ай бұрын

I'm surprised they couldn't figure out how to kill the servers. Once, our CTO fixed a production issue by driving to the colo and unplugging a network connection.

@drakerp2 3 ай бұрын

"There is no kill switch," my man, unplug the server, throw a bucket of water or a cup of coffee on the buildings circuit breaker, litterally anything will cost a lot less than 2.3 million to fix.

@EntityVsEntityInteractions 2 ай бұрын

2.3 BILLION*

@baktru 2 ай бұрын

You're assuming you have physical access to the servers involved. In the stock exchange world, most likely in 2012 you don't. Those servers were likely co-located in NYSE's datacenter.

@Hunter_Bidens_Crackpipe_ 2 ай бұрын

@@baktrulog in with a console and shut down the server or the app instance.

@CycloidalHeadache 2 ай бұрын

@@Hunter_Bidens_Crackpipe_yeah, hunter bidens crack pipe, I’m sure if you were there with your infinite wisdom, this never would’ve happened

@pepega4844 2 ай бұрын

@@baktru very likely. Im sure theres SEC rules that has regulations about where these systems can be held and likely has a secure facility access level requirement to run it. By the time they made it to the servers on site, they would have been bankrupt. I'm also sure it's probably not a server you can shut off remotely either even if you intentionally wanted to. They want those servers up 24/7 and isolated.

@wieliewiel2630 3 ай бұрын

"I don't always test my code, but when I do.. it's in Production" - 😅

@thenayancat8802 4 ай бұрын

Seems like the "buy high sell low" code is not something you want on your production machines, but then I'm not a financial expert like these folks

@JacobSantosDev 2 ай бұрын

Well, they did have it behind a feature flag. But reusing a feature flag was a huge mistake.

@thenayancat8802 2 ай бұрын

@@JacobSantosDev Again seems like just simply not pushing the test code to production machines is the safer option, but I'm not a financial expert/CS wizard like they are

@JacobSantosDev 2 ай бұрын

@@thenayancat8802 oh sorry. The entire purpose of a feature flag is to be able to turn on and off features that you are testing in production. Just because you tested something in other environments does not mean the feature will work as expected in the production environment. The point of a feature flag is to facilitate the feature used in a live environment where you will want to turn it off. Technically, it is the "kill switch" and based on the limited information, turning that switch off would have saved them. Except it doesn't sound like anyone had training or didn't have access to the feature flag. Different teams are going to have different procedures for how feature toggles are switched. Better of it is a page where product can manage but might be entirely engineer owned. "Not deploy test code" is a non sequitur as depending on how you define it, all code is test code. The correct terminology would be "dead code" as the code should never run but because there existed a condition where it could, once it is revived, fuckery happens. You never want dead code to revive. I have never heard of good things happening when dead code suddenly runs.

@jajordan2106 2 ай бұрын

It depeneds borrowing a stock at a high price selling the stock at a high price then buying when the stock price falls allows you to make some money although the potential losses are infinite

@cheesesniper473 2 ай бұрын

The problem here was that there was no distinction made between user and development software. This PowerPeg development software should have never been on a live program server at any point in time. It belongs on a dev server or stored on a HD somewhere.

@ally6438 5 ай бұрын

As a coder who recently broke out with shingles due to the stress of being given far to short a deadline for something that ships to 9 million people I feel for these devs, that’s insane

@DanielBoctor 5 ай бұрын

I know, I find that most of these production nightmares are due to time constraints being pushed on the devs

@JorgetePanete 3 ай бұрын

too*

@everythingpony 3 ай бұрын

Isn't shingles deadly?

@TokyoXtreme 3 ай бұрын

@@JorgetePaneteYou found the misspelling, yet missed the comma splice.

@ally6438 3 ай бұрын

@@everythingpony I don't think so, but you're not really supposed to get it until 50+. I'm early 30s

@scottpaul7427 3 ай бұрын

Interesting. The way I've heard it, Power Peg was not "intentionally lose money for testing", but a production option to try and "peg" stocks to a given price (even if that would lose money). After being deprecated, it was judged too difficult to remove without effecting other production code. It was still being tested in builds until the 2005 changes caused those tests to fail, and they were removed. There was a script to automate deployment to the servers, but one was down for maintenance so the connection timed out and it was skipped without logging an error (or nobody bothered to read the log.) During the event it was obvious that something was wrong, but not obvious that it was causing huge losses until after the rollback accelerated things.

@Smaylik03 3 ай бұрын

And that's why investing into code quality and tests is important.

@Entropy67 3 ай бұрын

I just wrote about 120 tests over the past week or two for what I'm currently working on. Good to know that someone will just delete them when they fail in the future instead of actually figuring out why they are failing... eh, if the tests failed when the code is broke my job is done.

@restoreleader 3 ай бұрын

As a QA, i would like to comment on this: hehe

@fredmercury1314 3 ай бұрын

@@Smaylik03 Google, Facebook, Twitter, Microsoft, Apple, and Adobe have entered the chat... "Testing is what users are for."

@SuppressedOfficial 3 ай бұрын

@@Entropy67 Meh. A test that never fails tells you nothing. A test that always fails tells you nothing. The only good tests are basically coin tosses, but then why write code that only works sometimes? =D I fuckin' hate tests. lol

@tomr6955 3 ай бұрын

So an average person makes a mistake and gets fired, but these guys lose half a billion and get bailed out.

@Jackson_Zheng 3 ай бұрын

they lost a company that took them 17 years to build from the ground up.

@SianaGearz 3 ай бұрын

@@Jackson_Zheng and nothing of value was lost?

@coren5911 2 ай бұрын

@@Jackson_Zhengwho cares, you could say the same for a small business owner

@natmarelnam4871 2 ай бұрын

They didn't get bailed out... you people are so f***ing stupid. They didn't lose the company either. They sold stock at a loss, then in a separate transaction, they SOLD the company. They had $350M in LIQUID ASSETS.... That doesn't get tanked by a $450M loss.

@AEVMU 2 ай бұрын

@@Jackson_Zheng They did already make many millions themselves though.

@Lyokou 3 ай бұрын

As a software engineer as soon as I heard one month, I was like yep. Been there, done that.

@Nope_handlesaretrash 3 ай бұрын

Oh no wont someone PLEASE think of the poor high frequency traders. Lol.

@natmarelnam4871 2 ай бұрын

typical pleb, hates the hand that feeds.

@Xilladan093 2 ай бұрын

@@natmarelnam4871 who , you?

@counterleo 2 ай бұрын

Any donation link?

@Bramble20322 Ай бұрын

@@natmarelnam4871 Stock markets are literally leeches on society and bring literally no value. Prove me wrong.

@dale3478 3 ай бұрын

Other rollback: "finally sh*t has stopped hitting the fan" SMARS rollback: "holy sh*t! There's now 8 times more sh*t hitting the fan!" But seriously, for a software that runs at a scale of thousands of requests per second and work with millions of dollars, there should definitely be some sort of kill switch or feature toggle built in from the start. Although the "rollback cause even more problem" is definitely a first for me

@MarcosAlexandre-no3qx 3 ай бұрын

probably someone saw every problem and said that they need more time to work on solutions and the company probably said. Just send anyway, it wont happen and we put in our next update, that never came.

@manankpatel2759 3 ай бұрын

As a developer i want to just say to all the companies and their executives, dont push too much for a little gain, give enough time as per the requirements or it might happen that the company may not even exist after launching that product.

@MKVideoful 3 ай бұрын

Doesn't matter, they will never listen.

@GamesFromSpace 3 ай бұрын

Also, don't try to pay the devs as little as possible. Because then this shit happens, and a good salary is a rounding error in comparison.

@enginerdy 3 ай бұрын

@@GamesFromSpaceI think they are probably well-paid, but there should have been 5x as many…

@MonsieurSansHonte 3 ай бұрын

Deploying code from dev to prod without a QA staging environment or subsequent smoke testing, is a recipe for disaster!

@BigJMC 3 ай бұрын

Something similar nearly happened to my father. He works at an investment bank and was called in to solve an issue. Their network was being clogged by packets and it seemed a cascading effect has been caused between every server as they were stuck in a loop. 10 minutes before the stock market opened he went into the server room and just started pulling out ethernet cables and disconnecting servers and 5 minutes before stock market opened he pulled out the right ethernet cable and disconnected the server causing the issues. They could’ve lost billions of dollars that day.

@Xalgucennia 2 ай бұрын

Lol I was wondering why thsrs guys weren't doing the same, literally fixed in 5 minutes

@camiscooked 2 ай бұрын

@@Xalgucenniacloud computing exists... They may have been unlucky enough to have gone cloud computing only

@ggsap 2 ай бұрын

@@camiscookedDoes cloud computing not have an off button?

@camiscooked 2 ай бұрын

@@ggsap well yeah. No off button, someone else is running the server and you have to access remotely. Most companies will only have a few people who truly know the steps to stop function.

@ggsap 2 ай бұрын

@@camiscooked Its not that hard to stop the server. Its literally a giant red button in most cases, especially more easier an than diagnosing the issue

@ryansamarakoon8268 5 ай бұрын

Really appreciate the amount of detail here! Most other coverage was surface level but you went into a lot of great detail here

@DanielBoctor 5 ай бұрын

I'm glad you liked it! Thank you for the support I appreciate it

@ychentt 3 ай бұрын

Laughed so hard. Just subbed. Can't believe for this quality you only have 11k subs.

@DanielBoctor 3 ай бұрын

thank you for the support ❤

@wwbcwp 2 ай бұрын

This reminds me of a time I accidentally uploaded an older version of a report I'd been working on in college, overwriting the new one and setting me back days.

@ryanpalo 3 ай бұрын

This makes all of the times I screwed up prod feel so much better. Thanks for the indepth analysis on this.

@DanielBoctor 3 ай бұрын

thanks for watching!

@MangaGamify 2 ай бұрын

You can't screw like them.. **loses 100k per second **

@user-uy8yt7ku4w 3 ай бұрын

I still don't understand why they didn't just stopped all their servers and cancelled all the order's that weren't filled. That would probably take a couple minutes, instead of half an hour

@chunkyMunky329 3 ай бұрын

They could certainly have cut the power, which would have been fastest but I don't think it would have been possible to cancel the orders that have already gone through

@ME0WMERE 3 ай бұрын

@@chunkyMunky329 better to cut their losses than continue to lose $2.5 M per second

@chunkyMunky329 3 ай бұрын

@@ME0WMERE Thats what I'm saying. Except I was saying that they should cut the mains power to the building instead of manually switching off each server

@SianaGearz 3 ай бұрын

Do you know where the master breaker is for your whole building? Well apparently they didn't either.

@sandworm9528 3 ай бұрын

@@SianaGearz Yep I do, and if we were losing 2.5 million a second I'd probably hear someone yelling 'kill the power' and i would flip it

@ninjaasmoke 3 ай бұрын

God! Imagine rolling back to a stable software and losing even more money. That would have sent people nuts!!!! Losing close to 20m every second.

@darrennew8211 3 ай бұрын

They didn't roll back the flags, though. That was their mistake. The problem started when they flipped the flag, and then they left the flag and replaced the code.

@ninjaasmoke 3 ай бұрын

@@darrennew8211 thank you captain obvious

@voidsp 3 ай бұрын

2:23 that's a bit of a fallacy IMO. There is a kill switch almost always. If affected services are on-prem - kill their Internet connection. Pull a plug on whole office/building if you have to. And if it's a datacenter, do basically the same - DC support can disconnect your servers/racks from the Internet.

@EatMyAstro 3 ай бұрын

Market-making (Knight's entire business) means being the middleman for every trade possible... with their infrastructure and resources, they were the #1 and were making a killing. IIRC Knight was responsible for nearly half of the volume across all exchanges in the stock market. If they seized operations here, they would lose everything. Their job is to remove friction between buyers and sellers by being a middleman, and as their reputation grew (along with their systems), it was paramount to always be online. It is not as easy as unplugging a box, and in many cases doing this would only make matters worse, logistics and business-wise. At the end of the day, yes there should've been a killswitch, and yes it should've been engaged. This was one of the first (if not the first) blowups in electronic markets that the industry had ever seen. And from the sounds of it, Knight was understaffed in their engineer/ops departments.

@MrAntiKnowledge 3 ай бұрын

I remember an old video where they switched a telephone network over and had to take the old one offline first. It involved 30 or so people with bolt cutters :D

@samramdebest 3 ай бұрын

I remember that video too Went looking for it a while back Couldn't find it anymore

@DanielBoctor 3 ай бұрын

Is this what you are referring to? This is fascinating kzfaq.info/get/bejne/qceCnNVqms-th58.htmlsi=uBbpgRjyGvrHR1_S

@samramdebest 3 ай бұрын

Yes it is And now I'm confused as to why I couldn't find it, apparently it was already in my likes? I must have been pretty tired when I went looking for it last time (or maybe it was set to private for a while 🤷‍♀️)

@wilkamania 2 ай бұрын

Gotta love the corporate Ops mindset, with directors that probably have more political experience than technical experience. It's like asking someone to cook a thanksgiving turkey in 30 minutes, and then blaming the cook for everyone's food poisoning. While the damage wasn't nearly as bad as this, I worked at a company that wanted to replace our entire system with a brand new one in 6 months. This was a legacy system integrated into EVERYTHING , and being a publicly traded company everything had an even longer process. My Sr. Manager and director kept using the stupid analogy of "the consultants are building us a ferrari and handing us the keys". I ended up leaving the company during this period, but heard that the Sr. Manager and Director started point fingers at each other when the project wasn't even 50% complete by 6 months. The Sr. Manager left before he could get fired, and the Director got fired. Neither of them really knew how the system worked.

@ShayneReigns Ай бұрын

I said out loud just now that I WOULD like to watch your videos about cybersecurity. It was thrilling to know about this story and I’d love to hear more about the aftermath. Great video!

@zamoqi 3 ай бұрын

Enjoying your content big time. Appreciate the work that you put in!

@DanielBoctor 3 ай бұрын

thank you for the kind words - glad you like it ❤️

@648 4 ай бұрын

Wow, no way this only has 3k views. Keep it up!

@DanielBoctor 4 ай бұрын

I'm glad you liked it! Thanks for the support 😊

@williamchamberlain2263 2 ай бұрын

Software development vs software engineering : the latter starts with system requirements that you have to be able to verify before production, the former can start with "you have one month..."

@derekhettinger451 3 ай бұрын

Your content is excellent Was invested by the end of the video for sure Subbed, keep kickin ass man

@DanielBoctor 3 ай бұрын

thank you for the support! glad you enjoyed ❤️

@ronatola 3 ай бұрын

@@DanielBoctorYep. As a fellow IT Tech, I've seen a lot of these debacles during my career (even caused one myself once - shhhh) SUBBED

@yashshende2786 3 ай бұрын

That's why chaos engineering and DR testing is important... They will surely build a kill switch now 😂

@Orionbae 2 ай бұрын

Its pretty cool to actually see what quant firms do behind the scenes great video 🔥

@miguelmartins3864 5 ай бұрын

Excellent video - very informative! I enjoyed the blend of finance and software. Given how intertwined they are these days, there's likely many more topics to explore!

@DanielBoctor 5 ай бұрын

THANK YOU MIGUEL! I completely agree as well ❤

@spyr0guy 2 ай бұрын

"Software will handle it!" Software:

@LevelofClarity 5 ай бұрын

Great video. Would love to see some follow-up stories relating to HFT industry. Read Flash Boys years ago and absolutely loved it. I hope too see more of this from you in the future 😎

@DanielBoctor 5 ай бұрын

Thank you! It's definitely a area that I want to dive into. Thanks for sharing - I actually never heard of Flash Boys before ❤️

@LevelofClarity 5 ай бұрын

@@DanielBoctor Flash Boys is a great book. If you’ve never read anything from Michael Lewis that would be a great one to start with. The last couple of years I’ve mostly been listening to Audible. Hopefully you’re able to check it out.

@ronanoke 3 ай бұрын

Fantastic vid! A complex topic made simple, great job

@DanielBoctor 3 ай бұрын

Much appreciated!

@stockstreamtwitch 2 ай бұрын

Thank you for documenting this history. ❤

@ooplesoft 2 ай бұрын

Excellent video! What a great summary dude well done.

@DanielBoctor 2 ай бұрын

Much appreciated!

@AryanKumar-jo1pz 2 ай бұрын

The editing and effects are amazing. Reminds me of Lemmino really well done

@DanielBoctor 2 ай бұрын

wow, I never thought that my content itself would be compared to the legend himself. thank you for the support ❤️

@rightwingsafetysquad9872 2 ай бұрын

No kill switch? Like there was no circuit breaker to flip, no power cord or network cable to unplug? If I were the acting executive I would have walked into the computer room with cable cutters or an ax or something and just started chopping. I understand there could be large penalties for failing to complete market orders left pending, but it can’t be worse than $2.5 million per second.

@eadweard. 2 ай бұрын

What music would have been playing as you did it?

@rightwingsafetysquad9872 2 ай бұрын

@@eadweard. kzfaq.info/get/bejne/f551gqR2vs3XdWQ.html

@hillaryclinton1314 2 ай бұрын

I blame usa education system

@rightwingsafetysquad9872 2 ай бұрын

@@eadweard. Bat Out of Hell by Meatloaf.

@sierragutenberg 3 ай бұрын

Bro, in this case the problem which caused all of this was literally naming. Looks like I'm not the only guy who struggles with naming things 💀

@MartinD9999 3 ай бұрын

That was awesome. Thank you.

@DanielBoctor 3 ай бұрын

Thanks for watching!

@Ch17638 2 ай бұрын

Wait who hold on ..... Major deployment left to one person ? And then when trading started not a single engineer was monitoring trades just to check if everything was working as expected ? Then the CEO takes a break on launch date. We once deployed a new process service for company payroll, and on day one we had all leads and seniors monitor the system with several layers of safety introduced (limits to transaction amounts, limits to amount of transactions on first run) that data got checked to with an inch of its life, then the next set and the next with reports filled by the dev managers that had to be signed by the CIO before the remainder of the transactions could go through but even then as it ran at a staggered rate we had someone ready to pull the cord if anything seemed off. There were redundancies for redundancies as this system could empty 3 bank accounts in no time.

@mrgyani 2 ай бұрын

Yeah. Makes no sense. 😂 What a clown-show it was.

@ccctube5721 3 ай бұрын

Hi Dan, I really enjoy the format of video you make, I think you may even be the person who pioneered this genre. Please keep them coming.

@DanielBoctor 3 ай бұрын

Thank you so much. I can't say I pioneered the genre, but I appreciate the words

@-na-nomad6247 3 ай бұрын

Two lessons to get from this : 1- Software development is research, you CANNOT rush it, if you want to build faster, get more builders, don't pressure the ones you already have with tighter deadlines. 2- Being a good software engineer doesn't mean not making mistakes or knowing every function and library in existence, being a good software engineer means you clarify your code, document it, ask for reviews and testing and push back when management tries to give you unsustainable goals. it's 25% programming skill, 25% planning and 50% politics.

@Bobrystoteles 3 ай бұрын

Your first point breaks the golden saying of engineering. Paraphrasing "More mothers don't birth a child faster" Brooks's Law: "Adding manpower to a late software project makes it later." Throwing resources at something won't necessarily solve it and what's needed is time and patience.

@AdityaWaghmare 3 ай бұрын

⁠@@Bobrystoteles This 💯 percent

@aliciasueyee 2 ай бұрын

@@Bobrystotelesaccurate. The Mythical Man-Month is such a good read!

@guerra_dos_bichos 2 ай бұрын

2 women dont birth a baby in 4.5 months...

@FallenStarFeatures 2 ай бұрын

@@Bobrystoteles You're assuming it's a unified development process like back when everything was in-house. Each layer of devs is hired to build new systems on top of the old systems. So it's more like a mother giving birth to a mother giving birth to a...

@spacetime3 Ай бұрын

That hit hard... "Hit the Kill switch (for the love of god !!) ...... There is no Kill switch....."

@HarshAnalysis 5 ай бұрын

I feel like your channel is going to blow up soon . great video and editing . Can i know what editing software you use?

@DanielBoctor 5 ай бұрын

Thanks! For sure, I use DaVinci Resolve 😊

@kylebroussard5952 2 ай бұрын

*If you have $440M and you can lose even 10% of it in a single day, you've done a terrible job. This right here is abominable*

@vistalover9607 2 ай бұрын

Sorry if I don’t get the details but why couldn’t they literally send a so command out little pull the plug. It was not days of the cloud yet so wouldn’t they be running their own servers via some sort of enterprise setup?

@rzdnx 5 ай бұрын

Awesome video!!❤

@DanielBoctor 5 ай бұрын

THANK YOU ❤

@orlandostevenson7214 Ай бұрын

Very well done - digging into devops lapses?

@miss_adventure 2 ай бұрын

Production tip: the endings of these episodes feel so abrupt, it’s kinda jarring. I think it would be lovely to have more of an intentional outro - maybe summarize the topics discussed, or talk about some takeaways and how things might be improved in the future or something. Also a pause between the end of the script and the start of the “if you’ve made it this far” to give an indication that we’ve reached the end. Love how well you talk about these topics!

@weakend 2 ай бұрын

What a crazy story... so insane to me to run a company moving that much money and not have integration testing. On first glance you'd want to blame the engineers here, but the majority of the blame would have to be on engineering management/upper management to allow prod code on financial systems to be deployed sans integration testing. This story is a great anecdote as to why infrastructure as code/virtualization is so critical.

@jonathans.972 Ай бұрын

Working for an SBA lending company, we have to be SOC 2 compliant. I cannot fathom why Knight Capital wasn't audited on any of their procedures, especially considering they are connecting to the New York Stock Exchange!

@feuermurmel 3 ай бұрын

Why didn't they stop their software, stopping all trading, when they noticed something was wrong? I mean, this is not Star Trek where a sentient program can refuse to be terminated.

@redluck01 2 ай бұрын

There is no kill switch? No dual human control that checks the deployment? The CEO is out during a major brand new trading deployment? This is all lies from the CEO trying to save his job. Anything that goes right or wrong is the CEO's responsibility.

@mariuszmoraw3571 2 ай бұрын

If you don't have software kill switch, you always have hardware kill switch. Disconnect malfuctioning algorith from web and run tests until reason is found and fixed.

@tupelov346 3 ай бұрын

Good video! Id recommend you make an effort to better show when you are directly quoting something. I noticed a couple times when you read from an article but at first glance it appeared that it was your own writing.

@shubashuba9209 2 ай бұрын

South Park: "Annnd it's gone."

@Zyphera 3 ай бұрын

I really like this content. This seems to a good channel. Insta-subbed.

@DanielBoctor 3 ай бұрын

Thank you so much! I'm glad you like it. Thanks for subbing

@richardgilson3512 3 ай бұрын

Old server number 8 is the hero we need ;)

@MakeItWork256 5 ай бұрын

great quality!

@DanielBoctor 5 ай бұрын

Glad you liked it!

@fernandorojo6311 3 ай бұрын

These videos are great.

@taralalram 3 ай бұрын

As someone who's worked in IT for 40 years from machine code programmer to head of engineering, this is definitely the CEO's fault. Testing takes time and yes men are too afraid to speak truth to power, instead ignoring all the advice from engineering. No one died here, it didn't work out so well for the people on the space shuttle.

@aegisofhonor 2 ай бұрын

I remember this on the news back then, but didnt't know much about what was really going on behind the scenes, interesting that it was basically just a computer bug. Just a VERY VERY expensive computer bug.

@hawaiioutdoors 2 ай бұрын

You took something technical & dry, and made it entertaining.

@DanielBoctor 2 ай бұрын

This is an awesome comment, glad you thought so! Thanks for the support ❤️

@debasishraychawdhuri 3 ай бұрын

why did not they turn off the computers? just pull the cables.

@Shogoeu 2 ай бұрын

"There's no kill switch" - just pull the power cord...

@devzozo 2 ай бұрын

I get like 30 useless emails daily about some system error for something I don't support, interspersed with random PTO notifications from coworkers and company/organization wide announcements that aren't relevant to me. I can totally understand just ignoring those emails.

@msmith2961 2 ай бұрын

Doing rollback and making the problem 8 times worse made me lol so hard.

@danser_theplayer01 3 ай бұрын

Imagine getting Power Pegged for -440 000 000 dollars💀

@MoosesValley 3 ай бұрын

A SMARS a day helps Knight Capital work, rest and play 🎵 🎶

@ForkCandle123 2 ай бұрын

It's the fault of the regulator allowing firms to buy shares that aren't actually available at that time. And it's the fault of the firm doing such a thing. Incompetence of the firm involved not to have taken proper precautions. Such profiteering shouldn't be allowed. As it was, they lost out. They'd not have complained if they'd accidentally made that sum rather than lost it.

@FallenStarFeatures 2 ай бұрын

So you're saying the regulators shouldn't have allowed them to build a house of cards out of a house of cards?

@jeshirekitenkatt1212 3 ай бұрын

Wait - so PowerPeg is explicitly something that would destroy things if active on their live servers, it's in the first line of the email, and still no one bothered to open it? ... Remind me what your average email client looked like back in 2012 again?

@DudeWatIsThis 2 ай бұрын

The day Knight Capital got power-pegged.

@josephduenas4718 2 ай бұрын

No kill switch or procedure to resolve this? Sounds like the devs got 2 days to brainstorm and the company said "yup! autobots roll out" 😂😂😂😂😂😂

@laowai2000 2 ай бұрын

4:29 Can see a riverfront apartment used to own. That's Australia's Gold Coast. Surprised Pikachu face!

@DanielBoctor 2 ай бұрын

no way LOL. that's insane

@geoffclapp5280 2 ай бұрын

Good work

@DanielBoctor 2 ай бұрын

thank you!!

@lightning_11 3 ай бұрын

Please please please give me some kind of mediator design pattern on this entire system!! So much pain.

@oldmanwilikers1252 2 ай бұрын

Great video, I can’t understand the quotes though the audio is too cooked through my speakers.

@beepbop6697 2 ай бұрын

2:21 "there is no kill switch" 🤣

@gFamWeb 3 ай бұрын

2:00 "two and a half seconds" did you mean minutes?

@DanielBoctor 2 ай бұрын

oops, good catch. should have been minutes. good catch! updated the pinned comment.

@Quell__ 3 ай бұрын

power peg is my usual strategy

@mikethespike7579 2 ай бұрын

One of my former professors once told us that computers are just hard working idiots. They will readily wipe you and your assets off the face of the earth if just one line of code tells them to.

@Osirus1156 Ай бұрын

I use this example all the time when telling higher ups why you shouldn't try to force the devs to move faster.

@daverei1211 3 ай бұрын

At some point they should have just put in a firewall rule to block connections with the trading server so no more trades could be made while they’d figure it out. Better to do this at $40m than $400m…..

@danielklimek6320 3 ай бұрын

How they couldn’t hit ctrl+c or kill the processes in the background remains a mystery

@Hebdomad7 2 ай бұрын

The problem was management all along. But they got golden parachutes as punishment ... The sooner these wallstreet types becomes personally liable for these kinds of screw ups the better.

@Cfomodz 2 ай бұрын

@2:00 - did you mean a 2 and a half minute period or am I misunderstanding the numbers here.

@DanielBoctor 2 ай бұрын

nope, you're right, it was meant to be 2 and a half minute period. a few other people mentioned this, and I updated the pinned comment. thanks for pointing it out 😀

@WendiArif 2 ай бұрын

Can i have the buy high sell low program, i wanna invert it.

@ResonantFractal 4 ай бұрын

Absolute nightmare

@renakunisaki 3 ай бұрын

Reusing a flag like that is a terrible idea.

@somebodythatiusedtoknoooooooow 7 күн бұрын

That's the issue with arbitrage bots, when they fail they lose years of profits in just minutes. Title should be "Manual Deployment ends up costing $440 Million. Maybe we need to hire some devops? "