Database Tuning at Zerodha - India's Largest Stock Broker

Рет қаралды 103,542

Күн бұрын

Speaker : Satyajit Sarangi @ Zerodha
Step into the world of Zerodha, India's largest stock broker, and uncover the secrets behind their database tuning and optimization techniques. In this exclusive video, we'll take you through:
Advanced Database Optimization Techniques: Learn how Zerodha keeps their databases running at peak performance.
Performance Tuning Insights: Discover real-world strategies and best practices from the experts at Zerodha.
Scalability Solutions: See how Zerodha handles massive data loads and ensures reliability for millions of users.
Whether you're a tech enthusiast, database professional, or just curious about the technology driving the stock market, this video is packed with valuable insights and practical tips to help you optimize your own databases.
Video Credit : ‪@pgconfin‬
🔔 Subscribe to our channel for more tech tips and tutorials:
👍 Like us on Facebook : / perfology
👍 Add us on Instagram: / perfologys
👍 Follow us on Linkedin: / perfology
#DatabaseTuning #Zerodha #StockBroker #DatabaseOptimization #TechTutorial #PerformanceTuning #SQL #DatabaseManagement #techinnovation
Chapters
00:00 Introduction
02:59 History Of Postgres usage
07:54 Managing Big Data
14:40 Understanding your query planner
20:38 Replica Master Slave
29:10 Postgres DB Rights
31:18 Postgres Learnings
39:16 Problems with Postgres

Пікірлер: 100

@mzs114 18 күн бұрын

Boring tech is usually better in most cases.

@pankajsharma-cb4ok 16 күн бұрын

By listening to it, I got to know that I also faced similar issues and worked on similar set of solutions except materialised views and redis. Happy to see myself on same path...

@haloboy777 18 күн бұрын

Sensible defaults go a long way, but they only work in certain scenarios. I commend their choice of sticking to the basics. To be honest, their main engineering efforts are focused on rewriting their codebase, and I believe that is their core competency. I wish more companies allowed frequent rewrites [where needed].

@gokukakarot6323 18 күн бұрын

The rewrites are not unwarranted, they hit scaling requirements and then improve. Their PDF sending blog is a great example. Some engineers would probably start at that, but that's not how things work in real life. Most startups face this problem as well. Starts with java or rails and then splits into microservicrs to scale independently

@manishmundra7303 18 күн бұрын

No doubt this is a gem talk.. Though I feel slides could be done much better with the kind of content talk had.. Still learned a lot & cheers to this great talk!!

@CoppsMusic 9 күн бұрын

Zerodha sits on top of SEBI's tech APIs. God, One has to listen from SEBI on how their orchestration is.

@arpanmukherjee4625 17 күн бұрын

I love this guy seriously. Very powerful insights.

@codingwithsam4992 16 күн бұрын

These are the guys and their content which people should be consuming and not those of LinkedIn didis

@HardikGajjarTechy 17 күн бұрын

PG always shock engineers with new features and flexibility

@nabhamgupta3626 13 күн бұрын

Super interesting :) more such insights tech talks🙌

@TheBrahmadath 14 күн бұрын

So finally found the responsible guy for the recent backend glitch..

@TesthelloUser2 18 күн бұрын

Thank you for the video....

@minimilitiapro6528 14 күн бұрын

Great talk , got to learn a lot from this video 😍

@anantmishra6783 18 күн бұрын

hahaha ..good one! Very valid points were made!! x)

@animalempire5513 18 күн бұрын

That's good to know😊

@velloer 17 күн бұрын

Awesome

@samannwaysil4412 18 күн бұрын

so informative

@mrpam4672 14 күн бұрын

Great!

@BhagyashreeJha-h1v 13 күн бұрын

Boring is good for you. Thanks for sharing the knowledge.

@ChristieDsouza 14 күн бұрын

this is gold !☺

@nikhilkartha9373 4 күн бұрын

Understanding this is as exciting as watching paint dry.

@NIKHILVERMA-xr7lc 14 күн бұрын

Good lecture ⭐

@Dipesh_Sol 12 күн бұрын

At 37:27, not just the speed of computation, but also the egress cost of transferring all that unnecessary data over the wire to your server from database to perform reduce type computations which can simply be done on the database layer itself. And if you find yourself querying such data frequently, then you can have materialised views for it, making it even faster. Finally, if your use case doesn't require the most latest data and is fine with eventual consistency, then you can add a caching layer in front of it, making it blazing fast.

@WaseemAshraf 16 күн бұрын

I don't agree with offloading all of the computations to the database. Yes some computations are good on the database side, but if you are doing some heavy computation that should be on the app side. The benefit of that is you can have multiple backend machines and single database. Using loadbalancer and multiple workers is very easy for backend as compared to having multiple databases.

@srikanth1281 16 күн бұрын

Agree, suffered a lot with my analytics platform (mongo) only because we offloaded queries to db. This is even after good amount of denormalisation, huge data means aggregation are consume all the resources.

@VIKASJAIN-no5wt 16 күн бұрын

Hope Kamath brothers are paying you well..

@sarbajitnandy9291 13 күн бұрын

Where does this kind of tech conferences happens?

@itshgirish 17 күн бұрын

am i missing something here? Able to bring up db 15-20 TB of fresh copy from S3 in < 2 mins!

@abhirajsingh8138 15 күн бұрын

yes he said that

@teluguinfofacts4600 15 күн бұрын

Is it true

@abhirajsingh8138 15 күн бұрын

@@teluguinfofacts4600 no idea

@an_other_world 14 күн бұрын

frequent code rewrites === frequent down times (if the code changes do not have sufficient coverage and are not properly tested by QA)

@YugalkishorJha-z2x 3 күн бұрын

Wow

@parasharp28 14 күн бұрын

It all felt like an interesting discussion, but I couldn't understand (won't say I couldn't digest) a few things he mentioned such as - copying terabytes of data and restarting the server under 1 min, letting the database do most of the calculative work, no replica sets (how about contingency planning) and in a way portraying Postgres as an 'all weather database'......seems like a wake up call for myself.....

@abhishekbhardwaj563 17 күн бұрын

I have always found simple dumb solutions to be the best solution that lasts long without any crash.

@ganeshbabu8263 18 күн бұрын

I feel dumb

@ans_human 13 күн бұрын

Can you share the RCAs on recent failures on the kite platform

@godisgreat1000 14 күн бұрын

why most of the crucial times zerodha app stock responding?

@Anonymous-im2dy 17 күн бұрын

1:42 why there is small laugh on mongodb 😂

@VarunKeshav 16 күн бұрын

Yeah WTF

@VarunKeshav 16 күн бұрын

That one slide of how do we manage big data pretty much throws light on points where MongoDB shines. Maybe they had a poor experience and wrote it off.

@shantanukulkarni8883 14 күн бұрын

I guess nosql ism't really suited for a very structured thing like a stock broker database. Others he mentioned like Mysql, Cockroachdb are SQL. It's kind of just like an experimental thing. But maybe he just remembered something funny about it.

@kparag01 11 күн бұрын

He wants to keep talking light but the audience is damn serious

@J0Y22 18 күн бұрын

damnn

@phoneix24886 6 күн бұрын

A little bit over provisioning when serving 2 million requests with an sla is better than being conservative.

@cringelord511 16 күн бұрын

is it okay to share your applications db architecture like this?

@BashDroid 15 күн бұрын

yes, because he mentioned that they had made this changes by looking on the specific data that are dealing with and anybody else doing the same thing with there data won't work the same

@cringelord511 15 күн бұрын

@@BashDroid was this organized by zerodha or postgres?

@BashDroid 14 күн бұрын

none, it's done by the postgres community in india.. pgconf

@user-hl6hv3qu9c 3 күн бұрын

Do they implement CQRS ?? not clear ; what kind of postgres scaling ?; Any SOR layer - System of record; what ORM ??; some high level banter about postgres DB tuning with no specifics; Query optimization no specicifs

@gangadharm 18 күн бұрын

Terabytes of data copied to a server and starting in less than a minute? Are you joking? Copying the data from S3 to your local server itself will take the time for terabytes of data. May be I am missing something here.

@Vikasptl07 18 күн бұрын

Now you know reason why zerodha is down every now and then. Haha.. but on lighter note i think for trading broker they should have robust backup mechanism in place, s3 to db in minute is a dream scenario but that works for them so who knows

@haloboy777 18 күн бұрын

I think they're not compressing the data. But that is way too wasteful.

@satyamverma 17 күн бұрын

they must be loading that 500 GB of cache data or max 1TB.

@v.rajasekar5044 17 күн бұрын

100% not possible even with newer instances with more than 100Gb of network speed. The ones with more than 200Gb of network speed requires special drivers and interfaces. Either he was bluffing or misremembering details. As they were partitioning by month, it could be possible that he is referring to restoring each DB hosting a month/s worth of data in under 2 minutes as that is possible. But data in that single server would be in the range of 1 or 2 TB and even then only with compression it is possible to do it under a minute.

@premsaivittal2030 15 күн бұрын

What's his name ?

@sekhar4033 14 күн бұрын

What is his name

@Grizzler231 4 күн бұрын

Bruh he laughed when he said mongo

@codingwithsam4992 16 күн бұрын

I want to connect with you? What is your name?

@keerthineeraj7937 13 күн бұрын

What is the name of the person speaking?

@kamleshgorasiya3854 16 күн бұрын

Can share RCA for Issue we are facing since last two months. I am Software engineer and also Customer of Zerodha. I want to know why other broker are runing fine and zerodha not.

@Desh-o7b 15 күн бұрын

no you are not, because if you were you would've asked this question from your team.

@deva8496 15 күн бұрын

Sorry you are not Robinhood of India in trading. Robinhood offers research and lot of reasearch and recommendations with lot of features. But you are one of best discount broker to save money of swing traders

@thiccboi6211 15 күн бұрын

It was a simple comparison for understanding the product and its domain. Its not an official statement of the company

@TeluguAbbi 9 күн бұрын

31:19 Postgres Learnings

@viputdBeast 6 күн бұрын

Atleast mention the speaker name in video title or in the description

@rishiniranjan1746 17 күн бұрын

People say c++ is retarded/obsolete. I dont agree. How you can achieve super-performace, responsiveness, better synchronous result without using a natively driven code.

@chandranshpandey1929 16 күн бұрын

too much tuning these guys are doing that their app having lot of glitches every now and then

@yashvarshney8651 18 күн бұрын

gold

@go1d 18 күн бұрын

ha bhai?

@RahulRaj-jc7os 18 күн бұрын

@@go1d

@battletremor 14 күн бұрын

What does he mean by checkpoints of data backup? As in they dont take entire backup from the start to current? Just a fixed period of time like a month?

@bangalicoder863 3 күн бұрын

It means a delta load with a specific start point instead of a full load

@srinivasreddyt9555 8 күн бұрын

at 26:08, you realize that it is a mallu asking questions.

@MrDasamitkumar 17 күн бұрын

15tb in 2 mins !!! Really?? How ??

@v.rajasekar5044 17 күн бұрын

100% not possible even with newer instances with more than 100Gb of network speed. either he was bluffing or misremembering details. As they were partitioning by month, it could be possible that he is referring to restoring each DB hosting a month/s worth of data in under 2 minutes as that is possible. But data in that single server would be in the range of 1 or 2 TB only.

@phoneix24886 6 күн бұрын

When did he say that?

@DK-ox7ze 16 күн бұрын

No replica? So what happens if the DB crashes and data gets corrupted? You loose customer data!

@ShaileshGuptaKoolwallost 16 күн бұрын

time to switch to another broker ? :P

@ShaileshGuptaKoolwallost 16 күн бұрын

DB backup, restoring everyday. postgres as caching layer - it seems like they just want to rely on one thing only which is absurd. Seems like they are trying to find solution to problem which shouldn't have been problem as such. Totally absurd, they are ok with everything running, donot want to do things at higher scale. How can you be ok with such kind of system.

@saralk18 5 күн бұрын

It works... until it doesn't

@krishankanhaya_ 12 күн бұрын

Caching is the answer to all of your questions in this talk 😂😂

@tanaygupta2009 9 күн бұрын

Caching might be a very bad idea in case of Trading apps.

@krishankanhaya_ 9 күн бұрын

@@tanaygupta2009 i may be agree but how

@randomvideos9541 17 күн бұрын

Ku6 bhi?? Pehle jo loss karwaye ho uske bare me video nikalo.

@prahladb5019 15 күн бұрын

now we understood why glitches are happening frequently in Zerodha..

@asif-ali 13 күн бұрын

Aaaaa.....aaaaaa.....aaaaaa......aaaaaaa.. Amm emmm aaaaaa aaaaaa... What that.... Silly

@harshpathak8029 16 күн бұрын

I think it is no longer the largest stock broker. Groww over took it since oct 2023

@SirShashanksonusahu 16 күн бұрын

Groww is stupid, just because of marketing it is gaining market share

@yuganderkrishansingh3733 16 күн бұрын

@@SirShashanksonusahu But that doesn't change the fact that they are the largest at present.

@abhijithganesh2064 14 күн бұрын

So much talk yet no stable apps😂

@GagandeepSingh-op8mb 13 күн бұрын

Zerodha doesn't have replicas and doesn't have slaves. All they have is a sorry when they server or db crashes. People lose money, and they think replicas are not important.

@ravishmahajan9314 16 күн бұрын

Only hardcore techies or IT professionals will find this video a Gem💎 . Well in the era of the Graph database, MongoDB, NOSQL etc. the legend in the video is talking basically about SQL database.😂 ❤ Yes he is definitely correct. With Oracle as an enterprise paid database, we have the king of open source database called Postgres, a boon for startups. I am really surprised, zerodha with millions of customers uses Postgres..... ❤ I thought they were using some high tech AWS virtual nosql databases 😂

@AyushKumar-ij2xd 14 күн бұрын

overloading postgres is not helpful and dont do it specially when you have no concept of nodes . postgres servers are not free even. if you are not able to write code for sum/avg for close to same speed as postgres service then you need better engineering. Architectures become complicated when you have distribution/SOA . Either this is half knowledge served or this platform is running on hopes and dreams. And also S3 is not that fast

@gyanpattnaik520 8 күн бұрын

Better if you can move to the cloud. AWS or Azure both provide great support.

@user-hh5vg9st1q 16 күн бұрын

This is feels like a 3rd year B-tech class, not a techie presentation. The presented points can be found across multiple b-tech computer science books. May be he is not a tech person thats why he didnt gave the examples on tuning.