I’ve been using SQL wrong this whole time

  Рет қаралды 23,409

CompSciGuy

CompSciGuy

Күн бұрын

I've just never not used it to its full potential.
Most people have used SQL at some point - but also just stick to the basics. When we need to do something complex, we instead do it in another programming language and send multiple queries. Even though it's very simple to solve in SQL. In this video I show 3 basic building blocks that will really supercharge your SQL usage - and make you an expert.
Not only that, with this you will be able to run extremally powerful and scalable data processing without building all the infrastructure yourself.

Пікірлер: 75
@youtubeenjoyer1743
@youtubeenjoyer1743 6 ай бұрын
>99% of software "engineers" cannot properly use SQL nor design a proper relational data model.
@ordinarygg
@ordinarygg 6 ай бұрын
agree, they will reinvent 10000+ libraries and abstractions just because they lazy to read full doc and examples
@minor12828
@minor12828 6 ай бұрын
So i am top 1 you mean ??
@axjkhl7699
@axjkhl7699 6 ай бұрын
ah yes, software engineering = SQL like it's 1995 🤦‍♂️
@tipeon
@tipeon 6 ай бұрын
I always said there are only two languages that you absolutely must learn as a software engineer: English and SQL
@magicmulder
@magicmulder 6 ай бұрын
@@ordinaryggNothing like a “software engineer” designing a 5NF database model and then complaining they can’t really see what’s going on by looking at the data, and then being forced to make a 1NF mirror because performance sucks. 😂
@DanielAbdelsamed
@DanielAbdelsamed 6 ай бұрын
I think the answer to the currency conversion problem is a temporary table, something that is hyper-criminally underused by novice developers. Create the temp table in the DB session, fill it with the current conversions, then join on that table in the aggregation query. If multiple aggregations occur, the same table could be re-used as well. The total number of currencies is around 200 meaning the insertion is trivial, and the table could be indexed with something lightning quick like a hash index to assure quick lookup.
@acasualviewer5861
@acasualviewer5861 6 ай бұрын
Or just have a permanent currency conversion table. That is updated every 5 mins or whatever so that it reflects the latest rates.
@DanielHauser
@DanielHauser 5 ай бұрын
@@acasualviewer5861 This is the way.
@gosnooky
@gosnooky 4 ай бұрын
I do something similar with currency conversion. My application periodically queries conversion rates on all the currencies we support and caches the data, then we just calculate the exchange and inject the values into the SQL query. No need for adding functions to PG/My or anything fancy. A caveat for this method is wildly fluctuating currencies like Zimbabwe or Venezuela that may become outdated very quickly, but our use case only requires major currencies like USD, CAD, AUD, GBP, EUR, CNY, JPY, THB, and other European ones, so mileage may vary. Sometimes the simple option is the best, like now NASA spent millions developing a pen that can write in zero-G, whilst the Soviets used pencils.
@staticvoidlol
@staticvoidlol 6 ай бұрын
Some unsolicited advice: be extremely wary of putting business and calculation logic in SQL - databases generally don't scale horizontally and by the time you have that problem you've invested too much into that approach to undo. But nice video, SQL is very powerful and it's well worth learning how to use it efficiently.
@BosonCollider
@BosonCollider 6 ай бұрын
@staticvoidlol This is what replicas and data warehouses are for.
@acasualviewer5861
@acasualviewer5861 6 ай бұрын
you don't need to "put it in the database" your SQL can reside in your application. (But of course your application needs to be consistent in how it calculates things)
@AntonioDoesMetal
@AntonioDoesMetal 6 ай бұрын
Great video! I remember the first time I used window functions to solve a use case it blew my mind. However there is a danger in using SQL for business logic in your application which is that if you decide to migrate to a different database, all of your business logic is not as portable as it would be if it were just in Java for example (especially when most of your business logic is in the form of stored procedures). Yes most databases use "ANSI Standard" SQL syntax, however there's almost always some nuanced implementation details with how your queries will perform based on things like physical table modeling just as an example. And the moment you stray away from the ANSI Standard spec you now open up a can of worms, for example "CREATE MATERIALIZED VIEW" in Postgres doesn't exist in MySQL. And depending on how often you use that type of syntax, you might have quite a bit of reworking your code to do. Certainly not advocating that people don't use SQL, just mentioning to use sensible caution because I've seen it take 1+ years of unfucking everything when a company decides to move off of one database for another (specifically in the data warehousing space).
@MechMK1
@MechMK1 6 ай бұрын
Fully agree. I've seen companies suffer from this very issue, that they wrote their entire business logic for some really old database software, then wanted to migrate because some other SQL software better suited their needs...and then ended up realizing this would mean basically rewriting the entire business logic from scratch. Or the fact that most software engineers aren't that familiar solving entire problems in SQL, so even finding people to maintain the code and develop new code is hard.
@datguy4104
@datguy4104 6 ай бұрын
"portable ". This term needs to die.
@HuyHoiHay
@HuyHoiHay 6 ай бұрын
There's a tool called "dbt" which aims to solving that. Used alot by Data Engineer and Analytics Engineer
@tipeon
@tipeon 6 ай бұрын
One time, one such migration failed because my "senior" colleague didn't understand the concept of collation, and accused PostgreSQL of being buggy. You see, even when there's zero business logic in the database, a migration can fail 😂
@magicmulder
@magicmulder 6 ай бұрын
In practice you have way more issues having your business logic in a programming language that deprecates functions with every major or even minor release, forcing you to either sit out an upgrade or rewrite parts of your code. Changing databases is a rare thing. And if you do, you hopefully factor compatibility with legacy code in to your decision which database to migrate to. Unless your decision maker is a fanboi who will force migration to a certain database “no matter what”, in which case you’re screwed anyway.
@lozyodella4178
@lozyodella4178 6 ай бұрын
Amazing video keep doing it this way
@chrstfer2452
@chrstfer2452 6 ай бұрын
This is why im glad for my couple years experience as a SQL dev. Im no expert or super guru or nothin, but just those couple years of business analytics have helped me think about data way better now that im transitioning to data science
@Reichstaubenminister
@Reichstaubenminister 6 ай бұрын
Great editing style and very concise and useful information. I subscribed for your Protobuf video, nice to see you back!
@Leonhart_93
@Leonhart_93 6 ай бұрын
What is the classical piece in the background? Sounds like something from baroque.
@zoop2174
@zoop2174 6 ай бұрын
I guess I'm lucky I had to write SQL queries for reports before ever doing CRUD.
@chrstfer2452
@chrstfer2452 6 ай бұрын
All you have is python 2? Thats *absurd*, its been deprecated for new development for like a decade...
@deidyomega
@deidyomega 6 ай бұрын
for that trades example, just create an extra table with the currency, then write a c function for the convert. Only call the api to update the currency table daily or whatever.
@warny1978
@warny1978 6 ай бұрын
Write create a c function instead of calling an UPDATE statement within a stored procedure ?
@redguard128
@redguard128 6 ай бұрын
We always knew about subqueries and functions/procedures in SQL, just that the time those take is way larger than just getting the data and processing it in our programming language of choice.
@Homiloko2
@Homiloko2 6 ай бұрын
I get that it's an example but for most use cases it'd be better to keep a table with the daily currencies and you call that function to insert new rows when the current day isn't found, instead of calling an external API whenever the query runs. But hey, I had no idea you could integrate Python into SQL which was really neat to see.
@pauldriessens715
@pauldriessens715 6 ай бұрын
Agree with the extra table. Btw did you use SQL in Python though?
@wojciechkrzyzanowski6546
@wojciechkrzyzanowski6546 6 ай бұрын
I remember writing subquery problems for a subject final, doing this on a piece of paper is not very fun imo
@RaigyoEcU
@RaigyoEcU 6 ай бұрын
I have a problem, i always use WITH instruction and i was doing a similar problem, but at the last part, ineed to filter the temporary table with the Max value, and it's way slow, like 0.5s, the temporary table has like 5rows, so the join at worst gives me the same 5rows; my tables already have indexes but a temporary table cannot have, doing the query in any other way runs faster than using the WITH instruction, any idea why?
@NostraDavid2
@NostraDavid2 6 ай бұрын
Slap EXPLAIN or EXPLAIN ANALYZE (which actually runs the query, so be careful with that) at the front of each query and compare the output. Not sure if there are programs that can compare the output, but you could use vscode for that, I guess. Yes, learning that output is an art/science in and of itself.
@XCanG
@XCanG 6 ай бұрын
If that would be me, in case of currency on the backend I would rather pull conversion rates first (with possibility of caching it if this endpoint call frequently), then during request I would use CTE like: WITH "conversion_rates" ("currency_from", "currency_to", "rate") AS ( VALUES ('USD', 'EUR', 123), (...) ) and then use it in expression in other CTE/subquery
@user-tb4ig7qh9b
@user-tb4ig7qh9b 6 ай бұрын
For first cte you could use subquery
@filipniklas
@filipniklas 6 ай бұрын
Great video but I'd appreciate less generic filler visuals. I'd actually prefer still sql screenshots or graphs made in paint than those "slick" moving images.
@stewartdahamman
@stewartdahamman 6 ай бұрын
I can understand your point of view, but what do you get out of life?
@azursmile
@azursmile 6 ай бұрын
Think you can simplify the average trade query using "having" keyword. I personally try to avoid named subqueries in all but the most complex queries.
@warny1978
@warny1978 6 ай бұрын
when you use HAVING, it often postprocesses the results of the main query (not always). Using a CTE, you are sure it preprocesses the fastest query first or parallelise the 2 processes when using the same table.
@azursmile
@azursmile 6 ай бұрын
​@@warny1978 often, it is preferable to leave it to the optimiser to determine the processing order. This avoids falling into writing in an imperative style and makes the query easier to follow. Appreciate, this is a balance, and CTEs can also avoid excessive subquery nesting when they are strictly necessary.
@mightyhelper8336
@mightyhelper8336 6 ай бұрын
Wait - and we aren't talking about recursive queries?
@sarafarron7844
@sarafarron7844 6 ай бұрын
python2 in 2023... and why is it py3 will be more complicated option?
@youtubeenjoyer1743
@youtubeenjoyer1743 6 ай бұрын
Imagine thinking even for a second about running python on a production database server.
@ba8e
@ba8e 6 ай бұрын
Just get the exchange rates every hour/min/sec and insert them in a table, which is joined in the main query. Making API calls from SQL is stupid.
@lolwingding
@lolwingding 6 ай бұрын
Instead of SQL people ask DSA, so people learn DSA and apply DSA.
@CrazyFanaticMan
@CrazyFanaticMan 6 ай бұрын
Whats DSA?
@lolwingding
@lolwingding 6 ай бұрын
data structures and algorithms, like problems on leetcode, basically all in-memory ops, no persistent structures and algorithms
@Septumsempra8818
@Septumsempra8818 6 ай бұрын
Any Django people here?
@pzramos
@pzramos 6 ай бұрын
The fact that subqueries and CTEs are not common sense sounds frightening, to say the least.
@eliaperli2485
@eliaperli2485 6 ай бұрын
I see, you use squeel
@magicmulder
@magicmulder 6 ай бұрын
Squall.
@user-qr4jf4tv2x
@user-qr4jf4tv2x 6 ай бұрын
biggest sin in sql 1)not using sql, 2) ORMS
@deidyomega
@deidyomega 6 ай бұрын
ORMs are great for trival coding, just make sure you know how and when to use the escape hatch
@KhoaNguyen96
@KhoaNguyen96 6 ай бұрын
Lol I thought you were about to say SQL as "squeal"
@edhahaz
@edhahaz 6 ай бұрын
The more I look at SQL the more it makes sense for it to be integrated in the actual programming language but nah.... passing strings to a black box is the way!
@tipeon
@tipeon 6 ай бұрын
People tried it ... with mixed results. (Google sql pro*C) Nowadays, the general consensus is that it's not worth the hassle.
@magicmulder
@magicmulder 6 ай бұрын
Which is why stored procedures exist which are effectively function calls from the application. Also from a security standpoint it makes no sense to allow your code to ever do a “SELECT * FROM users”, with it exposing password hashes and whatnot.
@tom_marsden
@tom_marsden 6 ай бұрын
That’s why LINQ exists
@edhahaz
@edhahaz 6 ай бұрын
​@@tom_marsden LINQ is an abstraction over SQL... so an abstraction that generates strings... when I put it like that, it sounds terrible. I think it's worse in practice, but I can't lie, using the same queries on data structures as you would on a database and possibly mixing the SQL results with a data structure seamlessly is very nice.
@reed6514
@reed6514 6 ай бұрын
Querying an array with sql would be super cool.
@ajcgpol
@ajcgpol 28 күн бұрын
making rest api requests from DATABASE ? hard pass...
@MrCompSciGuy
@MrCompSciGuy 25 күн бұрын
Heh, yeah I wouldn't recommend it, just couldn't come up with a use-case that fit neatly with the rest of the topic I was talking about
@lian1238
@lian1238 6 ай бұрын
The stock videos are distracting. Does your audience really have the attention span of a 4 year old, so you have to keep their attention with useless stock videos?
@CrazyFanaticMan
@CrazyFanaticMan 6 ай бұрын
Artistic expression
I've been using Redis wrong this whole time...
20:53
Dreams of Code
Рет қаралды 332 М.
The Absolute Best Intro to Monads For Software Engineers
15:12
Studying With Alex
Рет қаралды 600 М.
Super gymnastics 😍🫣
00:15
Lexa_Merin
Рет қаралды 107 МЛН
Can teeth really be exchanged for gifts#joker #shorts
00:45
Untitled Joker
Рет қаралды 17 МЛН
The Only External System You Need
6:24
CompSciGuy
Рет қаралды 51 М.
Top 7 Ways to 10x Your API Performance
6:05
ByteByteGo
Рет қаралды 311 М.
How principled coders outperform the competition
11:11
Coderized
Рет қаралды 1,6 МЛН
Raw SQL, SQL Query Builder, or ORM?
16:19
ArjanCodes
Рет қаралды 95 М.
Solving one of PostgreSQL's biggest weaknesses.
17:12
Dreams of Code
Рет қаралды 175 М.
how Google writes gorgeous C++
7:40
Low Level Learning
Рет қаралды 796 М.
Understanding B-Trees: The Data Structure Behind Modern Databases
12:39
Never write another loop again (maybe)
10:48
Dreams of Code
Рет қаралды 249 М.
Complex Schema Design with Drizzle ORM | Common Patterns
45:55
I tried 8 different Postgres ORMs
9:46
Beyond Fireship
Рет қаралды 388 М.
Неразрушаемый смартфон
1:00
Status
Рет қаралды 1,8 МЛН