This is the first vid that truly explained this N + 1 problem so easy a newbie would understand, and showed clearly the examples with and without. Nice job!

True story, i was asked this question in an interview, the interviewer just asked how would you fetch a foreign key in a one to many database. I didnt understand the question then, gave a non optimized answer , 6 months later i realize why i was rejected. 😢 Btw great video, thanks man. You are the first person i subscribe to on youtube. 😁

Been looking for documentation about N+1 problem and couldn't find anything practical about it. This video just saved my life. Thanks.

You continue to have some of the best content on KZfaq. Well done Ben.

What I like about DataLoader is two fold; one is that it can be used with any data source you may have, as explained in the video. The other is separating the concerns of fetching that data into a loader, rather than the resolver. Separating the two makes it easy to maintain and substitute the underlying fetching logic and it can even have multiple loaders for one resolver, based on some condition. While DataLoader may provide caching capabilities, it is not actually the main purpose of it; it is a data loading mechanism and it doesn't matter where the data comes from, it just ensure that a signal value by its key always gives you the same value (like what _batching_ does).

Brilliant presentation on using dataloaders and joins! 👍🏾

Ben gives very clear explanation of N+1 problem. Thank you, Ben!

Thanks Ben, This was awesome. especially learning about the 4th GraphQL argument.

That's the best and simple explanation of N+1 problem I have ever encountered.

The importance of this video cannot be overstated. Well done.

Anyone that knows SQL intimately will watch this video and hopefully say: "WTF! This is progress? So much code and META to do such a simple task".

Love the pragmatism. As always, measure before doing any optimisation.

Wow, couldn't have demonstrated the topic in a simpler way. Thanks so much, great pace and great use of examples.

This is the reason I'm always 100% advising people do not use GraphQL unless they know what they're doing. This is the most-basic thing about GraphQL and the thing I've had trouble with myself in the past. It's difficult to solve unless you understand how GraphQL is querying and managing resolvers. I didn't know it had a name, but I'm glad someone else is pointing this out. It's gotta be the one thing that I've consistently noticed in GraphQL services.

12:35 No need to reinvent the wheel. There are a couple of projects who do that. Notably graphql-parse-resolve-info

Had to implement DataLoader and needed exactly this! Thanks

Excelent content!!! Simple in describing the problem and then showing different solutions

Thank you a lot, Really easy for newbie to grasp the concept

Everything you need to know about n+1 problem and its solution options. Incredible to have achieved all of this in 16 min.

Very neatly explained. Thank you!

Very well presented sir.

Good description of the problem.

Thankyou so much. It's super helpful. I think example with dataloader ( can be seen at 8:00 minute ) to solve this n+1 is better, because in this way we have the power of resolvers i.e. reusability and dry code. Other than this, where we are doing every thing in single resolver, no possibility of reuse, wet wet code everywhere.

Superb tut Ben!

well explained and a good overview of the solutions.

In NoSQL DBs is common to use denormalized schemas, so in your example, we could just store the author data within the book records themselves and we wouldn't have this problem. Obviously we would then need to update the books on author changes, but this is fine if we assume that the number of reads is way larger than the number of writes. A safe assumption for most use-cases.

Will be nice to see how do you take advantages of graphql-parse-resolve-info. Great job! Thanks

Great video as always ben

Great explanation! Thank you.

One solution I use is to have the AuthorLoader (because your gql api can also have a authorById or any others "entrypoints") then in the hydrate function you can fill the AuthorLoader cache with its prime() function. It's a mix between your join and dataloader solution, to get only one database query and keep the dataloader advantages in complexes gql queries.

this is very useful for me thank you so much

If your data is stored in sql, you can also make use of views and transactions (or procedures) which is way easier to scale off and prevent redundant code.

Great video. Thanks

I'm from the REST world. I respect GraphQL and I'm pretty new in it because I didnt' have a chance to use it in my projecects. I saw a lot from Ben's channel and Ben is doubtless great teacher and explanator. But I also saw a lot of things from GraphQL those are 'reinvent the wheel'. As much as I understood, the point of GQL is to optimize req/res between client and server, to make more comfortable env for developers... Internet is more and more faster so there is a question when the speed becomes criticatl? And I saw a lot of complexity in the code for GQL implementations. So I didn't see quite enough reasons to jump to GQL in some of my projects, when it is much more simpler and readable with the REST, without all those packages and libraries like Apollo. But from new guys in web development, I suppose it's a good idea to start and continue with GraphQL. And finally I hope next tool for communication between server and client will be much simpler than this one. Why something is new and maybe quite good when it makes the things much confusinge and a lot of headache.

Hi Ben, you are right about join-monster.. too bad though... seemed promising.

Very well explained

Nice presentation of what I would call "resolving sub-queries". The next problem might be to resolve entities/items in use multiple. Eg think of a collection of books each having author(s) and reviews. And the reviews in themselves having authors as well. How would we make sure to "dataload" authors last?

Brilliant video Ben, thoroughly enjoyed it and really well explained. Am I right in saying that TypeOrm using cascade and populating relations in a user query with nested relations e.g. User.find({ relations: ['recipes']}) will do this automatically in a single SQL query compiled through LEFT JOIN?

Awesome! Thank you!

This is incredibly cool and still relevant 2 years later. I do have a question though: the hydrate function is doing the job of a field resolver, meaning e.g. if we want to add another sub-field on the author in the future, and that field is a relational field in a third table, the complexity of that hydrate function increases. Is it possible to keep the field resolvers but still have conditional joins? It seems like it might be possible because the context function takes in the request object... but I'm not sure how hard it would be to create that doesPathExist function when the request object is the parameter, or whether that fully solves the problem in the first place

Java jpa solves n+1 using entity graph that tells it how/when to fetch. Have to say I like that solution better than writing full sql or dataresolver.

It's great, Thanks! 👍

Thanks buddy ❤

I'm working on a graphQL clone - I'm gonna have to tackle the N+1 problem eventually.

I think you could pass some custom builder abstraction into the graphql ctx. All the resolvers use that, but only at the very root does it query the database from the builder. Haven't really used graphql so not sure if possible

Great video - what is the VS Code plugin that helps highlight brackets/parentheses pairs?

Hey Ben, excellent work, but for the conditional approach (best of both worlds) how do we deal with more nested types and fields without it getting too messy? And what about "hydrating" one-to-many relationships at arbitrary level of nesting (from flat table of query result)? E.g. Books > Authors > Reviews (note plural)

I love the idea of graphql but this was the first thing that put me off it... the number of independent network requests

Ben, field resolvers are cool, but is there a way to use Eager loading (JOINs) instead of field resolvers? I mean, use eager loading only when necessary using libraries like Typegraphql + Typeorm?

this is an excellent video as usual, but I'm wondering how to do a generic dataloader factory, especially when it comes to re-ordering the result of the query (the trick of AuthorMap in your video). The way you are doing it assumes that the type of the keys is a string or a number, as after, you are creating your AuthMap object which is a Record where, in your example typeof Key is the value of you Author.id. If your Author.id is a string or a number, no issue. But what about if the field of the author you want to "cache' is not a number of a string ? Below is a piece of code where I'm struggling to define the refMap record. I had to transform the type of the key in string to make it works (need also to enforce TTypeOfKey to extends string|number) export const buildLoader = < TEntity extends MyBaseEntity, TTypeOfKey >( entity: TypeObject, keyName: keyof TEntity ): DataLoader => { const loader = new DataLoader( async (valuesofkey: Readonly) => { const vals: TTypeOfKey[] = valuesofkey.map((val) => val); const unsortedValues = await getConnection() .getRepository(entity.name) .find({ [keyName]: In(vals) }); const refMap: Record = {}; { const theValue: TTypeOfKey= (val[keyName] as unknown) as TTypeOfKey; refMap[theValue] = val; }); return valuesofkey.map((val) => refMap[val]); } ); return loader; }; usage : const AuthorLoaderId = buildLoader(Author, "id"); const AuthorLoaderAge = buildLoader(Author, "age"); const AuthorLoaderWhatever = buildLoader(Author, whatever);

At 10:29 there is an example with JOIN. I think there is a hidden bug in addition to the inconvenience of the extra data being pulled. Before the proper mapping is done using hydrate(...), there is an object of shape { id, title, authorId, name }. Does this not mean that any column names which overlap between the joined tables will get overwritten? i.e. what if the book has a `name` column and so does the author - how does the hydrate(...) function distinguish the two?

AMazing bro!!!!

Looking at first sight to your authorLoader implementation, you could write a generic function with that implementation but receiving the name of the table to be called with knex and use it in every relation you need within the context, am i right?

hi, could you make a video about handling error in graphql mutations or queries, I'm having an issue about connection reset on graphql server due to a lot of timeout sockets are being hanged when throwing exception in case of error provided input. Any idea how to solve it? Thanks

Do Prisma resolvers already solve N+1 problem? I saw this video and went ahead to try out my GraphQL server with SQL debug from prisma and it looks that when I Query a 1:N relationship it does not trigger n+1 SQL queries. Very good video

Thanks

I think you had 'Books' & 'Authors' mixed up, but the rest was very good :p

Hi Ben, Do you know how to access "info" in a type-graphql resolver?

What's the point with the "plumbing to get back the authors in order"? Is it the case that simply returning "authors" does not ensure they are returned in order?

Great video Ben! Please consider a video on Apollo federation. Thanks!

Hi Ben,I am facing the same issue in spring Boot.Do you have any reference or course you have done on N+1 problem in GraphQL spring boot?

Thanks for this video on a very common problem. For many-to-one relationships, I'm doing joins like you are. But for one-to-many relationships, I still don't have a way to do it in only one SQL query. Do you have any insight on this?

the dataloader version can make Long sql queries that can fail if you hit the limit of the database, the join version do not get this problem

But with joins, you can't do circular references, like get a book > all authors -> books (e.g. you need books of the second author) and so on, as you don't have a field resolver anymore. And that's basically one of the important GraphQL ideas/features (without it, it's basically a kind of RPC).

Wouldnt eager loading or doing a left join on books and author fix the n+1? Or it really needs to be on a key resolver, kinda make sense, if you'd eager load Author on the Books Query function it'll load it whether you asked for author or not. Did I understand it correctly?

Great presentation but when you look at the number of ms you saved it doesn't really change/get better. Is this because the DB is so close and there is no ping?

One question, if author has something like company, and we want to do this query { books { author { company , Is it possible to use the last approach for getting the companies? It seems that for nested relations we need to go with Data Loader

Can you explain why dataLoader need to be created per relation. I created dataLoader per field and it works nicely. So for example all my entities have createdBy field, which is ID to user table/collection, I use dataLoader for all relations that need to get user by id and they all seem to be batched into single query.

You should not use await without try catch if you don't want errors being swallowed. You should return the queries directly without assigning the result to a local variable.

how to able to see info value, just cannot console.log to read the value

I know you mentioned Join Monster, but why use Apollo Server at all rather than an automated db frontend (postgraphile, edgedb) or other graphql server (hasura)?

Could you please tell me how to log the query like you?

cool tuts

Good stuffs

I see VIM keys, I upvote

what is knex in your code

Thanks for not shouting.

What if you are using a document database like MongoDB and you have nesting? Say for example, I have Bands with Albums but I am also nesting Songs in there. In this case, I could use includes on the Albums but I would have to use $in for fetching the Songs in a different query. See another way?

IMO, n+1 isn’t a real issue when we have to implement pagination. With pagination, the performance gain by eager loading doesn’t trade off the complexity. Eager loading is generally not scalable.

It is pretty cool, to take a look how the DL is implemented using some of nodejs(event loop) quirks -> github.com/graphql/dataloader

Hi Ben, Let we know what's you gonna learn in 2020?

SQL doesn't allow more than 999 params in the "in" clause.

Where you are printing these query request ?

why not create views table for relationship table and make query to view table. in that way we don't have to join every time or use any data loaders...

Only in Javascript would 5 months be considered dead.

Why do you have question marks instead of IDs ?

If you watching this in 2022, you can use prisma and graphQL using nexus with apollo server. Done! 🙂

Thank you, I learn a lot from your videos. Looking forward to some videos and tips regarding Apollo type generation. `apollo-codegen`

😻😻😻😻

You look like Shawn Mendes

looks like you haven't sleep well in a while

Sick

Is there a reason you used knex for this video? I usually see you using typeorm. I'm panicking, thinking you know something I don't...

I really don't understand why people always need to find clever ways of working around SQL. Back in the Ruby on Rails days they also had this N+1 Problem when they were using ORM. This was 2011, but Relational DB's have solved this N+1 Problem in like the 90s or even earlier. At this point i have to ask: why not access the db directly from the client with SQL. Graph Ql seems to me like "not invented here" methodology.

Does any of these techniques work in 3+ step queries? user { post { comments { } } }

What is wrong with the sort function? :'( It's right there

Be careful with dataloader. As with any caching mechanism, your implementation should NOT rely on some cache representation. That's exactly how to get stale reads and buggy logic

My young friend you just start scratching the surface of the challenges of serious application development. I was experimenting with GraphQL in early 2019, since I am open to learn everything new and innovative, and quickly realized that it is great for "hello-world-sized" demo applications, but when you are working on real business applications from the arena of warehousing, manufacturing, logistics, and others; this technology is a total dead-end and waste of time. SQL is with us for 40 or so years, nothing can replace it. Do you remember ODATA, for example? With your nifty resolvers you are just overcomplicating your system and make them dead slow, and enormously hard to maintain or debug, especially with typeless languages like JavaScript, Python. If you want a flexible query engine, simply receive SQL queries from your clients and return back JSON structures, dead easy and the most performant solution, the simplest to maintain and debug. No-SQL databases are anyway totally unusable for complex business database systems/applications.

Hi, i want to use dataloader with mongoose, i am using this www.instapainting.com/blog/use-dataloader-to-batch-and-cache-arbitrary-mongodb-queries examle, it works, but how can i use projection.is there any easy solution which i didn't know

A kitten dies every time an open 'where in (.....) is used' on a relational DB. If you select always the fields you want to return, the DB will not do the Join if is not needed. So, you don't need to do anything on the code. DB's are clever enough to not to do the join if you are not returning any value from the right side. Always use Joins, even for 1 result. Don't be stupid. Sincerely, your DBA.