How REST APIs support upload of huge data and long running processes

How REST APIs support upload of huge data and long running processes | Asynchronous REST API

Рет қаралды 124,905

2 жыл бұрын

We are very glad to start the REST APIs made easy course with the very first video on fundamentals of REST APIs, HTTP Methods, taxonomy around REST APIs etc. We hope that you get some value out of this course and these concepts help you thrive in your jobs, interviews and help you become better software engineers every day. Here are few links that you might want to refer before or during the course:
More on the topic:
✒developer.mozilla.org/en-US/d...
✒github.com/microsoft/api-guid...
------------------------------------------------------------------
Recommendations
------------------------------------------------------------------
Our full courses on youtube:
✒ System Design Primer Course: • System Design Primer C...
✒ REST APIs made easy: • REST APIs MADE EASY
Some paid courses that we recommend:
✒Educative.io: bit.ly/3qnW5ku
✒Interviewready.io: get.interviewready.io/ (Use coupon code SUDOCODE for extra discount)
------------------------------------------------------------------
About Us
------------------------------------------------------------------
Created and Instructed by:
Yogita Sharma
✒ LinkedIn - / yogita-sharma-83400b55
✒ Instagram - / sudo.code1
✒ Facebook - / sudo.code
✒ Medium - / yogita088
Post-production(editing, thumbnail etc) managed by:
CiKi
✒ Website: www.ciki.co.in
✒ LinkedIn: / 74735937
Colors and design by:
Naini Todi
✒ LinkedIn - / nainitodi
Both Arpit and Yogita are software engineers and want to help other software engineers become better by providing high quality and well researched content by adding their creativity and teaching twist.
------------------------------------------------------------------
Join Us
------------------------------------------------------------------
Hangout with sudoCode:
✒Discord Server: / discord
For business:
✒Email: sudocode.yogita@gmail.com

Пікірлер: 135

@warnercooler4488 2 жыл бұрын

I think what you have described here is the request/response pattern for long running jobs. This should NOT be confused with uploading a large file using HTTP, where you need to keep the connection alive until the content for that request is fully uploaded. If you send a 202 right away, the HTTP connection will be terminated and there is no way for the client to send the rest of the file. You can however, have multiple requests in parallel, where some are uploading different chunks of the file, while one request keeps polling the overall status. I think this is how youtube uploads work.

@Krishkarthik84 2 жыл бұрын

I was about to comment the same. Large file upload is not an example for Async API (in this video context). Better example would like order processing where it involves payment, shipment, etc.. better to handle this via Async.. Example: Amazon order, you get a message as order accepted immediately, later you get a mail confirmation on your order if all of the internal API's are processed successfully.

@aryankumar87771 Жыл бұрын

@@Krishkarthik84 Is that an appropriate example ? Wouldn't that be an example of a message queue as opposed to an Async API since amazon does not return a polling url to check the status of the internal API's after the user places the order.

@Krishkarthik84 Жыл бұрын

@@aryankumar87771 You are correct. Order processing is an async communication not an example for Async API.

@nandanp.c.7775 2 жыл бұрын

Thanks for the video. Awesome explanation! Especially how you use all technical vocabulary to explain examples- long running operation/process, put/post api, Resource, Endpoint, latency, Asynchronous, Synchronus , token, fetch the data, user experience,

@ayasswain 2 жыл бұрын

Very Nicely explained. Another good example of asynchronous flow is the life cycle of an order on an e-commerce site starting from placing of an order till it is delivered to the customer. Customer checks the status of his order on the e-commerce portal which in turn calls an Rest API in the backend to fetch that data.

@Kpr2000 2 жыл бұрын

Much needed one 😃 Thanks a lot.

@shyamsundervr5749 2 жыл бұрын

As always, a simplified, easy to understand explanation. A logical next step would be to cover webhooks as a form of async APIs

@nrk7772 2 жыл бұрын

I was looking for this ... thanks 😊

@sanathkumaru6358 Жыл бұрын

Thank you for such wonderful , precise content.

@MLLErourouwita 2 жыл бұрын

Thank you for this video, thatjust cleared up a lot of confusion in my head

@shubhamjaiswal2322 2 жыл бұрын

Thanks for this video, very nicely explained.

@Sushil2874 2 жыл бұрын

Very nicely explained. Thank you..!!

@mrsbootsworkouts 7 ай бұрын

Loved it, thank you!

@RaunakShrivastava 2 жыл бұрын

Hi Yogita, Thanks for making such an awesome content on System design. Can you please make on video on detailed explanation of communication between different components of a system with real-life example. Thanks again!

@shashankcool 2 жыл бұрын

thanks for such videos , nice & clear explanation.

@InderjeetSingh-jp3ln Жыл бұрын

Thanks for such awesome content!!

@MrPS98 3 ай бұрын

Nicely explained. Thanks for this video👍👍👌

@hiteshgupta968 2 жыл бұрын

Thats a very rich content a big thankyou from society..

@adityapoddar4275 2 жыл бұрын

Thank you Mam, very well explained!!

@streammxc 2 жыл бұрын

Thank you for this great video! One question here, if I want to create a progress bar to show the upload progress, do I need to call the status URL once a while, or keep the connection alive? What would be the best practice?

@tejasbhat7920 2 жыл бұрын

Hi @sudoCODE, during this async upload, where will you store the large file ? Let's say you've a queue that needs to process some meta data, but you've raw files as well which you can't put in queues right as queues like Kafka takes small messages. You can't store the video in memory as well. I'm not clear on this part.

@Sparshchokra 9 ай бұрын

its a HOT question in interview for me, Thanks for this video

@pa7235 4 ай бұрын

Now I understand! that's cool, thank you!

@vishnudeo1981 2 жыл бұрын

Thanks a lot....it's really useful .

@shivamramdhani8151 2 жыл бұрын

Great work. 💯

@ameeralikhan9288 2 жыл бұрын

Great video, we can use it for bulk delete operation also.

@shivamshah6854 2 жыл бұрын

Thank you for this great video

@yousufbaig821 27 күн бұрын

Very well explained !!

@iamnoob7593 Жыл бұрын

High Quality Content , lIked it , clicked like button

@shehzadali6713 2 жыл бұрын

Please upload videos more frequently and it would really be great if you make a series on pl/sql

@anishanagpal5081 2 жыл бұрын

Thanks for the great explanation and content. If I want to do a project based on rest apis in java, any resource you could recommend ?

@rohitm8965 2 жыл бұрын

Such an amazing explanation. Can you explain how does the upload of large file happens ? What goes in the background when it shows % of Progress?

@vietanh722 11 ай бұрын

Your channel is so simplified and easy to understand. But some video does not have subtitle. So I hope you can add subtitle for anyone can be easy to follow what you're saying. Thanks.

@ISHANICUTIEPIE Жыл бұрын

Nicely explained...

@himachal8252 2 жыл бұрын

I think instead of file upload operation making an order is a better example. The server can return a response with your req has been acknowleged. Meanwhile a lot of operation can take place in bg for eg. Checking if the inventory is available fir that order, in which location the inventory is available, what would be the most optimal location to serve that order from etc.

@reyazahmed4855 2 жыл бұрын

Great video. I had one query: In case of file upload, our api would be sending chunks of data continuously. Then how is it asynchronous?

@deepakgunjal5330 2 жыл бұрын

This is the confusion as the initial request was to upload a file then i would assume file chunks are getting transferred to the server in the same session. In that case would this immediate response not end the session? Or there is something we are missing?

@contactdi8426 2 жыл бұрын

Same question!! Any clarifications?

@luis96xd 3 ай бұрын

Great video 💯 I have a question, I have an system that make and HTTP request to API endpoint, this endpoint loops through a list of codes and makes another http to another API, when it fetches a result list, i saves each item of the list in the database, currently it has 30 codes, and 10 items per request, so it has 300 database writing, and it takes a lot and mostly returns TIMEOUT error, 120 seconds of timeout, Is this a good approach? should I use the approach presented in this video? Or I could try another approaches

@jenaamarnath 2 жыл бұрын

Nice explanation

@saiashok28 2 жыл бұрын

Hi, what is the benefits using the async/await pattern for an API service ?

@HalimaBegum-jk2ej Жыл бұрын

best explanation ever

@ShivaSomapur 2 жыл бұрын

Do you plans to cover application driven sharding for working with data stores/Message Brokers with examples?

@psw1 2 жыл бұрын

One instance where asynchronous APIs are used : while booking an uber , which is a long running process involving searching for cabs, sending the booking request to available drivers etc. Please correct if wrong.

@kuberyadu868 2 ай бұрын

Hey Yogita, this video is super helpful, I have a question, does asynchronous processing increase the efficiency of an api in any way?

@asakhala 2 жыл бұрын

Thanks for this video didi. Can you please make a video in detail of hotel reservation system design. You may start with one for single hotel and can cover in detail searching, booking, payment. What happens in case two people try to book same room, how it is avoided. How can kafka be used for this? Also data model in detail . Please 🙏🏻

@kishorekumarmalla 2 жыл бұрын

Thanks for very informative video,I am regular follower for your videos, Thanks for your efforts. How to get the percentage of status to show in progress bar by using order/{order Id}/status API end point?

@niesh20us 2 жыл бұрын

Hi Yogita, Thanks, nice content. questions here is , Is it safe to say that , this scenario is true for video upload?

@daniyalhassan6942 7 ай бұрын

Hello guys, i have a question, instead of checking the status again and again from the 2nd api, can i use the server-side events. Is it a better approch?

@myLifeSk98 2 жыл бұрын

#50k soon yeeh ❤️ thanks sudocode

@xotic37 4 ай бұрын

GOOD WORK TC

@ak-ot2wn 2 жыл бұрын

I want to upload 100GB big file. I didn't get how both of these implementations will handle it. I understand that once I call an endpoint for uploading a file, I either receive 202 or 303. But I do not understand how this 100GB file gets uploaded after the client receives the 202 or 303 response.

@grifith2196 2 жыл бұрын

If the API is asynchronous, does that mean it can have multiple clients hitting that endpoint and uploading files? Even if there was one http status code, with multiple uploading files, that is similar to having 2 http status codes with a 202 and multiple uploading files, how does having one or two codes affect async operations in a multi threaded env?

@Ankt-nq8tw 2 жыл бұрын

In case of file upload, will file body at same instant or handle internally

@RamKrishna-ou6li 2 жыл бұрын

Hi, thank you so much for presenting in such a simple way. It would be really helpful if you can provide or share resources or the practical implementation of this topic. It will really help a lot. Especially how we can track the progress of the jobs. I have heard about async jobs but I am not sure how to track their progress. For example, suppose a job takes 5 sec to complete and I design an API that will trigger a background job to perform this task and I will send the response to the client that your request has been received please wait for completion. Now in order to make a progress tracker i.e., the API which tracks the progress of this async job I can check the progress of this job at a regular interval or there is any other approach?

@agarwalrahul35 2 жыл бұрын

You can implement check status api, which will check the status of your progress at regular interval of time.

@Rohitkumar-hu1qs 2 жыл бұрын

Another fantastic video. Thanks again. However, you did mention about your header video which I am unable to find on your channel. Can you please share the link in case I am missing something?

@sudocode 2 жыл бұрын

Hey Rohit. The headers video will be released soon. 🙂

@sajjad006 2 жыл бұрын

Hi ma'am, it would be great if you could make a video on the practical implementation of asynchronous APIs. And is there any relationship between async APIs and message queues? I mean can we use message queues to implement async APIs?

@larssonsoder4824 2 жыл бұрын

Yes! This is what I'm looking for too! I've worked with some systems: - Some of them use message queues to be asynchronous - But some of them use REST APIs to insert to the db and return 202, then there are some batch jobs picking these records and process I hope someone can help me clarify the pros and cons of these 2 approaches, when we should use the former and the latter

@kumarshashank3954 2 жыл бұрын

Yes I also want to know that

@subhamburnwal5115 2 жыл бұрын

Same request from my side as well..

@ZahidHasan-cc8tf 2 жыл бұрын

Same request. Please do consider.

@atreyikulshreshth2344 2 жыл бұрын

Same request

@sarath2873 2 жыл бұрын

Hai thank you so much ..Can please make videos on dependency injection concept ..

@anchalpandey3619 2 жыл бұрын

This channel is underated

@tarunstv796 2 жыл бұрын

because the content is :D No code example. No advance real-time use-cases at work etc.

@alokprakash2017 2 жыл бұрын

I found many channels that are more under rated than her.

@aanadinatu383 2 ай бұрын

Maam can you make an elaborate separate video for filtering and pagination . Like a detailed video with code snippets and examples

@vijayvaswani9717 4 ай бұрын

you are awesome 😊

@vijmar 2 жыл бұрын

i have long running transaction. requirement is during in progress few rows needs manual intervention for correction. please let me know your design thoughts

@siddharthgaur7919 2 жыл бұрын

using gRPC streams here would be a good idea, thoughts??

@fullstack_projects Жыл бұрын

hi mam im a ReactJs Developer ,in most of my apps im use json server as a database ..Is json server good for large applications?? in fact i have little knowledge on backend langauges

@sujith5441 2 жыл бұрын

Keep uploading new videos.✌🏾

@VinodKumar-tu1bq Жыл бұрын

asynchronous/synchronous calls are not based on response time. asynchronous calls do not block the caller, while the connection making synchronous call is blocked until a response arrives.

@shantanushende6 2 жыл бұрын

Can you do a video on "Design a system to see real time stock market data(gathering data from various stock exchanges)"

@Ankt-nq8tw 10 ай бұрын

Awesome

@sukeshreddy5297 9 ай бұрын

I have a doubt . is there any limit on the size of the file that can be uploaded this way .

@parvezmulla3324 2 жыл бұрын

interesting use case!

@sudocode 2 жыл бұрын

Thanks Parvez

@djohn0909 2 жыл бұрын

Hi! Could you please make a video covering REST api and GraphQl api topic?

@ritwikdas5443 10 ай бұрын

How ser er handles huge query output?

@deshtechno 3 ай бұрын

So, the video has no answer on how to upload huge data. If I want to upload a huge file from my disk and I get response code 202, how the upload even get started and progress. How to stream data from disk to server according to this api.

@priteshugrankar6815 2 жыл бұрын

Another good example would be withdrawing money from ATM vs withdrawing cash from bank after getting a token.

@YumT-vo3xc 5 ай бұрын

Brilliant

@anandprakashshah8092 Жыл бұрын

hello ma'am actually i want to ask that i want to hit api one time and get data in batch. how can we do?

@KeshavChoudhary-qd3mz 2 жыл бұрын

simple explanation

@sree562 Жыл бұрын

I think instead of uploading a file as a example of asynchronous ,we can consider An order from any e-commerce site and tracking for the shipment with the shipment code or tracking code like that ....

@subee128 Ай бұрын

Thanks

@parth7450 2 жыл бұрын

Pls make video on making dll for c# in command prompt

@ChandraShekhar-by3cd 2 жыл бұрын

Thanks a lot for such an informative video. Could you please upload some full System design vide as well , like UBER, Whatsapp etc , so that we can have some idea about how to tackle the system design rounds at FANG. It's been quite long waiting....... :(

@gokulmahajan1713 2 жыл бұрын

Hello All Could anyone please suggest me a software for windows laptop which can record videos and can have drawing and animation tools similar to the one shown in this video. Would be a great help. Thanks.

@kmishy 2 жыл бұрын

Ma'am I think uploading large file is not best suit for this scenario! Instead, face detection may be the best example for asynchronous request. You upload a photo, and server do their job of matching faces with other available photos at the server

@ganeshvbidve 2 жыл бұрын

Hello Yogita Take some session on REST Security

@anishtaneja5665 2 жыл бұрын

You haven’t talked about asynchronous/await keywords ?

@funnyvideorocks Жыл бұрын

Please elaborate the asynchronous example with another or with code

@PrashantMishra18 8 ай бұрын

This example is exactly how short polling works.

@d.kcoding210 Жыл бұрын

Thank mam

@aayush5474 2 жыл бұрын

Interesting

@devprakash5320 2 жыл бұрын

very well explained

@saurabhahuja6707 2 жыл бұрын

How we are sending 1 gb file in http request. ? Are we doing chunk at client side ? And uploading these chunk one by one and show progess to user? Or we are sending 1gb file directly ? If we are sending 1gb file directly how we show progress to user ? ...please clear these points and thanks for making video

@alokprakash2017 2 жыл бұрын

You can't send 1 GB at once. Program divide large file into chunks and send it one by one to server.

@ameeralikhan9288 2 жыл бұрын

Can you please make LLD design video on fleet management system or vehicle tracking/management system by uber or ola.

@ameeralikhan9288 2 жыл бұрын

Basically i wanted to know convenient way to store GPS data for larger fleet or so many vehicles. If anyone can suggest some link it would be helpful too.

@swaruppulkit 2 жыл бұрын

What you've done here is that you've confused yourself between two different things, Asynchronous Request/Response Pattern, and uploading of a large file using HTTP, where the connection is kept alive till file (/data) has completely uploaded on the server (and/or the file is broken down to chunks and uploaded in parallel request). The difference here is that Asychronous Request/Response Pattern is typically used in places where you anticipate the time to process data on backend will breach the SLA (and the connect would either terminate or hold the process for too long.). Earlier this was also done through HTTP Polling which is typically implemented using HTTP 202 which you've mentioned here but again it is for the downstream (response time) data

@sudocode 2 жыл бұрын

I think the large file upload example has confused folks with streaming uploads. Even if one has to upload a 1 Mb file it can be done via rest api using a json payload and using asynchronous request reply pattern. Thank you for raising this. I can create a part 2 now.

@swaruppulkit 2 жыл бұрын

@@sudocode Not sure. But looking forward to learning :)

@ZeroCommentsZeroDocs7Bugs 7 ай бұрын

Is the status code really 303? Normally that status code is 302.

@Mrsmartyfun 2 жыл бұрын

Can you please share the code sample

@jagadeshkiran764 2 жыл бұрын

What about Call Back API ?

@debmalyapan53 2 жыл бұрын

waiting for your rate limiting video...

@cut-a-lyst 2 жыл бұрын

we use method 2

@rahulsoni-lx5rb 5 ай бұрын

🤩🤩🤩

@vikasnaik5361 2 жыл бұрын

and who will write the code - Mam ? Everyone knows about rest api's

@sudocode 2 жыл бұрын

How about you?

@mohitmehra8114 2 жыл бұрын

Do some implementation

@theredbaron28 2 жыл бұрын

Wouldn't that be a huge task ?

@aayaanlazytornado7913 Жыл бұрын

Echoing Warner and other, this is not Asynchronous API. This is a a custom way of managing long processed by splitting the process into two logically associated APIs. I request the video owner the put out the correction by simply changing the title to - using REST APIs for long processing scenarios.

@sudocode Жыл бұрын

Can you please share resources to support your claim and I would happily change the video title.