How processes get more memory. (mmap, brk)

Рет қаралды 71,062

6 жыл бұрын

Patreon ➤ / jacobsorber
Courses ➤ jacobsorber.thinkific.com
Website ➤ www.jacobsorber.com
---
We've all called malloc to get more memory for our programs, but what is REALLY happening when we call malloc? This video goes under the hood and shows you how a computer process (running computer program) gets more memory from the operating system.
***
Welcome! I post videos that help you learn to program and become a more confident software developer. I cover beginner-to-advanced systems topics ranging from network programming, threads, processes, operating systems, embedded systems and others. My goal is to help you get under-the-hood and better understand how computers work and how you can use them to become stronger students and more capable professional developers.
About me: I'm a computer scientist, electrical engineer, researcher, and teacher. I specialize in embedded systems, mobile computing, sensor networks, and the Internet of Things. I teach systems and networking courses at Clemson University, where I also lead the PERSIST research lab.
More about me and what I do:
people.cs.clemson.edu/~jsorber/
persist.cs.clemson.edu/ Note that any Amazon links in my video descriptions are generated by Amazon. If you click one of them and then buy something it helps support this channel. Thanks.
***
Welcome! I post videos that help you learn to program and become a more confident software developer. I cover beginner-to-advanced systems topics ranging from network programming, threads, processes, operating systems, embedded systems and others. My goal is to help you get under-the-hood and better understand how computers work and how you can use them to become stronger students and more capable professional developers.
About me: I'm a computer scientist, electrical engineer, researcher, and teacher. I specialize in embedded systems, mobile computing, sensor networks, and the Internet of Things. I teach systems and networking courses at Clemson University, where I also lead the PERSIST research lab.
More about me and what I do:
www.jacobsorber.com
people.cs.clemson.edu/~jsorber/
persist.cs.clemson.edu/
To Support the Channel:
+ like, subscribe, spread the word
+ contribute via Patreon --- [ / jacobsorber ]
+ rep the channel with nerdy merch --- [teespring.com/stores/jacob-so...]
Source code is also available to Patreon supporters. --- [jsorber-youtube-source.heroku...]
Want me to review your code?
Email the code to js.reviews.code@gmail.com. Code should be simple and in one of the following languages: C, C++, python, java, ruby. You must be the author of the code and have rights to post it. Please include the following statement in your email: "I attest that this is my code, and I hereby give Jacob Sorber the right to use, review, post, comment on, and modify this code on his videos."
You can also find more info about code reviews here.
• I want to review your ...

Пікірлер: 86

@akj7 4 жыл бұрын

You should have also mentioned what malloc actually writes into the memory, which is why free doesn't need a size.

@JacobSorber 4 жыл бұрын

Ah, good point. I've been thinking that it would be good to do a more in-depth dive into how allocators work. I'll see what I can do. Thanks.

@gowthamanv8740 5 жыл бұрын

Your way of presentation Awesome. Keep doing more videos. I subscribed.

@qaiyyumhakimi7947 4 жыл бұрын

Your way of explaining things is one of the best that i've found on youtube! Thumbs Up!

@JacobSorber 4 жыл бұрын

Thanks.

@anshumansingh6252 3 жыл бұрын

What a crisp and clean explanation! I am impressed man. Thanks a ton.

@SRedienhcs 4 жыл бұрын

Awesome! I've been looking for a good presentation of this topic for days. Subscribed!

@odaialkhodary7872 7 ай бұрын

I just stumbled upon one of your videos, and man, I went down a rabbit hole with your excellent guidance! Your explanations are so clear and thorough; Thanks for breaking down complex concepts into digestible bits. Your passion for teaching really shines through, and it's making a huge difference in my understanding. Keep up the fantastic work!

@mhanna2 6 жыл бұрын

Found this helpful and enjoyable, thank you!

@JacobSorber 6 жыл бұрын

You're welcome. Glad it helped.

@WatchTower71 4 жыл бұрын

Holy dang dude, you’re literally the most knowledgeable person about C libraries I’ve seen on KZfaq! You should be a teacher or something. Great videos, please keep em coming!

@yannisran7312 4 жыл бұрын

Actually if you click into his profile link, it says he is an associate professor of an univ in the states.

@rabiraju9414 2 күн бұрын

wow, great explanation

@RegularObamahedron 4 жыл бұрын

Clean presentation!

@nupurgupta6373 4 жыл бұрын

amazing information , please more videos on mmap

@saurabhshrivastava224 3 жыл бұрын

Great brother. Keep on doing more..

@lllbenderlll 4 жыл бұрын

please don't stop. I'm not quite good in English but the way you present the INFO is something new and good. PS: please don't stop.

@JacobSorber 4 жыл бұрын

Thanks. I'll try to keep things going.

@MrUmang40 5 жыл бұрын

You are my life savior......your every teaching helps me move a step ahead....you came late on KZfaq.....but at least you came......love you sir.....

@JacobSorber 5 жыл бұрын

Glad I could help.

@MrUmang40 5 жыл бұрын

@@JacobSorberSharing knowledge is a noble job......and you are pulling this off beyond good....thanks for everything.......

@OretYT 5 жыл бұрын

Great video

@shushens 3 жыл бұрын

This is a fantastic video. One of the best you made yet. Strangely, it is one of the videos I watched last. There are two things that I feel have been left out a bit. First, when you showed that increasing the size of the memory being allocated dramatically made malloc use mmap instead of brk, you did not really tell us why that happened :) (I assume it happened because in the first case, there was enough left from the last page allocation, so all that was necessary was to raise the break?) Second, when allocating small chunks (in the order of bytes), I sometimes see multiple mmaps (with strace), but it always ends with brk, which means mmap is a higher level command and cannot function completely independently of brk. A bit of explanation would be super useful :) But regardless of all that, great video!

@majdwatad 5 жыл бұрын

Thanks for the video

@JacobSorber 5 жыл бұрын

You're welcome. Glad you liked it.

@morgengabe1 Жыл бұрын

I once tried to write an "infinite storage glitch" before I learned that the address limit is not actually the physical limit. And of course, I tried to write it in python.

@ruynobrega6918 Жыл бұрын

I think you will like this video: kzfaq.info/get/bejne/gMl6hrpot9bWmaM.html

@kathiravankathir3089 5 жыл бұрын

your profile is awesome

@gonzajuarez4918 3 жыл бұрын

hello, been really enjoying your videos and learning a lot. i've got a question though: should we use munmap once we finished? kinda like free() is used after malloc(). and if so, is it the same for brk/sbrk (like can sbrk called to shrink or does it have a shrinking counterpart). I've done a bit of strace and found than munmap is in fact called by free() but haven't seen a counterpart with brk sbrk (or something like calling sbrk with a negative number perhaps as two's complement of the initially allocated memory)

@44r0n-9 4 жыл бұрын

This video is perfect.

@engines8268 4 жыл бұрын

Great video.

@JacobSorber 4 жыл бұрын

Thanks!

@hamamsofiane7098 5 жыл бұрын

Yes, more of these please !!

@JacobSorber 5 жыл бұрын

Will do. Anything specific you would like to hear more about?

@hamamsofiane7098 5 жыл бұрын

@@JacobSorber maybe more about back end programming, processes, strace, signals, buffer overflows and alike.

@JacobSorber 5 жыл бұрын

@@hamamsofiane7098 Ok. thanks. I'll see what I can do.

@ruynobrega6918 Жыл бұрын

Great content, as always. If you can answer me: could you explain how actual RAM dump actually works? I always wondered that bc of the Program Break.

@kathiravankathir3089 5 жыл бұрын

u r just awesome..i found u late..

@JacobSorber 5 жыл бұрын

Thanks. Better late than never. :)

@MECHANISMUS Жыл бұрын

I get exact amount of bytes from sbrk. Not 4k aligned. Upd: I see it's aligned none the less: when incrementing less then a page and then writing into Third+1 memory, it doesn't segfault; but it does segfault when the increment is full page and thus Third is the real program break.

@broleaf4448 5 жыл бұрын

Thanks for the Precise introduction. As I know current computer has two size of page. Is that true?And how do I know my page size with what command or C code?

@JacobSorber 5 жыл бұрын

You can get the size of standard pages for your machine with getpagesize() on UNIX-style OSes. Not sure how it's done on Windows. As for the other size, I'm guessing you're talking about HugePages, which are usually 4-256MB and are handled a bit differently. Might be a fun topic for a future video.

@rajanjha5797 5 жыл бұрын

I liked this channel and have subscribed it..i have a question can you generate page fault error using mmap??

@JacobSorber 5 жыл бұрын

Thanks, rajan. Welcome. Can you be more specific? A page fault happens when you access a page in your page table that isn't resident in memory. You pay a penalty because the OS has to move the data into RAM from disk. You can use mmap to request pages of memory. How you use them will determine whether or not you have page faults.

@rajanjha5797 5 жыл бұрын

@jacob sorber Thanks for the prompt reply.. i mean to say "major page fault" its an error which occur when page is not found in disk also .

@JacobSorber 5 жыл бұрын

@@rajanjha5797 I still think I'm misunderstanding. A Major Page Fault is when the requested page is not in RAM and the data must be read in from disk (it's slow but it's not an error). A Minor Page Fault is when the data is already in memory, but the page table needs to be updated (much faster, since no disk access is required). If the program tries to access memory that isn't mapped in the page tables, then you get a Segmentation Fault (sometimes called an Invalid Page Fault). You don't need mmap to create Invalid Page Faults. Just dereference NULL or any other unmapped address. I guess a rare case would be when you try to access a page that the OS thinks is out on disk, but then when it goes to the disk it can't find it. That would mean that either the OS messed up or maybe you had a hardware error (disk went bad or RAM failed). That shouldn't happen during normal operation-even with buggy programs.

@rajanjha5797 5 жыл бұрын

@@JacobSorber You are right about it ...its a kind of segmentation fault but of different kind i recently came across of debugging such a situation but now i am not able to generate it ...maybe this link will able to help you understand that which i am talking about : smackerelofopinion.blogspot.com/2010/02/kernel-oops-page-fault-error-codes.html?m=1

@rajanjha5797 5 жыл бұрын

Debugging page fault error is one of the popular question for interviews also would truly appreciate if you can try to explain such a topic as not much is talked about this over net ... :)

@sonmai3526 3 жыл бұрын

3:40 0xDEAD is my new favorite hex number :)

@godnyx117 3 жыл бұрын

Yep!

@moccaloto 3 жыл бұрын

Can the stack size limit be modified at runtime or compile time ?

@jannalorette6980 5 жыл бұрын

I using layered fascia uncle lenny& anty Ingrid I am full scaled

@apoalquaary4438 4 жыл бұрын

Hello Jacob I hope you are fine .. I just had a question .. how can I get the first and last addresses of data, heap and stack and dimensions for C language in my machine? (Linux)

@ravisankar6779 2 жыл бұрын

Hi Jacob, Thanks for your videos. I have one doubt here. In first program, void *second = sbrk(4096) ; here your are trying to increment to 4k bytes. But at 3:38 you are saying that "trying to increase 10byes but page size(4Kbytes) is rounded up. I couldn't able to relate.

@junodb 5 жыл бұрын

Hello Jacob. Thanks for your wonderful videos. I need clarification on the statement where you said adding 10 bytes further using sbrk would round it up to 4k as page size is 4k. I was playing with sbrk and see that adding 10 bytes and then checking the pointer address just shows it incremented by 10 bytes and not 4K, as given in the man page of brk. Is there a better way/example to understand this. Thanks CODE: void *s1 = sbrk(0); //to find the current brk void *s2 = sbrk(4096); //move the brk by 4K i.e. one page void *s3 = sbrk(0); //find the new brk void *s4 = sbrk(10); //move the brk by 10 bytes void *s5 = sbrk(0); //find the new brk printf("First: %p ", s1); printf("Second: %p ", s2); printf("Third: %p ", s3); printf("Fourth: %p ", s4); printf("Fifth: %p ", s5); OUTPUT IS : First: 0xbe5000 Second: 0xbe5000 Third: 0xbe6000 Fourth: 0xbe6000 Fifth: 0xbe600a

@JacobSorber 5 жыл бұрын

Hi Neetish. What operating system (and version) are you using? Some may handle this differently than others.

@JacobSorber 5 жыл бұрын

And, regardless of your answer. My comment refers to how modern virtual memory works. When you move the program break, it's effectively mapping more pages to the process's memory space. And, since pages are fixed size (usually 4KB), that means that the boundary between Ok and SegFault always increases by a multiple of the page size. Even if the OS implementation tells you it moved the program break by 10 bytes, the real boundary either didn't increase at all (because the needed page was already mapped) or it increased by 1 page (4096 bytes).

@RAMB0VI 3 жыл бұрын

Can you elaborate when and why this mmap is used?

@JohnIdlewood 4 жыл бұрын

If one can't read/write data to the space above the Program Break, that means, that he can't access Stack or can he?

@pinksky7501 9 ай бұрын

2:10 According to the official brk manual, the program break is the first location after the end of the process's data segment and the uninitialized data segment. So if we look at the memory layout of processes we see program break isn't the top of the heap. Am I right ?

@thithugiangao3949 2 жыл бұрын

What compiler did you use? Thank you

@brianb9280 2 жыл бұрын

he uses clang

@jiuli5687 3 жыл бұрын

Hi Jacob, I'm experimenting with the same code as yours (using sbrk) to see what will happen if I access an address beyond the program break. But there is no segmentation fault. I'm using Ubuntu 18.04.

@andrewnorris5415 Жыл бұрын

Did you use sbrk(0) ? Also, virtual addresses do exist beyond there - the globals and the code. But there is a MASSIVE gap - to allow expansion of the heap. But if you added a REALLY big amount you could end up in that area of the Virtual memory!

@andrewnorris5415 Жыл бұрын

Also if you allocated a local var AFTER reading the sbrk(0) - then you may be accessing that local var - which is valid in VM.

@zxuiji 2 жыл бұрын

I'm not understanding the point of the program break, I've implemented a memory allocator before using mmap under the hood to get whole pages at a time and just linked them together with some pointers inside them, I never once used sbrk

@ikllnp 2 жыл бұрын

I have been curious of the Thai books you have on your book shelf, what's that about :)

@JacobSorber 2 жыл бұрын

They're Khmer, but yeah, I lived in Cambodia for a few years and try to keep up with the language as best I can.

@ikllnp 2 жыл бұрын

@@JacobSorber Your videos have really helped me crack some coding interviews. Thank you

@MultiNova100 6 жыл бұрын

Didn't understand the part where you said sbrk will (ey?) a page even if supplied an argument smaller than page size.

@JacobSorber 6 жыл бұрын

Memory is allocated to processes in pages (typically 4KB). If you use sbrk to request that the program break move up 10 bytes, the kernel can't give you just 10 bytes. If it gives you anything, it will give you a multiple of 4KB. That's all I was saying.

@MultiNova100 6 жыл бұрын

Jacob Sorber so you are implying the sbrk return value will always be a pointer to the beginning of a new page?

@JacobSorber 6 жыл бұрын

I'm not sure about that. Effectively, yes, the kernel will only hand out memory in full pages. But, I suppose an OS could allow the program break to be different from what is actually mapped. What I mean here is this. Let's say you request the program break be increased by 5B. One option would be to just increment it by 5B and then check to see if another page needs to be mapped to ensure you won't seg fault when you access those 5 new bytes. Alternatively, it would just round up to the nearest 4096B and move the break appropriately. I'm not sure what that would benefit you, though. Try it out and see.

@shruthiabirami232 4 жыл бұрын

what is first and second ? to what data type they belong to ?

@JacobSorber 4 жыл бұрын

They are (void*), so just pointers (memory addresses) with no implied type information about what they point to.

@shruthiabirami232 4 жыл бұрын

@@JacobSorber what is uint8_t ? can u explain what is fd and offset ?

@JacobSorber 4 жыл бұрын

@@shruthiabirami232 uint8_t is an unsigned 8-bit int. I have a rather old video about these types (kzfaq.info/get/bejne/n75hqKqjm7nNYHk.html). As for, fd and offset, they relate to memory mapped files. More about that here (kzfaq.info/get/bejne/o511bNOarcnKomg.html)

@bonbonpony 3 жыл бұрын

So, if I `mmap` some memory as writable and executable, and then put some machine code in it, can I then jump into that code and execute it? :q Of course, it would be better to change that memory to read-only after I prepared the executable code in there. How can I change permissions on a pre-existing memory mapping? And how can I actually jump to that code afterwards? Is it enough if I make a fake function pointer type, and then use the address returned by `mmap` like this?: typedef int (*Func)(int x); Func *code = mmap(NULL, 4096, PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); /* populate it with some code */ /* change its permissions to PROT_READ | PROT_EXEC - how? */ int retval = code(123);

@bonbonpony 3 жыл бұрын

Answering my own question again :J The protection flags can be changed this way: mprotect(addr, size, PROT_READ | PROT_EXEC);

@subhankarkarmakar2650 2 жыл бұрын

#include #include int main(void) { void *first = sbrk(0) ; // will tell where the memory break currently is void *second = sbrk(2) ; // move the break above/more by 2 bytes void *third = sbrk(0) ; // again tell where the memory break currently is printf("the memory break currently is in : %p ", first) ; printf("moved the break above/more by 2 bytes, new position : %p ", third) ; printf("trying to use beyond this region will lead to segfault ! ") ; int *ptr = (int *) third + 1 ; *ptr = 2 ; printf("%p ", ptr) ; return 0 ; } I wrote this, but it did not segfault, why ?