Congrats on 30k! Absolutely love these types of videos on your channel. Linux Internals playlist is second to none.

I always use a swap partition under Linux because when I started on my Linux journey all the installers would ask to allocate a swap partition and windows was using a swap file

It seems many of the comments I have received, most believe that either 1) Paging solves fragmentation and 2) Fragmentation has already been solved...since there are so many comments around this discussion, I will do a follow up video addressing both of these questions perhaps as soon as week after next.

You intro graphics get better every episode 😊

Thanks DJ👍That was great! and congrats on 30k.

This was really interesting. I was under the impression that virtual memory had something to do with the kernel obfuscating physical memory addressing from applications in order to non-contiguously allocate application memory. Good to know it's just storage partitioning

Another awesome vid! Thanks DJ! 👍👍

I would like to see a video giving examples of swap file vs swap partition and performance.

Congrat's on 30k!!!..Dedication always pays off!!

What a pleasure! I still remember times than we configure linux to bypass bad SIMMs 🙂 But, where you miss zram? Please, make vid about.

Always great insight

Great explanation!

Thank you for very useful videos. And congratulations on your way up to get more thousands of subscribers!

Since the time I started with Linux the recommendation has always been to setup a separate partition, and being rather old school I've always complied. I've only got 12gb of RAM, though I wanted a more rounded number, so I setup a 32gb swap partition. I think the most I ever saw it get used was 6gb, but that was fairly heavy use with about 20 browser windows, each with about 10 tabs apiece and a media player in the background with some music going, and obviously about 20 terminal windows across 5 desktops, plus an emulator. Usual day to day usage is maybe 5 browser windows at 20 to 30 tabs total and 8 terminals, and even then it's usually not more than 2gb of swap used. If that constitutes a profile in which I've severely over-allocated my swap space, I'd still do it just the same. I've got the disk space and I've never felt it get sluggish. Even on the old computer, when I had 256mb of RAM and a 1gb swap partition, Slackware still felt snappy.

It's possible to use multiple swap partitions. It might be possible to extend a partition by adding a swap file, adhoc.

First things first. I thought this "memory fragmentation problem" was solved many years ago, by implementing MMU - a unit of CPU, that maps and thus disconnects physical addresses from addresses, mapped to the the particular process. In other words, the OS can map any number of separate free memory pages into any number of continuous free memory areas of any length. Having addressable space of a size, that is much greater than physical memory size, it is always possible to pick an address and map free pages there. In IA-32 MMU was introduced at least as early as 80386 some decades ago. And even before that, in ancient pre-NT Windows, that used the real mode x86, the "memory fragmentation problem" was solved by using "handles", that are generally speaking are pointers to pointers (and by concept of locking handles while in use) - when there is no sufficient memory region for allocation, the OS can defragment the memory, patch the internal pointers of handles and thus create a region of a sufficient size. The second thing. Swap is not about memory fragmentation. It is about fitting the unfittable ("впихнуть невпихуемое") - again, solved long ago programmatically with handles and then by hardware using bits and traps of MMU.

Thank you for your KISS narative ;)

Lots of interesting things to know and think about regarding computer memory management. Other interesting policy topics could be copy on write for shared memory between programs, and how much memory is needed to start a new process. I recall Unix V required you to have enough memory (virtual) for the process, but BDS allowed you to start a process if you had enough to satisfy the initial needs.

Thanks again for another great video! 👍 What is your view on vm.swappiness in this context? The default in most Linux distributions is 60, but e.g. Ubuntu recommends setting vm.swappiness to 10 for desktop installations. If you want to deepen this topic even furter: vm.vfs_cache_pressure might be another interesting topic. 🙂

Interesting video, but you didn't quite explain why the pages need to be adjacent in the first place. I've always thought that the paging was invented to address memory fragmentation (together with access rights and virtual memory). Apart from HW related regions (like DMA buffers) and the kernel itself (maybe even its disk cache), the paging tables can be constructed so that each application sees a contiguous block of memory despite of the physical pages being scattered all over the place. As I understand it, the fragmentation might just be an inefficiency of the buddy allocator that the kernel is internally using. But I still don't get what prevents the OS from breaking large block sbrk() calls into many small ones so that in the virtual address space these would form a contiguous block.

Swap partition can be used by other linux, multi boot system, even live os ...

There was a trend for claiming that you didn't need swap any more some time back. Didn't seem a good idea, and indeed it wasn't, so I've been relentlessly old-school with a swap partition twice the size of physical memory for moderately sized machines. But there does come a point where that's no longer sensible. I've got 16G of RAM and 32G of swap on this machine and I don't think that I have *_ever_* seen it use the swap, although it may have done and I never noticed. 😀

My take as well, before @DJWare posts his next video: paging does solve fragmentation to a large extent. According to my investigations Linux really does allocate physical memory in contiguous blocks. There are several cases where this is really needed: (1) kernel datastructures larger than 4k, DMA buffers for devices that do not support scatter/gather DMA, huge pages (e.g. 2MB or 1GB on x86 architectures). There may be others, but these should be the most common. However, I would argue they are not very common. Huge pages are rarely used and the user should be well aware of the possible fallback. Modern DMA-capabe devices should support scatter/gatter. And, in any case, there is a separate memory zone for DMA buffers. The bigges misunderstanding seems to be about what happens when you "launch" an application. And I put launch in quotes because most people would conflate launch with load, and that is not correct. Almost nothing is loaded when a process is launched. The appropriate sections of the executable are merely memory mapped into the process virtual address space. Sure, that would allocate kernel datasructures for the book keeping but these are not huge; it's unlikely that even heavilly fragtmented system will not find memory for these. And there are compaction mechanisms so it will simply slow down the launch. Anyway, the point is that no physical memory is allocated at this point. The pages are makred as non-present and will trap into the OS when accessed giving the OS the chance to load them from the executable file. And since this happens for a single page at a time, only a single physical page will be allocated. Which takes care of fragmentation because you can always allocate a single page (unless you are really out of memory - but that has nothing to do with fragmentation). And the memory mapped sections of the executable look like a read-only extension of your swap. The physical memory backed by them can be freed at any time to make place for some other allocation because they can always be read back from the file. Isn't that neat?! Then, while the application runs it can request memory from the OS. However, in a similar way, no physical memory is being allocated to service the request (apart from the book keeping data which is always relatively small compared to the size of the requested allocations). It's only allocated when it's first accessed. And again, this happens for a specific page. So again, fragmenation will not prevent this allocation. BTW, this mechanism allows Linux to overcommit - i.e. to allocate more memory than it has available (available memory is physical memory + swap + memory mapped files). Not all allocated memory is actually being used and delaying its actual allocation allows the system to pretend to have more memory than it actually does. Well, if it turns the memory is actually being used, bad things will happen. The system will have to choose which process to kill in order to free physical memory. And by Murphy's law, it will always choose the worst possible. In short, if you are a regular user (desktop or server, doesn't matter) you don't have to worry about fragmentation. And you don't have to worry to much about your swap either. It's a good idea to have a couple of Gigs because sometimes the kernel may swap out unused parts of existing processes in order to use the memory for disk cache. But the old rule of twice the RAM is outdated. If you have so little RAM that it would be needed, then you are heavilly into thrashing teritory and your use case demands more RAM. If you have >= 16GB. You don't really need any swap although you will certainly benefit from having some. But only a little. You will never see 32GB of swap being used on a 16GB machine because ti will be unusable even with an NVMe drive. But you should prefer having your swap on a separate block device. Just manage your storage with LVM and it will be a very simple matter to extend or shrink a swap volume, or any volume for that matter. Cheers,

Maybe what you could do is have a table of the official memory address and then the actual memory, so you can move around the actual memory and alter the table mapping the program to actual address. Then you could do on the fly sort of defragmentation of the memory by putting it in a storage region while moving around that many bytes. I admit it would add time and lose memory with the table, so much so it may not be practical. Also there could be a means of buffering to avoid changing cpu caches.

In the graphic at 11:00, there are 3 programs and 3 maps with various pointers to RAM and swap. That map layer makes the RAM and swap appear contiguous to each program. A similar strategy isn't being used to allocate non-contiguous RAM to a program?

What about the difference between heap and stack?

@DJ What I'd like to ask you is on the side of th esystem crashing. IF one uses the HDmem over RAM- what kind of protection does that provide us (if any and I'm assuming there has to be something)??

Now I am confused! I was assuming that PAGING would solve the problem of external fragmentation. Also, I thought the problem was in dynamic allocation, especially for processes that continuously "mallocing" and "freeing."

Hi, Why do physical memory have to be contagious in the virtual memory scheme? With the virtual to physical mapping, can't the physical pages be loaded any where and the Page Table updated accordingly? Even for DMA memory, they have scatter gather for the case where the physical memory isn't contagious. Could you help me understand why the physical pages need to be contagious?

There's no question that swap is an important tool, however I do think that it's also mostly an anachronism from when ram was actually expensive and every system vendor was selling systems starved on ram to compete on price because many people viewed performance simply as clockspeed. I think swap has a place on monolithic server applications where unexpected load could cause downtime as you pointed out. However if you're deploying an application as a collection of smaller redundant components, k8s, microservices, etc, the behavior of each node is well defined and if a node dies it shouldn't really matter, you log it, fix it. If load goes up you spin up more instances, if it goes down you have less of them, at no point should swap matter here. I actually also disagree, and always have (to a degree), in a normal desktop context. Nothing demolishes system performance like swapping to a spinning disk. I was at a client the other day that I found was using a 10+ year old workstation which had a spinning disk. He had a ton of programs running, it was working.... depending on your definition of working lol. It literally took me 10 minutes to get enough stuff closed so that the system was actually usable. Back in the day, your system might have 8MB of ram, even then, it wouldn't take long to saturate that, to a degree swap was necessary for general use cases. But now if your system has 16-32 GB... what amount of swap would ever be relevant here? If you were tempted to add a full 32gb of swap, literally anything I could be doing that would actually take that much memory would be a sign that I'd need more ram instead. Things like rendering, 3d modeling, or whatever because using 32+ is not a one off normal thing that could just happen in normal operation. Applications that need that much temporary storage are generally written to consume it directly, not just shove it all into vram and hope for the best. The proportionality here is just completely different from how it used to be. 32gb is a lot right?.... _it costs 80$_ at that price why would you ever want to allow swap to occur? 32 not enough? sqrew it, shove 128 in there and forget about it.

there is debate these days whether or not one needs a swap partition; the formula for allocating usually is whatever ram you have x2. I have 32GB of ram on my PC , and I do still have a swap partition, and it is 64GB on an SSD . free -m shows swap used 0 :P

never seen my system ever use swap on my 12 gig desktop, so i've usually just always just disable it/unpartition it and reallocate it to the root or home partition one-off tangant, but I hate how on FreeBSD, the system either chugs or freezes until the memory is cleared, with swap or not, all thanks to zfs' high memory usage

Hey thanks very much for this video. I have a question about fixing memory fragmentation issues: Could you not have a temporary file on the computer with a specified size. Then if you had four non-continuous blocks of memory free (the same as in your example 8:26), and you were trying to open a new application that needed four continuous blocks in memory, Could you just move the green block into the file, than into memory again in a different place so that the four blocks would be continuous? Then you could open the application. If that makes sense?

So considering you don't think zram is a good idea, is zswap any better, or would it have the same problems? Zswap at least still uses a regular swapfile / swap partition.

Fragmentation: Is the Linux architecture still so primitive, that it only allows one program load operation with one physical consecutive IO buffer? SWAP: I have 2 SWAP partitions; a 2 GB high priority one on my nvme-SSD and a 18 GB one on my HDD. The last one is basically to allow hibernation, whenever it will be supported by Ubuntu. Now I compromised and I suspend the PC after 30 minutes, but it slowly kills my 2-pin fans running 24/7. I noticed no performance punishment using the nvme-SSD swap partition, if the HDD swap partition gets used, I suddenly feel the urgent need to kill a VM.

3:00 C'mon 20 year-old patches have absolutely nothing to do with today.

Hi I am using Linux because I believe into open source software. Tho recently when I heard the news , I felt like I was scammed. News- "Patches for Baikal refused to be accepted into the Linux kernel for political reasons Jakub Kicinski, the Linux kernel networking subsystem maintainer, refused to accept patches from Sergey Semin, arguing that he felt uncomfortable accepting changes from Baikal Electronics employees or for its hardware. " I mean how come open source software can be political ? Even north Korea people deserve , and should to have be able to have a computer with operating system on it! And I believe there thousands or millions of people are in forbidden countries should be forbidden to use open source software??? How come it's open source then? Open source software should be available to everyone, no matter political reasons, you are poor, you are criminal. Everybody should be available to use it. And to participate in it. Best wishes.