Congrats on 30k! Absolutely love these types of videos on your channel. Linux Internals playlist is second to none.
@darukutsu Жыл бұрын
whaaat, for some reason I thought he was over 90k.
@CyberGizmo Жыл бұрын
welcome
@JohnnieWalkerGreen Жыл бұрын
@@darukutsu +1
@oscs4556 Жыл бұрын
I always use a swap partition under Linux because when I started on my Linux journey all the installers would ask to allocate a swap partition and windows was using a swap file
@arteeFartee-e9 Жыл бұрын
SAME! 👋Trying to get NetBSD running on an old Motorola 68030 (yes, I tried re-using a Mac IIci!) and it you didn't know and master how to setup all, and I mean ALL your partitions prior to bring up the OS for the first time, it just wouldn't boot. Failed over and over, back to drawing board, repartition, and start unzipping things, making things, building things.
@CyberGizmo Жыл бұрын
yep, my first encounters with setting up partitions were the same, or two week later my root partition would be full, because I forgot to partition the log files to its own partition, gotta love the learning curve on UNIX and Linux
@CyberGizmo Жыл бұрын
It seems many of the comments I have received, most believe that either 1) Paging solves fragmentation and 2) Fragmentation has already been solved...since there are so many comments around this discussion, I will do a follow up video addressing both of these questions perhaps as soon as week after next.
@admercs Жыл бұрын
You intro graphics get better every episode 😊
@guilherme5094 Жыл бұрын
Thanks DJ👍That was great! and congrats on 30k.
@HaydenLikeHey Жыл бұрын
This was really interesting. I was under the impression that virtual memory had something to do with the kernel obfuscating physical memory addressing from applications in order to non-contiguously allocate application memory. Good to know it's just storage partitioning
@CyberGizmo Жыл бұрын
I going to do another video extending more on this topic in a couple of weeks, I hope
@rashie Жыл бұрын
Another awesome vid! Thanks DJ! 👍👍
@First_Principals Жыл бұрын
I would like to see a video giving examples of swap file vs swap partition and performance.
@dawnjustme Жыл бұрын
Congrat's on 30k!!!..Dedication always pays off!!
@CyberGizmo Жыл бұрын
Well all I can say is thanks to you all!
@eugenesmirnov252 Жыл бұрын
What a pleasure! I still remember times than we configure linux to bypass bad SIMMs 🙂 But, where you miss zram? Please, make vid about.
@wendellbirkett4224 Жыл бұрын
Always great insight
@Siskiyous6 Жыл бұрын
Great explanation!
@constakalmykov8318 Жыл бұрын
Thank you for very useful videos. And congratulations on your way up to get more thousands of subscribers!
@anon_y_mousse Жыл бұрын
Since the time I started with Linux the recommendation has always been to setup a separate partition, and being rather old school I've always complied. I've only got 12gb of RAM, though I wanted a more rounded number, so I setup a 32gb swap partition. I think the most I ever saw it get used was 6gb, but that was fairly heavy use with about 20 browser windows, each with about 10 tabs apiece and a media player in the background with some music going, and obviously about 20 terminal windows across 5 desktops, plus an emulator. Usual day to day usage is maybe 5 browser windows at 20 to 30 tabs total and 8 terminals, and even then it's usually not more than 2gb of swap used. If that constitutes a profile in which I've severely over-allocated my swap space, I'd still do it just the same. I've got the disk space and I've never felt it get sluggish. Even on the old computer, when I had 256mb of RAM and a 1gb swap partition, Slackware still felt snappy.
@OlivierVogt Жыл бұрын
It's possible to use multiple swap partitions. It might be possible to extend a partition by adding a swap file, adhoc.
@CyberGizmo Жыл бұрын
Yes that is theoretically possible I think you would need to set an order of priority though, have never tried it myself, I prefer to keep things sinple
@user-lk4ls2xx5y Жыл бұрын
First things first. I thought this "memory fragmentation problem" was solved many years ago, by implementing MMU - a unit of CPU, that maps and thus disconnects physical addresses from addresses, mapped to the the particular process. In other words, the OS can map any number of separate free memory pages into any number of continuous free memory areas of any length. Having addressable space of a size, that is much greater than physical memory size, it is always possible to pick an address and map free pages there. In IA-32 MMU was introduced at least as early as 80386 some decades ago. And even before that, in ancient pre-NT Windows, that used the real mode x86, the "memory fragmentation problem" was solved by using "handles", that are generally speaking are pointers to pointers (and by concept of locking handles while in use) - when there is no sufficient memory region for allocation, the OS can defragment the memory, patch the internal pointers of handles and thus create a region of a sufficient size. The second thing. Swap is not about memory fragmentation. It is about fitting the unfittable ("впихнуть невпихуемое") - again, solved long ago programmatically with handles and then by hardware using bits and traps of MMU.
@CyberGizmo Жыл бұрын
I wish it was solved, I just checked a 12th gen Intel CPU running Linux, first day after boot my fragmentation was 4%, one day later, this is what I see: pi@blueice:~$ uptime; sudo ./show_frag.sh 18:38:06 up 1 day, 16:48, 4 users, load average: 0.00, 0.00, 0.00 DMA=4% DMA32=67% Normal=65% And this is using Linux kerel 5.15 the one with the compaction fix. I would say... not solved
@geoffreyrenemoiens3089 Жыл бұрын
Thank you for your KISS narative ;)
@CyberGizmo Жыл бұрын
Any time!
@ericanderson2482 Жыл бұрын
Lots of interesting things to know and think about regarding computer memory management. Other interesting policy topics could be copy on write for shared memory between programs, and how much memory is needed to start a new process. I recall Unix V required you to have enough memory (virtual) for the process, but BDS allowed you to start a process if you had enough to satisfy the initial needs.
@CyberGizmo Жыл бұрын
That sounds like something I would be interested in doing, thank you Eric for the suggestion will add it to the list
@crazycatman4171 Жыл бұрын
Thanks again for another great video! 👍 What is your view on vm.swappiness in this context? The default in most Linux distributions is 60, but e.g. Ubuntu recommends setting vm.swappiness to 10 for desktop installations. If you want to deepen this topic even furter: vm.vfs_cache_pressure might be another interesting topic. 🙂
@32bits-of-a-bus59 Жыл бұрын
Interesting video, but you didn't quite explain why the pages need to be adjacent in the first place. I've always thought that the paging was invented to address memory fragmentation (together with access rights and virtual memory). Apart from HW related regions (like DMA buffers) and the kernel itself (maybe even its disk cache), the paging tables can be constructed so that each application sees a contiguous block of memory despite of the physical pages being scattered all over the place. As I understand it, the fragmentation might just be an inefficiency of the buddy allocator that the kernel is internally using. But I still don't get what prevents the OS from breaking large block sbrk() calls into many small ones so that in the virtual address space these would form a contiguous block.
@CyberGizmo Жыл бұрын
yeah to some degree but its the contiguous part that prevents fragmentation, otherwise you would have 4K blocks all over the place, with the system having to work harder to locate them
@32bits-of-a-bus59 Жыл бұрын
@@CyberGizmo I thought it would not matter. The paging tables make it fragmentation-tolerant. The right ordering of those 4KB blocks is stored in the paging tables and the mapping is performed transparently by the CPU using its TLB cache. The only issue would be page coloring so that it would fit nicely into CPU's data cache of limited associativity (if I remember correctly, BSD does this). This may restrict our choice of which 4KB block to allocate next. Another reason for preferring contiguous blocks would be if we wanted to allocate memory without storing detailed virt->phys mapping for each process. I can imagine that the OS might want to keep this info in a compact form of spans (although it would mean an extra work of expanding it to its detailed form on each task switch -- at least on i386 architecture (AMD64 offers more possibilities, I guess)).
@jyvben1520 Жыл бұрын
Swap partition can be used by other linux, multi boot system, even live os ...
@calmeilles Жыл бұрын
There was a trend for claiming that you didn't need swap any more some time back. Didn't seem a good idea, and indeed it wasn't, so I've been relentlessly old-school with a swap partition twice the size of physical memory for moderately sized machines. But there does come a point where that's no longer sensible. I've got 16G of RAM and 32G of swap on this machine and I don't think that I have *_ever_* seen it use the swap, although it may have done and I never noticed. 😀
@alexxx4434 Жыл бұрын
If you don't ever fully load your RAM, then the fragmentation shouldn't be a problem. The valuable point agaist using swap, not mentioned in this video, is that it is very slow compared to RAM. So, killing another app might just be faster that resorting to swap (not in a stable server environment obviously). But then again, in server environment you better allocate as much RAM as to never resort to swap preferably.
@CyberGizmo Жыл бұрын
I would contend if the program is idle or waiting on I/O, the paging to swap in not a factor in performance, unless you start seeing thrashing which I have talked about before. If the program isn't doing anything, the swap won't even matter to you. If the program is busy executing the algorithm will second chance the page swap (meaning it will not swap this time) but it will come back and check to see if it is idle, that why they put the pagetypes in Linux.
@darukutsu Жыл бұрын
Also be aware that sometimes due to "swappiness" default 60 is swap used, even if plenty of RAM is free. At least for me I had 2/16 GB + 2 GB of swap used which could lead to SSD degradation IMO. So setting to "20" was ok for me.
@nmihaylove Жыл бұрын
My take as well, before @DJWare posts his next video: paging does solve fragmentation to a large extent. According to my investigations Linux really does allocate physical memory in contiguous blocks. There are several cases where this is really needed: (1) kernel datastructures larger than 4k, DMA buffers for devices that do not support scatter/gather DMA, huge pages (e.g. 2MB or 1GB on x86 architectures). There may be others, but these should be the most common. However, I would argue they are not very common. Huge pages are rarely used and the user should be well aware of the possible fallback. Modern DMA-capabe devices should support scatter/gatter. And, in any case, there is a separate memory zone for DMA buffers. The bigges misunderstanding seems to be about what happens when you "launch" an application. And I put launch in quotes because most people would conflate launch with load, and that is not correct. Almost nothing is loaded when a process is launched. The appropriate sections of the executable are merely memory mapped into the process virtual address space. Sure, that would allocate kernel datasructures for the book keeping but these are not huge; it's unlikely that even heavilly fragtmented system will not find memory for these. And there are compaction mechanisms so it will simply slow down the launch. Anyway, the point is that no physical memory is allocated at this point. The pages are makred as non-present and will trap into the OS when accessed giving the OS the chance to load them from the executable file. And since this happens for a single page at a time, only a single physical page will be allocated. Which takes care of fragmentation because you can always allocate a single page (unless you are really out of memory - but that has nothing to do with fragmentation). And the memory mapped sections of the executable look like a read-only extension of your swap. The physical memory backed by them can be freed at any time to make place for some other allocation because they can always be read back from the file. Isn't that neat?! Then, while the application runs it can request memory from the OS. However, in a similar way, no physical memory is being allocated to service the request (apart from the book keeping data which is always relatively small compared to the size of the requested allocations). It's only allocated when it's first accessed. And again, this happens for a specific page. So again, fragmenation will not prevent this allocation. BTW, this mechanism allows Linux to overcommit - i.e. to allocate more memory than it has available (available memory is physical memory + swap + memory mapped files). Not all allocated memory is actually being used and delaying its actual allocation allows the system to pretend to have more memory than it actually does. Well, if it turns the memory is actually being used, bad things will happen. The system will have to choose which process to kill in order to free physical memory. And by Murphy's law, it will always choose the worst possible. In short, if you are a regular user (desktop or server, doesn't matter) you don't have to worry about fragmentation. And you don't have to worry to much about your swap either. It's a good idea to have a couple of Gigs because sometimes the kernel may swap out unused parts of existing processes in order to use the memory for disk cache. But the old rule of twice the RAM is outdated. If you have so little RAM that it would be needed, then you are heavilly into thrashing teritory and your use case demands more RAM. If you have >= 16GB. You don't really need any swap although you will certainly benefit from having some. But only a little. You will never see 32GB of swap being used on a 16GB machine because ti will be unusable even with an NVMe drive. But you should prefer having your swap on a separate block device. Just manage your storage with LVM and it will be a very simple matter to extend or shrink a swap volume, or any volume for that matter. Cheers,
@CyberGizmo Жыл бұрын
Just checked one of my less busy servers: This one is a Debian 11.6 server running a database (mariadb) DMA=4% DMA32=65% Normal=73% 08:54:40 up 2 days, 3:12, 1 user, load average: 0.02, 0.07, 0.08 My systems do not believe paging solves memory fragmentation and neither do I
@CyberGizmo Жыл бұрын
Swap ok you are misunderstanding me. I said in the video I start with the 2x memory for swap file size UNTIL I study how much I need based on the application mix for that system, and believe me they are all different. I did NOT say use 2x physical memory for swap. Go watch the video again
@nmihaylove Жыл бұрын
@@CyberGizmo Ok, probably not being native English misleads me a little. But you do not comment on the content of my post. Only on the headline. There I describe why I think fragmentation is solved. Not because it is not present. But because it is not the problem you make it out to be. It's not a problem. That's what I mean by solved.
@nmihaylove Жыл бұрын
@@CyberGizmo the swap part of my post is not only a reply to your video. I am also addressing the concerns of other posters. I do agree that you should consider your own workload and adjust accordingly. And it may turn out that you can benefit from higher quantities of swap, e.g. if you have a program that leaks memory. Having enough swap might let it run without slowing it down too much because leaked memory is by definition memory is never again used. So it can be swapped out and never brought back in again. But as a rule of thumb, staring with 2x your RAM is outdated. For most use cases it simply is. Most people need only a couple of gigs. That's what I mean. Not that it always is. Cheers,
@CyberGizmo Жыл бұрын
@@nmihaylove It still exists today, it is better than it used to be, but I still see it and it is still affecting performance of my servers., what more is there to say,. I am not sharing an opinion, I have documented proof it exists still to this day
@MatthewSuffidy Жыл бұрын
Maybe what you could do is have a table of the official memory address and then the actual memory, so you can move around the actual memory and alter the table mapping the program to actual address. Then you could do on the fly sort of defragmentation of the memory by putting it in a storage region while moving around that many bytes. I admit it would add time and lose memory with the table, so much so it may not be practical. Also there could be a means of buffering to avoid changing cpu caches.
@collectorguy3919 Жыл бұрын
In the graphic at 11:00, there are 3 programs and 3 maps with various pointers to RAM and swap. That map layer makes the RAM and swap appear contiguous to each program. A similar strategy isn't being used to allocate non-contiguous RAM to a program?
@catsupchutney Жыл бұрын
What about the difference between heap and stack?
@cleightthejw2202 Жыл бұрын
@DJ What I'd like to ask you is on the side of th esystem crashing. IF one uses the HDmem over RAM- what kind of protection does that provide us (if any and I'm assuming there has to be something)??
@JohnnieWalkerGreen Жыл бұрын
Now I am confused! I was assuming that PAGING would solve the problem of external fragmentation. Also, I thought the problem was in dynamic allocation, especially for processes that continuously "mallocing" and "freeing."
@CyberGizmo Жыл бұрын
What do you mean by external fragmentation?
@JohnnieWalkerGreen Жыл бұрын
@@CyberGizmo From the Silberschatz book: "Fragmentation in which available memory contains holes that together have enough free space to satisfy a request but no single hole is large enough to satisfy the request. More generally, the fragmentation of an address space into small, less usable chunks."
@deepakr82619 ай бұрын
Hi, Why do physical memory have to be contagious in the virtual memory scheme? With the virtual to physical mapping, can't the physical pages be loaded any where and the Page Table updated accordingly? Even for DMA memory, they have scatter gather for the case where the physical memory isn't contagious. Could you help me understand why the physical pages need to be contagious?
@entelin Жыл бұрын
There's no question that swap is an important tool, however I do think that it's also mostly an anachronism from when ram was actually expensive and every system vendor was selling systems starved on ram to compete on price because many people viewed performance simply as clockspeed. I think swap has a place on monolithic server applications where unexpected load could cause downtime as you pointed out. However if you're deploying an application as a collection of smaller redundant components, k8s, microservices, etc, the behavior of each node is well defined and if a node dies it shouldn't really matter, you log it, fix it. If load goes up you spin up more instances, if it goes down you have less of them, at no point should swap matter here. I actually also disagree, and always have (to a degree), in a normal desktop context. Nothing demolishes system performance like swapping to a spinning disk. I was at a client the other day that I found was using a 10+ year old workstation which had a spinning disk. He had a ton of programs running, it was working.... depending on your definition of working lol. It literally took me 10 minutes to get enough stuff closed so that the system was actually usable. Back in the day, your system might have 8MB of ram, even then, it wouldn't take long to saturate that, to a degree swap was necessary for general use cases. But now if your system has 16-32 GB... what amount of swap would ever be relevant here? If you were tempted to add a full 32gb of swap, literally anything I could be doing that would actually take that much memory would be a sign that I'd need more ram instead. Things like rendering, 3d modeling, or whatever because using 32+ is not a one off normal thing that could just happen in normal operation. Applications that need that much temporary storage are generally written to consume it directly, not just shove it all into vram and hope for the best. The proportionality here is just completely different from how it used to be. 32gb is a lot right?.... _it costs 80$_ at that price why would you ever want to allow swap to occur? 32 not enough? sqrew it, shove 128 in there and forget about it.
@entelin Жыл бұрын
Great video though, I don't want to sound negative, I always enjoy your talks on these topics!
@CyberGizmo Жыл бұрын
All I can say is, Fragmentation is still a thing, and swapping helps more than not having a swap. And yeah memory is cheaper but so are nVME drives. I am planning to do a more in-depth on this. One thing about the linux community I notice KZfaqrs will say "This is the truth....just do this....." but have you ever noticed they never back it up with facts and experiments using their systems?
@CyberGizmo Жыл бұрын
@@entelin hahah that's ok post away I love the comments
@breadmoth6443 Жыл бұрын
there is debate these days whether or not one needs a swap partition; the formula for allocating usually is whatever ram you have x2. I have 32GB of ram on my PC , and I do still have a swap partition, and it is 64GB on an SSD . free -m shows swap used 0 :P
@CyberGizmo Жыл бұрын
LOL yeah there sure is, but have you noticed the ones saying this and that never back it up with experiments using their own systems?
@polinskitom2277 Жыл бұрын
never seen my system ever use swap on my 12 gig desktop, so i've usually just always just disable it/unpartition it and reallocate it to the root or home partition one-off tangant, but I hate how on FreeBSD, the system either chugs or freezes until the memory is cleared, with swap or not, all thanks to zfs' high memory usage
@mytech6779 Жыл бұрын
Same for me. I actually had worse performance with swap on because on rare occation something would get moved to disk to make room for some other runaway garbage program and then it would be very reluctant to move stuff back to RAM after more memory was free. With swap off the only problem is that the runaway application gets killed, and to me killing it quick is better than trying to manually kill it with extreme swap-lag.
@TheOfficialNathanMacLeod Жыл бұрын
Hey thanks very much for this video. I have a question about fixing memory fragmentation issues: Could you not have a temporary file on the computer with a specified size. Then if you had four non-continuous blocks of memory free (the same as in your example 8:26), and you were trying to open a new application that needed four continuous blocks in memory, Could you just move the green block into the file, than into memory again in a different place so that the four blocks would be continuous? Then you could open the application. If that makes sense?
@CyberGizmo Жыл бұрын
I have not tried using a temp file as swap, One thing I would need to check, is if it is located inside the /tmp directory, those files are cleared on each reboot, so you might run into trouble where your swapfile is gone. As for the 2nd question if the green block of memory is labeled as moveable, yes that could happen too.
@GeekIWG Жыл бұрын
So considering you don't think zram is a good idea, is zswap any better, or would it have the same problems? Zswap at least still uses a regular swapfile / swap partition.
@CyberGizmo Жыл бұрын
I didn't say I didn't like it I said I don't think its a good idea, and that would include zswap as well since it results in dynamic allocation of physical memory.
@bertnijhof5413 Жыл бұрын
Fragmentation: Is the Linux architecture still so primitive, that it only allows one program load operation with one physical consecutive IO buffer? SWAP: I have 2 SWAP partitions; a 2 GB high priority one on my nvme-SSD and a 18 GB one on my HDD. The last one is basically to allow hibernation, whenever it will be supported by Ubuntu. Now I compromised and I suspend the PC after 30 minutes, but it slowly kills my 2-pin fans running 24/7. I noticed no performance punishment using the nvme-SSD swap partition, if the HDD swap partition gets used, I suddenly feel the urgent need to kill a VM.
@CyberGizmo Жыл бұрын
I have trouble recognizing any perceptual delay....now if it I was using one of the old DEC drives from 1970's oh yeah I would notice
@bertnijhof5413 Жыл бұрын
@@CyberGizmo I would not know, I only used more modern 1978's DEC drives on a PDP11/34 :) My swap problem is related to swapping a huge amount of pages at once, when you start one VM too many from the same HDD :)
@weekendwarrior3420 Жыл бұрын
3:00 C'mon 20 year-old patches have absolutely nothing to do with today.
@CyberGizmo Жыл бұрын
And why is that?
@Rom2Serge Жыл бұрын
Hi I am using Linux because I believe into open source software. Tho recently when I heard the news , I felt like I was scammed. News- "Patches for Baikal refused to be accepted into the Linux kernel for political reasons Jakub Kicinski, the Linux kernel networking subsystem maintainer, refused to accept patches from Sergey Semin, arguing that he felt uncomfortable accepting changes from Baikal Electronics employees or for its hardware. " I mean how come open source software can be political ? Even north Korea people deserve , and should to have be able to have a computer with operating system on it! And I believe there thousands or millions of people are in forbidden countries should be forbidden to use open source software??? How come it's open source then? Open source software should be available to everyone, no matter political reasons, you are poor, you are criminal. Everybody should be available to use it. And to participate in it. Best wishes.
@shallex5744 Жыл бұрын
the "open source" "movement" is literally a political co-option of the Free Software movement and always has been