multiprocessing: fork() vs. spawn() (intermediate) anthony explains

Рет қаралды 14,533

Күн бұрын

today I go over the low level details on the difference between fork and spawn as well as how that's relevant for flake8!
playlist: • anthony explains
==========
twitch: / anthonywritescode
dicsord: / discord
twitter: / codewithanthony
github: github.com/asottile
stream github: github.com/anthonywritescode
I won't ask for subscriptions / likes / comments in videos but it really helps the channel. If you have any suggestions or things you'd like to see please comment below!

Пікірлер: 31

@Jakub1989YTb Жыл бұрын

At first, I wasn't really fond of the new video thumbnails. But I can say, they grew on me 🙂

@Itwasmeeeeeee Жыл бұрын

Glad to see your paint game has improved !

@sadhlife Жыл бұрын

I'm glad to see how simple the config parsing part of flake8 is. Pylint on the other hand, has it split in so many parts it's hard to even keep in your head. There's actually two different config parsing modules, one that uses argparse, and an old one that uses optparse.. (Granted they'll be releasing 3.0 soon that removes the old one)

@anthonywritescode Жыл бұрын

it's most of what I've worked on the last few years

@Quarky_ Жыл бұрын

Thanks! I never understood the difference, now it's pretty clear :)

@justinmusti 8 ай бұрын

I should have watched this video like a week earlier. it would have saved me all this rework. Thanks for the video. Great explanation.

@Khushpich 5 ай бұрын

Thanks Anthony, great explanation

@d3stinYwOw Жыл бұрын

MacOS is only compatible with ancient versions of POSIX, so... Plus, do you run Paint in VM or in Wine? :D Great video as always!

@liangyu3771 9 ай бұрын

love the video

@AceofSpades5757 Жыл бұрын

On Windows, we're used to being slow.

@alexandrugheorghe5610 11 ай бұрын

Wholesome.

@amir.hessam Жыл бұрын

nice

@snek3205 Жыл бұрын

What about in fork, if you modify the global variable between entering the Pool context and running the map? (i.e. I do `global glob; glob = 3` before calling print(list(p.map(...)))) I would guess, based on the paint explanation, that the child process is still watching the memory of the parent process, so it will print [4,5,6]. But when I tried it I still got [3,4,5]. Thanks!

@anthonywritescode Жыл бұрын

once forked the memory spaces are separate -- writes in the parent won't be reflected in the child

@agarbanzo360 Жыл бұрын

How does fork’s ROW memory interact with reference counting or other intricacies of Python memory management?

@anthonywritescode Жыл бұрын

in general, poorly -- an update to the reference count of an object pages in the whole PyObject* structure. there's ways to help this with `gc.freeze()` which I plan to cover in another video. I've also seen an alternative python implementation optimized for fork-based workloads (think like uwsgi / apache) which moves the refcounts off of the objects and into a reserved area of memory -- though this trades off recount lookup speed for memory and memory is usually cheaper.

@agarbanzo360 Жыл бұрын

@@anthonywritescode makes sense, thanks! I did a test and found that using fork() memory would grow over time, not all the way to the memory used by spawn() but enough that I would rather pay the upfront cost and deal with higher but stable memory use than trying to rely on fork() preserving memory.

@malakarakesh3139 6 ай бұрын

hey at 5:04 could you explain the OOM issue in fork() and not in spawn(). because spawning also requires a whole new process to start right? and the memory is limited.

@anthonywritescode 6 ай бұрын

when using fork the original process is copied, with spawn it starts from 0

@malakarakesh3139 6 ай бұрын

thanks.@@anthonywritescode . also want to know how the __name__ works with multiprocessing.. because I assume that spawned process also gets the __name__ as __main__. then shouldn't the child process recursively spawn another child process and son on?

@anthonywritescode 6 ай бұрын

are you observing that? I think you can answer your own question

@malakarakesh3139 6 ай бұрын

interestingly that (recursive spawning) is not happening.. and I was wondering why.. perhaps you could help me with it.@@anthonywritescode

@anthonywritescode 6 ай бұрын

clearly it doesn't set name to `__main__`

@AdamMaczko Жыл бұрын

Hi, I have to use spawn in one of my projects (because of CUDA). I need to spawn multiple processes, but when the first process is spawned it runs through main and terminates and no other process is spawned. I don't really get why this happens. Why could this be the case?

@anthonywritescode Жыл бұрын

it's impossible to know without seeing your code -- maybe stop by the discord with a minimal example? kzfaq.info/get/bejne/qM-ko5eYpteyfX0.html

@AdamMaczko Жыл бұрын

@@anthonywritescode yeah definitely

@TheAulto Жыл бұрын

Ohh no! Thumbnail and video title are off by one! (491 vs 492)

@anthonywritescode Жыл бұрын

oops at least it was just the title being wrong

@malakarakesh3139 6 ай бұрын

from multiprocessing import Process global_variable = 10 def modify_global(): global global_variable global_variable += 5 print(f"Child process: Modified global_variable to {global_variable}") if __name__ == "__main__": global_variable = 4 print(f"Parent process: Original global_variable is {global_variable}") child_process = Process(target=modify_global) child_process.start() child_process.join() print(f"Parent process: After child process, global_variable is still {global_variable}") In the above code, the child process can still access the global_variable assignment in the __main__ function. I thought I understood that the child process only cares about the program state from the line where it is spawned - and also the global variables - here the glob var assignment is before it is spawned and still it accessed the modified glob variable

@anthonywritescode 6 ай бұрын

it depends whether you're using fork or spawn. the default depends on your python version and operating system