1 Billion Rows Challenge

Рет қаралды 109,851

4 ай бұрын

Recorded live on twitch, GET IN
/ theprimeagen
Become a backend engineer. Its my favorite site
boot.dev/?promo=PRIMEYT
This is also the best way to support me is to support yourself becoming a better backend engineer.
MY MAIN YT CHANNEL: Has well edited engineering videos
/ theprimeagen
Discord
/ discord
Have something for me to read or react to?: / theprimeagenreact
Kinesis Advantage 360: bit.ly/Prime-Kinesis
Hey I am sponsored by Turso, an edge database. I think they are pretty neet. Give them a try for free and if you want you can get a decent amount off (the free tier is the best (better than planetscale or any other))
turso.tech/deeznuts

Пікірлер: 284

@AlecAkin 4 ай бұрын

As a Python connoisseur, let me loop through this and get back to you guys in about a millennium.

@AlecAkin 4 ай бұрын

Though perhaps Mojo might be better for my health

@MrGeometres 4 ай бұрын

Should be easy with pandas/polars/pyarrow.

@CottidaeSEA 4 ай бұрын

@@MrGeometres No external libraries.

@cryptonative 4 ай бұрын

If you use asyncio you might solve it in two millenia

@wlockuz4467 4 ай бұрын

As a JS cook, let me pull out my .reduce I will see you in a 0.99 millennium

@wlockuz4467 4 ай бұрын

Just checked out the repo author, works at a company called Decodable that specialises in real-time stream processing Definitely not sus lmao

@vercolit 4 ай бұрын

Lmaoooo

@ryanshea5221 4 ай бұрын

It could be a job application

@eddster2309 4 ай бұрын

Decodable is powered by Apache Fink which is written in Java and Scala, not sus what so ever

@wlockuz4467 4 ай бұрын

@@eddster2309 Haha I didn't even look that far

@rich1051414 3 ай бұрын

Real-time stream processing would allow the file reading to occur on one thread while another thread processes it. As long as the program's calculations are faster than the file IO, it would be just as fast as the file io.

@mdfalexis 4 ай бұрын

This is 100% for a company

@rawallon 4 ай бұрын

Pffft, what do you meann? No way a company would get people to do job them for free, its just take home test

@wlockuz4467 4 ай бұрын

Absolutely, the choice of Java makes its very obvious.

@JohnSmith-pn2vl 4 ай бұрын

netflix

@SpikeTaunt 4 ай бұрын

I don't see how this would be useful for a company in any way, there's already faster ways to load big data into a useful format

@kphaxx 4 ай бұрын

Equally likely is someone just issuing a challenge to settle an argument. Just some guy

@doublel6545 4 ай бұрын

Craziest thing is the fact that the fastest implementation is less than 6.2 seconds and it keeps improving.

@gergogyalus7707 4 ай бұрын

Dotnet ones are going sub-3s last time I saw lmao

@vitalyl1327 4 ай бұрын

A valid C++ implementation is below 0.2s. Does not look good for JVM, does it?

@stariyczedun 4 ай бұрын

@@vitalyl1327 reading 12 GB file in 0.2s means your SSD read speed is 60 GBps. ORLY?

@alexanderdaum8053 4 ай бұрын

@@stariyczedun May be realistic if the file is cached in RAM already. 60GBps is a bit more than 2 channels of DDR4 3200 peak throughput, so could be achieved on a 4 channel machine.

@stariyczedun 4 ай бұрын

@@alexanderdaum8053 my idea of realism is that the answer is already cached as well 😀

@sarjannarwan6896 4 ай бұрын

Anyone submitting answers to this should use a non commercial license.

@909crime 4 ай бұрын

SO TRUE

@daumienebi 4 ай бұрын

can you please explain more ?

@cryptonative 4 ай бұрын

It’s not about the code but the idea behind

@j_stach 4 ай бұрын

@@daumienebi Because then if a company wants to use the algorithm, they have to make their code open source

@TheNewton 4 ай бұрын

@@daumienebi there's a chance projects like this are astroturfing psyop brain rape campaigns by businesses to crowd source their problems exploiting peoples passion and labor to improve that business proprietary closed-source software.

@andzagorulko 4 ай бұрын

The fact that the challenge requires a specific language is what makes it the most suspicious

@uGetkilled 4 ай бұрын

I think the main point of doing this challenge in Java, is to showcase the performance benefits of using virtual threads over normal java concurrent threads, which were included in the latest major release.

@ryanshea5221 4 ай бұрын

If you wanna showcase Java's performance you gotta compare it with another language.

@aiyazmostofa1501 4 ай бұрын

@@ryanshea5221 No one be disputing Java is slower than other languages. But is it too slow? Compared to standard Java how can you optimize it? Though adding other languages could be useful, though supporting those runtimes is more work.

@TheNewton 4 ай бұрын

@@ryanshea5221 java versions are another language.

@chigozie123 4 ай бұрын

@aiyazmostofa1501 uhh, I hate to be the bearer of disappointing news, but I have witnessed Java beating C in a very predictable manner. I took a course where we had the task of building Conway's game of life in both C and Java. To everyone's shock, all of our C programs were either tied in terms of speed with Java, or were beat by Java. At the end of the day, it all came down to AOT vs JIT optimizations. Turns out if you give Java enough time, it starts to do some really smart things that C can't.

@lyndog 4 ай бұрын

I used to be a Big Data Dev using Java. I'd probably load a page worth of results at a time (whatever page size that fits neatly into JVM memory) and process it in chunks simultaneously using thread pools and queues. Essentially doing a map-reduce style solution.

@karpfenboy 4 ай бұрын

a nice way to get specific work done for free lol

@HenrikoAlberton 4 ай бұрын

Brazil mentioned, today is a good day!

@FilippsBlog 4 ай бұрын

Let's go Prime, show us some Enterprise Java Coding.

@SpikeTaunt 4 ай бұрын

The current second place is the vice president of software development at oracle lol

@Jake9066 4 ай бұрын

"You just need four numbers: min, max, sum, count" And now I feel validated, I had the same thought as a senior developer at Netflix

@fabianschwarzfritz 4 ай бұрын

Hey @ThePrimeagen I know you're not the java lover, but this sounds like fun, no? Are you planning on doing this on stream :)?

@rdean400 4 ай бұрын

The interesting thing is that there's a lower-bound on the performance you can get with the code still looking friendly. Once you go beyond that, the techniques aren't quite as fun to read, but they're fun to write. :)

@AlecAkin 4 ай бұрын

I believe Tails runs purely from RAM disk for security/privacy reasons (no forensic artifacts left on disk/persistent storage)

@LtdJorge 4 ай бұрын

Yep

@tordjarv3802 4 ай бұрын

The big challenge is how to read the file as efficiently as possible. If you have everything in RAM the problem is embarrassingly parallelisable. Each thread can keep there own values for min, max, sum and count per station and just combine everything at the end with a single reduce. You would need some kind of hash map to look up the stations and ideally it would be a perfect hash map (no need for collision handling but that might be premature optimization).

@occamsrazor1285 4 ай бұрын

8:34 Military operating systems (like the OS the runs the Abrams MBT) work exactly like that. Have since the late 70s

@utenatenjou2139 4 ай бұрын

awk, cut, sort, immediately come to mind when seeing these.

@imulion668 4 ай бұрын

Would love to see you do this Prime :D

@H4KnSL4K 4 ай бұрын

If the VM has 32GB, then it may be cached in memory from the first run (which is discarded), so after that it will go fast

@stariyczedun 4 ай бұрын

fastest entries do it in one SSD read basically so you won't beat them with this

@jonathanjacobson7012 4 ай бұрын

Using OCaml (with Base and parallel domains), I managed to reach 55 seconds on an old 4-core i7. I'm new to OCaml so it could probably be optimized more. My plan is to try Owl on it to see what it can do.

@stariyczedun 4 ай бұрын

The problem is not CPU but reading the file in a way that maximises SSD read throughput. I did several attempts in Java and the biggest jump was from reading the file using multiple mmaped regions in parallel (like 16 or even 32). With that you cut time from around 2 minutes to 15 seconds. The fastest version then was running in 10 seconds on my laptop so I lost interest, everything else you can do (vector math, improve JVM startup time) won't yield much improvement.

@jonathanjacobson7012 4 ай бұрын

@@stariyczedun Sounds like you know what you're talking about. I have no idea about SSD read optimization so I read the file as a whole into a memory buffer (didn't take more than 20 seconds) and then CPU was actually the only place left to optimize. I suppose that if you parallelize file reads then you can combine parse on the fly and reach much better results. Thanks for the feedback.

@stariyczedun 4 ай бұрын

@@jonathanjacobson7012 probably my wording was a bit confusing. Yes, you basically need to parallelize reads and parsing \ calculations. I did it with mmap, probably there are other ways to do it. I think it is doable in ocaml as well. Basically, split file into chunks, run a bunch of threads to process those chunks. mmap just provides a nice way to read regions of file completely independently like memory, with OS handling the real SSD reading underneath.

@MenkoDany 4 ай бұрын

can't wait for someone to write this in C and it's literally just doing 12gb at the speed of dram

@fuzzy-02 4 ай бұрын

Drammn man!

@vitalyl1327 4 ай бұрын

Is not it a point of this whole challenge? To optimise the memory access pattern (it need to write, not just read, and the output may not fit L3)

@ryanshea5221 4 ай бұрын

I'd like to see it go toe to toe with C# fully Jitted

@renderwood 4 ай бұрын

Nothing prevents you from forking that contest for all languages. Just design cool t-shirt and launch it. Maybe re-use the hosting, instance-type and OS-distribution constraints so that the Java results are comparable with this one. I recall competing in shortest DOS-program to print "Imphobia!" to console, it was fun at least.

@stevenhe3462 4 ай бұрын

Can you include all the stuff to enable JNI and a compiler in one file and embed some native code?

@KangoV 3 ай бұрын

It's now down to 1.5 seconds. Holy crap!!!!! But amazingly the challenge has now ended so Gunnar Morling ran the top ten on all 32 cores/64 threads and got 0.3 seconds. Bloody hell. Java scales!!!

@VLUURIE 4 ай бұрын

I thought you solve the challenge, but you only read the challenge 😂 he got me

@astronemir 4 ай бұрын

Compress the file in blocks then you can read it faster if you decompress in memory and read parallel blocks. After like 12-20 threads it should be faster than the overhead

@eddyrose3254 4 ай бұрын

Mr primeagen i love that you can automate the upload to your youtube, but can you toss a link to the content you react to. Thx -fellow SD'an

@stillWonderingWhyMe 4 ай бұрын

What are you doing in Brazil and where are you going? Gonna be in BH for one month starting on February 8th

@chinoto1 4 ай бұрын

0:30 How about running Rust compiled to WASM in a JVM WASM runtime? 🤣 Unfortunately, there doesn't seem to be a path directly from Rust to JVM bytecode 😢

@sutirk 4 ай бұрын

I wanna see someone doing this in Excel

@thom1218 4 ай бұрын

TLDR; find out how much you can avoid using java while still technically using java 🤣🤣

@rene_epc 4 ай бұрын

Our idle is coming, brothers! 🟢🟡🔵

@demolazer 4 ай бұрын

Would be fun try this on a bunch of languages to compare

@richcole157 4 ай бұрын

Dude it is going to be io bound for sure

@ShankingDisaster 4 ай бұрын

after watching a course video on recursion in java, the language just looks like C + Py structurally. EASILY readible showing synergy of syntax with other languages. Princeton has an entire CS database for programming in java... I'd say it's gonna be on the list of popular languages soonish

@samho321 4 ай бұрын

14s is the best when Prime is watching this repo, now the best is 6.159s, crazy

@thedeemon 4 ай бұрын

still too slow compared to non-JVM versions

@mortred4144 4 ай бұрын

@@thedeemon 2.575s is the best one right now.

@nsuid7499 4 ай бұрын

3:46 what do you mean by "sorting is a constant time operation in this case"? How can it be constant time?

@frogery 4 ай бұрын

reading and calculating the values in this case takes much much longer than it would take to sort the results, and the number of values being sorted isn't based on the input to the program since you could have any number of unique weather stations (to a max of 10 000) in the input file, so it's treated as a constant time operation that happens at the end.

@pushpindersingh785 4 ай бұрын

This is really interesting, anyone know of any website or collection of similar more realistic programming performance challenges?

@Blubb3rbub 4 ай бұрын

Project Euler, maybe.

@EikeSchwass 4 ай бұрын

@@Blubb3rbub they just focus on correctness not performance

@Blubb3rbub 4 ай бұрын

@@EikeSchwass the idea is that you won't get a correct solution in time without being smart about it.

@edhahaz 4 ай бұрын

Who will eat all this exposure generated by the very smart guys racing to the top ?

@atcen 4 ай бұрын

Everything in RAM is just the past of computing before storage was invented

@u9vata 4 ай бұрын

I really wish this would be open to other languages - but I guess in a way it is... I just create the data, then write teh cod3 and that's it. It took me some time to realize the "mean" is just fancy name of "average" so its a bit bad task to illustrate java performance because no memory is involved at all so they can be very fast unless implemented by idiocy. Managed languages can easily number-crunch nearly as fast as native code with very thin margins - but when they fall apart is when you do memory too......

@PixelThorn 4 ай бұрын

Nothings stopping you from doing it in any language, you just need to generate the data with Java, after that, you can use whatever. If they want results back in Java, well, tough break

@fizzlefresh14 4 ай бұрын

sounds very fun! It'd be cool if similar challenges like this existed in other languages like C# or Python. Anybody know of any, or where to find them?

@CTimmerman 4 ай бұрын

Vercel accepts contributions to its Programming Language and compiler Benchmarks.

@zsomborgyenge4359 4 ай бұрын

someone should creat an x86_64 assembly version to troll java devs

@PythonPlusPlus 4 ай бұрын

Why would that troll java devs?

@zsomborgyenge4359 4 ай бұрын

@PythonPlusPlus java runs in vm and it has some overhead. Native code can be faster even if it does the exact same thing. The competition is about speed so assembly can beat them

@PythonPlusPlus 4 ай бұрын

@@zsomborgyenge4359 The competition requires you to use Java. So you won’t be able to enter the competition.

@EikeSchwass 4 ай бұрын

@@zsomborgyenge4359 good luck beating AOT and JIT optimizations when doing assembly by hand

@HrHaakon 4 ай бұрын

The fastest solution is pretty cool actually.

@ragsdale9 4 ай бұрын

Tinycore Linux is run in ram I believe. Initially stored in a flashdrive or HDD then loaded entirely into ram, is meant to be as minimal as possible.

@marceloferreira8068 4 ай бұрын

there's also Alpine Linux

@darkdudironaji 4 ай бұрын

Is this relevant to the video? Did I miss something?

@LtdJorge 4 ай бұрын

@@marceloferreira8068 although Alpine doesn't live in RAM by default

@LtdJorge 4 ай бұрын

@@darkdudironaji having the entire disk just be RAM, you already have the file in RAM, as well as the executable, etc

@darkdudironaji 4 ай бұрын

@@LtdJorge Right, but I'm sure this is standardized. So the code is submitted and then run on the same machine. Otherwise a researcher could just use a supercomputer.

@user-qp5rh9iv7n 4 ай бұрын

in fact tere are even "in ram" linux distros like tiny core

@Bianchi77 4 ай бұрын

Nice video, thanks :)

@baguettedad 4 ай бұрын

Brazil Mentioned mentioned

@1000marcelo1000 4 ай бұрын

I'll be waiting for you in May in Brazil :3

@thomas-hall 4 ай бұрын

"same dataset for all submissions" just screams "reverse engineer the dataset to optimize this specific answer"

@KangoV 4 ай бұрын

Wow. 6.2 seconds!!! Off-heap memory! No GC at all. Java is bloody fast. London stock exchange runs Java and processes 6,000,000 transactions per second on a single thread (LMAX).

@SimonBuchanNz 4 ай бұрын

On a typical CPU this is a little under 1k cycles per transaction. Considering all the interesting stuff like DB and network would be amortized across chunks that sounds like way more than enough for basic book keeping. Don't get me wrong; you need to be pretty clever and careful in how you do your chunking and scheduling of said interesting stuff so you're not stalling out, but otherwise CPU-wise this isn't a huge workload and could probably run in any language. Maybe not python 😋

@besknighter 4 ай бұрын

Primgean is coming to Brazil in May?? LESGOOOO

@PaulJohnsonM 4 ай бұрын

"Burnt-in RAM." Isn't that called ROM? I mean, you could alter the non-burnt bits I guess, but supposing you don't do that, that would just be ROM.

@edantas 4 ай бұрын

My boy will come to Brazil lets goooooo

@stephenthumb2912 3 ай бұрын

only an application programmer would think of something like this. the thought would never cross a data guy's mind, his first thought would and only be how to get this into a database and THEN how do i deal with 900 billion row tables. And that first thought would be start with a bulk insert, whatever i do avoid logging.

@dalizdr 4 ай бұрын

This problem look like one example from the book about using map-reduce with hadoop 😊

@thedeemon 4 ай бұрын

I won't be surprised if hadoop was super slow here compared to a straightforward program that does one thing well. (inspired by "Scalability! But at what COST?" article)

@datainsight1724 4 ай бұрын

The fastest you can get this is about 2-3 seconds so there is still a lot of headroom 😄

@mustakrakish 4 ай бұрын

They now have it for Rust, Go, C++, and others

@AntranigVartanian 4 ай бұрын

shshshssshh :DDDD you will LOVE the pf firewall. the name of it, at least :D

@baguettedad 4 ай бұрын

Gente pera aí, ele ta vindo pro Brasil em maio???

@pedroluiz9670 4 ай бұрын

sim

@eliasgill2453 4 ай бұрын

Momento de viajar a Brasil

@CodingThingsIRL 4 ай бұрын

One BILLION dollars 😮

@MrAbrazildo 4 ай бұрын

0:42, if I would take care of this, I would start by rewriting this file to something much smaller. After all, 12GB is too much. Let's say names have average of 8 letters and numbers can go up to 99.9. So they are (8 + 1 (;) + 2 (2 numbers) + 1 (.) + 1 (1 number) + 1(new line char) )*8 = 14*8 = 112 bytes per line. So, writing in a binary file, 4 bits for the last fraction, 7 for the number and ~11 for a number simbolizing the name, which should be searched later in a separated table. So 4 + 7 + 11 = 22 bits, 3 bytes. If 3 is ~2,7% of 112 bytes, it means that those 12 GB would be reduced to 12x0.027 ~= 324 MB. This would make any kind of search a lot faster.

@TheTim466 4 ай бұрын

A single scan is enough so I don't really see the advantage in what you describe.

@litfill54 4 ай бұрын

thats so much reduction

@nagoshi01 4 ай бұрын

OK, what if you encounter a station name that's 100 chars long? Read the rules

@ar1i_k 4 ай бұрын

⁠⁠⁠@@nagoshi01It took me like 3 re-reads to realize that he does not proposes to assume max character limit on name and write it like that in binary. His solution is to store all the names in the separate file. And instead of the actual name, use line index of the name from that file. The real problem is that he assumes that 11 bits (0 to 2048) should be enough to cover all the names, while rules clearly state that there can be up to 10,000 unique names.

@nagoshi01 4 ай бұрын

@ar1i_k Ah gotcha

@diegolikescode 4 ай бұрын

HEY HEY HEY, if you are coming to Brasil you gotta notify before hand so we can prepare to meet you 🙏

@JeremyAndersonBoise 4 ай бұрын

Prime wouldn’t hire Kyle Kingsbury 😂 I’m dead

@KvapuJanjalia 4 ай бұрын

I can't believe .NET 1BRC is faster than Java.

@thuthupsy2 4 ай бұрын

I just realized how much similar your voice is to Rick Sanchez from Rick & Morty.

@esbrasill 4 ай бұрын

Gonna be where in Brasil?

@SanixDarker 4 ай бұрын

read byte per bytes the file, start processing as soon as possible while still reading and split the process on all available cores, then merge it. In C, it took around `89ms` .

@titbarros 4 ай бұрын

Make sure to come to olegário Maciel. Raval Rio bar. You'll be king there. Let Em know Tiago is your holmie

@SpikeTaunt 4 ай бұрын

People are already on 6 seconds, that's crazy

@fewunderstandthis7355 4 ай бұрын

Tails OS runs in memory… on a USB stick!

@CorvinhoDoMal 4 ай бұрын

Are you really coming to Brazil in May?

@ThePrimeTimeagen 4 ай бұрын

yes

@joelgiovinazzo5058 4 ай бұрын

At my company we pxe boot Ubuntu and the entire filesystem runs on ram

@chebrubin 4 ай бұрын

CSV and Java 11 will get this done. Java NIO buffer reader and in memory db.

@thedeemon 4 ай бұрын

why would you need a db instead of a single for loop basically

@chebrubin 4 ай бұрын

@thedeemon 16 million lines needs to be chunk read by the Java NIO buffer streamer like 32k at a time and then written into a in memory DB like hazlecast or redis or hsqldb if a single node? If we want to run clusters of JVM than we need redis and a the file IO needs to know which means ate the 32k lines from the file.

@EikeSchwass 4 ай бұрын

@@chebrubin no external libs

@thedeemon 4 ай бұрын

@@chebrubin We don't need no clusters, the data fits in RAM of one machine, and people using C solve it in less than a second (if a file was read once before and cached by the OS). No DB will accept a billion rows this fast.

@PITERPENN 3 ай бұрын

there are results now with execution times of less than 2 seconds. ouch

@nchomey 4 ай бұрын

A better version of this challenge (not a company sneakily getting people to solve their problem) comes from this video for a sorting challenge where people made it 400 million times faster than the initial Python code. kzfaq.info/get/bejne/mZljdL1z07ism6c.htmlsi=ASPe_4enPDK6dUwg

@fuzzy-02 4 ай бұрын

I wonder if they'll still post my time if I run it by hand kekw.

@pandabearguy1 4 ай бұрын

Would be cooler to see who could do it fastest with pen and paper (real skill)

@georgehelyar 4 ай бұрын

I haven't read the rules in detail so this is probably disallowed but since the slowest run is discarded, just store the results and then if the results exist then print them out instantly

@peachezprogramming 4 ай бұрын

companies are getting smarter

@dioneto6855 4 ай бұрын

Hell yeah! Brasil mencionado.

@pietro4507 4 ай бұрын

BRAZIL MENTIONED

@mathiflip 20 күн бұрын

are you gonna try it in Rusty?

@browntigerus 4 ай бұрын

Same, some company needs this so they created this "FAKE CHALLENGE" in a single language to avoid using a database. Not participating ...

@TheTigerus 4 ай бұрын

can you use GPU for calculations?

@stariyczedun 4 ай бұрын

yes

@TheTigerus 4 ай бұрын

@@stariyczedun so wouldn't it be a good idea to utilize GPU in this case? As GPU is basically CPU with bazzilion cores, but each can perform quite easy operations.

@thedeemon 4 ай бұрын

@@TheTigerus nope, there are very few calculations in this task, it's basiclly IO bound

@stariyczedun 4 ай бұрын

@@thedeemon well you might get additional second or two off, it could give an edge 😀

@thedeemon 4 ай бұрын

@@stariyczedun unlikely, GPU are not that good working with strings and hash tables. And loading data there will take more time than finishing the task on CPU. Look at other comments here, some people in C do the thing in less than 1 second.

@nerdycatgamer 4 ай бұрын

THEY SENT PRIME TO BRASIL

@ehsansasanian9653 2 күн бұрын

dude if you think 14 sec is fast what do you think of 1.5 sec which is the winner?

@stavinke 4 ай бұрын

It's 3 seconds now...

@JuusoAlasuutari 4 ай бұрын

How is 1 billion data points a large amount? Oh Java, you cray

@CamaradaArdi 4 ай бұрын

Write it as a Rust library and just call it from Java

@oleksiistri8429 4 ай бұрын

It's not Brazil mention, it is 1 Bircoin

@atcen 4 ай бұрын

They are right now at 6s 😮

@thedeemon 4 ай бұрын

just 2x slower than C# heh

@mortred4144 4 ай бұрын

@@thedeemon best one is at 2.5s atm

@nathanrapport8661 4 ай бұрын

Write a Java program with one line of code calling an executable compiled from Rust.

@felixnotthecat4249 4 ай бұрын

please, don't tell me you're going to Rio de Janeiro.

@blinking_dodo 4 ай бұрын

1. Use Linux and copy the file to a mounted ram-disk. 2. Have one thread read the file as fast as possible to byte array's. 3 Have multiple threads start processing at different starting points. (use the text bytes for (partial?) 256-tree lookup.) 4. Aggregate and print out the results. That would be my first attempt.

@PythonPlusPlus 4 ай бұрын

copying the file to a mounted ram-disk is pointless since you only need to read the contents once. You may as well load the entire file into a buffer instead, and have the data immediately accessible to the program.

@LtdJorge 4 ай бұрын

@@PythonPlusPlusbut you are copying it manually before executing the program. Cutting the time of moving it to memory out of the execution.

@blinking_dodo 4 ай бұрын

@@PythonPlusPlus Accessing a local byte array is a LOT faster than accessing an abstracted away mapped file that is only readable trough syscalls. It's better to just copy huge chunks to local arrays than to read each byte on its own. The syscall overhead of reading single bytes would be huge.

@daasdingo 4 ай бұрын

mmap

@PythonPlusPlus 4 ай бұрын

@@blinking_dodo That was my point…..