New Go Billion Row Challenge w/ Great Optimizations | Prime Reacts

  Рет қаралды 102,938

ThePrimeTime

ThePrimeTime

3 ай бұрын

Recorded live on twitch, GET IN
/ theprimeagen
Become a backend engineer. Its my favorite site
boot.dev/?promo=PRIMEYT
This is also the best way to support me is to support yourself becoming a better backend engineer.
Article link: r2p.dev/b/2024-03-18-1brc-go/...
By: Renato Pereira
MY MAIN YT CHANNEL: Has well edited engineering videos
/ theprimeagen
Discord
/ discord
Have something for me to read or react to?: / theprimeagenreact
Kinesis Advantage 360: bit.ly/Prime-Kinesis
Hey I am sponsored by Turso, an edge database. I think they are pretty neet. Give them a try for free and if you want you can get a decent amount off (the free tier is the best (better than planetscale or any other))
turso.tech/deeznuts

Пікірлер: 208
@ivanovcharov7534
@ivanovcharov7534 3 ай бұрын
OMG ITS MY FAVOURITE PROFESSIONAL YAPPER!
@yaaaayeet745
@yaaaayeet745 3 ай бұрын
5 DOLLARS A MONTH 🗣🗣🗣🗣🗣✋✋✋✋✋
@apexdude105
@apexdude105 3 ай бұрын
"professional yapper" what a good job description for a streamer lmao
@charlesyoung601
@charlesyoung601 3 ай бұрын
nl clears
@oat1000
@oat1000 3 ай бұрын
nl my goat ​@@charlesyoung601
@jostasizzi818
@jostasizzi818 3 ай бұрын
Why do I feel this is every so called tech KZfaqr right now
@neruneri
@neruneri 3 ай бұрын
Asking Flip to take something out seems like the most reliable way to ensure that it absolutely does not get taken out.
@istasi5201
@istasi5201 3 ай бұрын
is flip even real?
@FakeDumbDummy
@FakeDumbDummy 3 ай бұрын
Yes ​@@istasi5201
@R4ngeR4pidz
@R4ngeR4pidz 3 ай бұрын
Narrator: Flip did, in fact, not take that out (16:00)
@teejaded
@teejaded 3 ай бұрын
Flip. Take this anti-flip propaganda out.
@flipmediaprod
@flipmediaprod 3 ай бұрын
I stand against the establishment
@Kannatron
@Kannatron 3 ай бұрын
@@flipmediaprod truly and upstanding and forward thinking editor. You kept it in for the people, 👏🤯🤯🤯
@GermanClaus
@GermanClaus 3 ай бұрын
He sounds like he is begging :D
@Sw3d15h_F1s4
@Sw3d15h_F1s4 3 ай бұрын
the JDSL implementation would be 10x faster. Tom's a genius!
@jerichaux9219
@jerichaux9219 3 ай бұрын
JDSL would have melted the CPU from how fast it would be parsing those rows.
@strangnet
@strangnet 3 ай бұрын
Wow: a 4.7HGz with 6000mhz memory. Those millihertz come in handy with the HenryGigaz processor...
@MHarris021
@MHarris021 3 ай бұрын
Tip for remembering stalagmites and stalactites. "Stalagmites have a g for ground and stalactites have a c for ceiling", it's how I remember which is which. It was a tip in a Xanth novel by Piers Anthony. I think it was "Man from Mundania", but I'm not sure because I haven't read them in 20+ years. Gosh, that makes me feel old. :)
@retropaganda8442
@retropaganda8442 3 ай бұрын
Ahaha, the true mnemonic is actually just the etymology of the word. I don't know if it's Latin or Greek, but for example, in french it's m for monte (raise) and t for tombe (fall). Simple.
@collinstasiak4994
@collinstasiak4994 3 ай бұрын
Stalagmite sounds like dynamite and you don't wan to put that on ceiling is how Ive always remembered it
@Eutropios
@Eutropios 3 ай бұрын
Stalactites stick tight to the ceiling. Stalagmites might grow upwards
@metropolis10
@metropolis10 3 ай бұрын
Primeagens reactions in this video "wow that's a lot slower than I would have thought... well I GUESS it is a BILLION items" x1 Billion
@jackevansevo
@jackevansevo 3 ай бұрын
I love these posts, there's a lot of tidbits of information to learn.
@MrDadidou
@MrDadidou 3 ай бұрын
French gang: Stalag-mite (M like "monter" in french, to go UP) Stalag-tite ( T like "tomber, to fall)
@_kostant
@_kostant 3 ай бұрын
Always remembered it from the C in stalactite being “ceiling” lol.
@OnStageLighting
@OnStageLighting 3 ай бұрын
'might go up, tights come down.'
@microcolonel
@microcolonel 3 ай бұрын
​@@OnStageLightinggiggity
@itsthesteve
@itsthesteve 3 ай бұрын
Stalag (ground), Stalac (ceiling)
@hierax49
@hierax49 3 ай бұрын
the author has a brazilian name. brazil mentioned
@rawallon
@rawallon 3 ай бұрын
Dev do Gamers club do fallenzão (2:06)
@Thoer
@Thoer 3 ай бұрын
let's go!!!
@user-zg2bx4oz2p
@user-zg2bx4oz2p 3 ай бұрын
It is also a Portuguese name
@microcolonel
@microcolonel 3 ай бұрын
Nobody lives in Portugal 😂​@@user-zg2bx4oz2p
@michealkinney6205
@michealkinney6205 3 ай бұрын
"Managers be like push it to prod! We're done... Good enough!" @ 20:16. Lol, like every non-technical manager ever.
@Olodus
@Olodus 3 ай бұрын
Dammit, now I feel like I will have to do this in Zig or something... But great article. Really shows the experimentation and learning process.
@Thorarin
@Thorarin Ай бұрын
FYI: Buffer size of 1024 is terrible, because most modern disks use 4kB sectors nowadays. So some multiple of 4kB is immediately better.
@rapzid3536
@rapzid3536 3 ай бұрын
mmap Split the memory space into the number of Cores Hand out pointers start/end to threads Walk all but the first pointer start forward until after the next new line or EOF. Start ripping from there. Profit.
@sanderbos4243
@sanderbos4243 3 ай бұрын
It drives me bonkers how they used 10 instead of ' ', and even went so far as to describe the magic integers at 28:21 with comments like "if b == 45 { // 45 == '-' signal"
@andyvisser
@andyvisser 3 ай бұрын
My guess on the read buffer and diminishing returns: I bet you get max performance when the buffer size aligns with the underlying hardware's size. Like it's best when you read a sector at a time (or however SSDs are addressed/broken down in firmware).
@TehKarmalizer
@TehKarmalizer 3 ай бұрын
Or file system block size. Typically reading in multiples of the block size is most efficient.
@i_sometimes_leave_comments
@i_sometimes_leave_comments 3 ай бұрын
4:35 Assuming go's `map` is a self-growing (via reallocation) array (like C++ `vector` or C# `List`), as the `map` grows, you'd have to mem copy the whole underlying array, and a bunch of pointers would be way cheaper than a `struct`
@anon1963
@anon1963 2 ай бұрын
you can do vec.reserve(n) in c++. eliminates need for expensive reallocation
@evergreen-
@evergreen- 3 ай бұрын
This video gives me huge flashbacks
@PhilipAlexanderHassialis
@PhilipAlexanderHassialis 3 ай бұрын
I like how its from 95s to 1.96s whilst inside the article a sub-second result is mentioned.
@hosseines276
@hosseines276 3 ай бұрын
whoa! really enjoyed!
@danielmccann2979
@danielmccann2979 3 ай бұрын
For one second I read that as milli hz of ram and was like why is you ram going only 6 hz, are you manually clocking that thing
@SimonBuchanNz
@SimonBuchanNz 3 ай бұрын
I did some basic aggregation with node on a 2 GB ini file: from memory with a bunch of work i got it down from 40s done somewhat naturally to about 7s done by a crazy person. The dumb 10 line Rust code took 3s or something.
@kodekata
@kodekata Ай бұрын
A Goroutine is Go's syntax for Tony Hoare's Concurrent Sequential Processes (CSP, not like the browser's CSP though). Fun fact: the creator of Go had made several previous languages, all with CSP baked in. In Clojure[script], the simple syntax for CSP was enabled via a library. CSP has been implemented in JS via generators, but there are implementations with more usage (eg. for Clojure).
@hinzster
@hinzster 3 ай бұрын
Oh damn, for-loops are now considered boomer loops? What about while(true)/break loops? Are those dinosaur loops?
@hinzster
@hinzster 3 ай бұрын
Also, back when I was doing that obscure shift organizer program for hospitals, I used my own fixed point package to optimize stuff - everything was one single digit of precision anyway, so I just worked with ints determining 10ths of hours (another "problem"). Worked well, fast, and didn't use as much space as those pesky floats. I did this before the FP coprocessor was included in intel processors (ie. before the 486. My actual development machine was an original IBM PC XT, running an 8088 at 4.77MHz! I needed all the speed I could get).
@weakspirit_
@weakspirit_ 3 ай бұрын
nah, the dinosaur loops are the asm branch loops 🦖
@FakeDumbDummy
@FakeDumbDummy 3 ай бұрын
Well, go don't have while loops, so yes dinosaur loop for me
@SandraWantsCoke
@SandraWantsCoke 2 ай бұрын
Those are biblical times loops
@valhalla_dev
@valhalla_dev 2 ай бұрын
"I have very little experience in these kinds of investigations" Me: Oh, word, he and I will be talking on the same level ... Me: Oh, shit, I understand none of this
@fuzzy-02
@fuzzy-02 3 ай бұрын
Renato Pereira alone sounds like a cool secret agent driving a very fast classical car
@khuntasaurus88
@khuntasaurus88 2 ай бұрын
Its a common brazilian name
@StrengthOfADragon13
@StrengthOfADragon13 3 ай бұрын
Can't wait for the "what is your 1 billion row challenge time" question in interviews. (Actually though, taking a legit stab at the challenge for myself sounds super fun and I really wanna see if work will greenlight letting me work on it as part of my training hours)
@TurtleKwitty
@TurtleKwitty 3 ай бұрын
The mighty stalags rise, while the other stalags hold tight is my way of remembering which is which hahah
@retropaganda8442
@retropaganda8442 3 ай бұрын
Etymology: m for monte, t for tombe
@mikejohnstonbob935
@mikejohnstonbob935 3 ай бұрын
Devin's out there taking notes. This whole article is honestly like an AI overtraining on a specific dataset. Its language capabilities even degrades as it reaches the its max context window
@retropaganda8442
@retropaganda8442 3 ай бұрын
I just clicked on the first search engine result for the one billion rows challenge in c language and the result of the guy beats the "official" java winner. Not surprised.
@morosis82
@morosis82 Ай бұрын
Not that surprising, the first result is likely to be the best linked (highest ranked) when everyone is talking about fastest implementation in language X.
@bluecup25
@bluecup25 3 ай бұрын
Prime, do it. Just do it.
@absurd0000
@absurd0000 3 ай бұрын
Flip, more like Slip, cuz he be slipppppin
@thatmg
@thatmg 3 ай бұрын
PORTO MENTIONED!
@parikshitpatil1421
@parikshitpatil1421 3 ай бұрын
I guess best java solution used mmap.
@9remi
@9remi 3 ай бұрын
16:00 flip did NOT take that out
@MikePaixao
@MikePaixao 3 ай бұрын
I remember having to parse 600TB databases in the gamedev industry, I ended up using python and the windows copy buffer to just snapshop the file into memory
@dv_xl
@dv_xl 3 ай бұрын
Interesting , have a few questions. Obviously it can't loat 600TB into memory at once, did you chunk your reads or were the underlying DB files split up naturally? Were you using a network file system? Did you run multiple processes and map/reduce or just a single process? I'm curious how long it took in either case
@MikePaixao
@MikePaixao 3 ай бұрын
@dv_xl the first layer was using perforce, so any previous work or code could compare against cached version of all unchanged files locally synced Next you need to break up Parallel loops based on file types, ascii files are super easy to write regex logic (think file mirroring) I would quickly build a list of all file dependencies (if I was parsing a game map, I listed all the models, if it was a 3d model, it connected what maps and textures used it etc etc... Now for the copy trick, depending on file size, when having to parse through larger 1gb+ files you can choose to either copy an entire folder or individual files, and binary format you need to do the painful thing of writing a custom binary parser for the now copied into memory data I remember back on wolfenstein a couple of times having to checkout the entire repo because German lawyers were like "nein! You cannot have any file names with verboten naming on disk" and when you need to edit file names across an entire project that is weeks away from gold master.. not a lot of wiggle room :P
@MikePaixao
@MikePaixao 3 ай бұрын
​@@dv_xl So the data was all stored in perforce, so I would store a snapshop with a perforce timestamp, so I could choose a chached or fresh mapping depending on folder/file size, sometimes you could copy entire folders to parse through larger files... it really depended on file types or single files at a time with custom binary interpreter. so you could skip entire sections of files and pull out relevant info (I was tracking all assets, where they showed up in engine or in a map and then all the related textures, models, audio etc..) It was a reflection system across data formats :P All done in parallel, and a weird reason to do batches of folders and not file by file is the limited number of threads python would spin up before hitting some per machine arbitrary number of threads windows can keep track of :P (also, early exist everywhere, I don't need to parse a 3D models vertices, or the animation sequence in a skeleton!) At some point I was checking out the entire project because german lawyers were like "Nein! Verboten! you cannot have nazi named file folders on the shipped disc" "but it's wolfenstein?" -> glad I added the "find and replace" option so I could do mass edits while it was parsing through :D timing I had it under around a few seconds, under 1s if the perforce cache existed (db was stored as sql file with no read/write locks in perforce)
@KaydotOrigin
@KaydotOrigin 3 ай бұрын
Would be awesome to see you do it in ts/js
@rezyadlf
@rezyadlf 3 ай бұрын
2 business days got me)))
@GermanClaus
@GermanClaus 3 ай бұрын
KOTLIN mentioned!!!
@willembeltman
@willembeltman 3 ай бұрын
8:00 reason is the buffersize of your hdd/ssd.
@weakspirit_
@weakspirit_ 3 ай бұрын
i'm calling it, multithread/multiprocess overhead is going to show that his single process/thread solution is actually faster
@idkwhattonamethisshti
@idkwhattonamethisshti 3 ай бұрын
not for i/o operation with 1billion rows
@anon1963
@anon1963 2 ай бұрын
no lol
@MrWalrus3451
@MrWalrus3451 3 ай бұрын
Flip ain't taking it out brother.
@ReedoTV
@ReedoTV Ай бұрын
They should have used their "4.7HGz" PC to run a spell checker
@caedenw
@caedenw 3 ай бұрын
I can’t believe I have to point this out but his SSD can’t do 13GBps and so this is all coming from his page cache in RAM. Don’t expect anything close to these results if you flush the cache. In light of that, he should be seeing a much better score if implemented correctly since he has so many threads.
@lskywalker5
@lskywalker5 3 ай бұрын
GOD DAMN IT FLIP
@rogerdinhelm4671
@rogerdinhelm4671 3 ай бұрын
Current top Java implementation reaches 300ms, but measurements are done on reference hardware (32 cores / 64 threads), and thus might be different to whereever the Go guy was running it at.
@thekwoka4707
@thekwoka4707 3 ай бұрын
Probably could do pretty fast with Bun. Bun.file has some good ability to read file partials, so you could see how big the file is, spawn a ton of threads and handle only the parts for each.... JavaScript does also have cool things like SharedArrayBuffers that could enable some more low level style memory control...
@marcomassa84
@marcomassa84 2 ай бұрын
I got the 1BRC down to 5.5 sec with nodejs. Bun has a bug with highwatermark option that make it less performant than node (at least in my test)
@anon1963
@anon1963 2 ай бұрын
remember about Amdahl's law
@JackDespero
@JackDespero 3 ай бұрын
I am sorry, but you are wrong. Boomer loops are GOTO and CONTINUE loops. The simulation code that we use at work was written in modern FORTRAN (FORTRAN 77, not 65) and is full of GOTO 1000 Do stuff 1000 CONTINUE
@Sw3d15h_F1s4
@Sw3d15h_F1s4 3 ай бұрын
someone should do the 1 billion row challenge using vim
@SimonBuchanNz
@SimonBuchanNz 3 ай бұрын
"mutex is a spin lock" technically mutex is just the semantics, not an implementation, and there's a few ways to do it, with different trade-offs. They generally *start* with a spin lock, but that's just an optimization assuming the lock time is short. They then need a way to put the aquiring thread to sleep, and there's a bunch of ways to implement that. You can do it in user space with just thread sleep and wake functions, which can be good for "fair" locks, but you can also use events or explicit kernel mutexes, which might be better for thread residency.
@Kane0123
@Kane0123 3 ай бұрын
I’m going to give you a like based purely on the amount of text. I’m happy for you though, or sorry that happened.
@rawallon
@rawallon 3 ай бұрын
technically, anything is just the semantics
@jhk940
@jhk940 3 ай бұрын
I must have missed something. The SSD (Kingston SSD SV300S37A/120G) has a maximum read rate of 450MB/s, so reading the 13GB should take 28.88 seconds minimum. wat. Can someone explain?
@jhk940
@jhk940 3 ай бұрын
Well, I guess the complete 13GB file is cached in RAM by Windows.
@TurtleKwitty
@TurtleKwitty 3 ай бұрын
@@jhk940 Yup every os keeps hot files in ram; the java one actually had a final implementation with a ramdisk instead so the ssd overhead didnt matter
@truehighs7845
@truehighs7845 3 ай бұрын
2 business days: from Friday to Monday.
@birdbrid9391
@birdbrid9391 3 ай бұрын
flip did not cut it out
@burkskurk82
@burkskurk82 3 ай бұрын
Prime, what about Redis changing licensing model and Garnet (by Microsoft) written in C# outperforming Redis in C++. Help us make sense of it.
@MichaelSalaverry
@MichaelSalaverry 3 ай бұрын
One billion comments, lets go!
@issacwessing4945
@issacwessing4945 3 ай бұрын
I'm having some problems solving this in HTML
@retropaganda8442
@retropaganda8442 3 ай бұрын
The word "buffer" CRIES for underoptimised implementation with data being copied between kernel memory and user space process memory. I think i'd start by doing an mmap of the whole data on disc, assuming it's already in the fs cache.
@Tony-dp1rl
@Tony-dp1rl 3 ай бұрын
forEach, map, etc. are the devil in JS
@RenThraysk
@RenThraysk 3 ай бұрын
Unfortunately produces corrupt data. If run it multiple times over the same 13Gb dataset, it'll produce a different result each time. Some temperature values end up in the 10s of thousands, and also new locations appear. Signs of race/memory corruption issues.
@anon1963
@anon1963 2 ай бұрын
What? Your program or the program in the video?
@RenThraysk
@RenThraysk 2 ай бұрын
@@anon1963 The solution in the video.
@anon1963
@anon1963 2 ай бұрын
@@RenThraysk ah ye, they probably ran finished program once and were like: "good enough!"
@dand4485
@dand4485 3 ай бұрын
I'm thinking one way to convert the temp (float) is have a hash map for all 100 possible different values i.e. map("99.9") simply return 99.9....
@imaymakesomevids
@imaymakesomevids 3 ай бұрын
There are 2000 values, cos of the decimals. The hash and lookup would be a lot slower than just parsing the numbers directly.
@retropaganda8442
@retropaganda8442 3 ай бұрын
Don't hash it! Just make a 2000 element array, use the raw bits as an index, and it's gonna be fast.
@michelvandermeiren8661
@michelvandermeiren8661 3 ай бұрын
Java has proven to be the fastest lang on earth with this challenge ! No other lang can compete
@dv_xl
@dv_xl 3 ай бұрын
Firstly this statement is inherently false, it can never be as fast as the fastest asm or c. But more importantly, where did you get that idea? I looked up the results for Java from the test and they were 6 seconds. It's not clear what the hardware used for the testing was, but it doesn't look to me like there's a good cross language comparison table anywhere
@michelvandermeiren8661
@michelvandermeiren8661 3 ай бұрын
@@dv_xl fastest java took 1.4 sec
@CipovPeter
@CipovPeter 3 ай бұрын
i an wondering why you need mutex when reading from file. why not open file x times for reading ? and using seek start reading from right position ? right positions can be computed in main thread at the beginning. sort of index. did not test ot but suppose ut would remove a lot of merge logic from the end of article
@Jesse_Carl
@Jesse_Carl 3 ай бұрын
I was also wondering this
@olhoTron
@olhoTron 3 ай бұрын
Before even watching the video I'll guess the biggest gains will come from reducing allocations
@soggy_dev
@soggy_dev 3 ай бұрын
I actually prefer specific syntax for multiple return parameters 🤷‍♂️ The language is almost certainly creating an anonymous struct under the hood anyway, so I'd rather it be more obvious they're connected/contiguous. Plus you have the option of passing around the entire tuple or destructuring into the components depending on what's the most convenient which just seems objectively better to me. I love go but that's up there with lack of sum types on the list of things that bother me
@aurele2989
@aurele2989 2 ай бұрын
we do a little struct { int a, b, c; } fn(int in) { /* ... */ return (typeof(fn(0))){ a, b, c }; }
@JackClawson
@JackClawson 3 ай бұрын
Boomer loops sounds like a great cereal, now with fiber.
@thekwoka4707
@thekwoka4707 3 ай бұрын
forEach is faster than boomer loops in newer versions of node and in bun. Pretty wacky, but true.
@ThePrimeTimeagen
@ThePrimeTimeagen 3 ай бұрын
Actually not true This test was done in 20.x, 18.x, and 16.x By the very definition they cannot be faster. They can be of equal speed if extremely clever compiler stuff happens. This would require jit to take place as well
@lucsoft
@lucsoft 3 ай бұрын
​@@ThePrimeTimeagen Mmmh i tested NodeJS 21 and actually found it was faster: const array = Array.from({ length: 1_000_000 }).fill(1); time = performance.now(); array.forEach((e) => e); console.log(performance.now() - time); // run was between 10 - 14ms compared with time = performance.now(); for (e of array) { e; }; console.log(performance.now() - time); // run was between 14 - 20ms Wonder why its faster
@rasalas91
@rasalas91 3 ай бұрын
flip did not take that out
@Kane0123
@Kane0123 3 ай бұрын
I’m waiting for a cloud vendor to suggest just running all billion in serverless - scale up to what you need to scale down when you’re done bro, e.z.
@Wielorybkek
@Wielorybkek 3 ай бұрын
I don't get it, the File Read Buffer took only 0.98 s!!!! Why everyone is ignoring it!!!
@FaZekiller-qe3uf
@FaZekiller-qe3uf 3 ай бұрын
Joelang
@mechmaverick
@mechmaverick 3 ай бұрын
I just found your channel and your the dr disrespect of software, get some sunglasses
@Alguem387
@Alguem387 3 ай бұрын
MMAP?
@ytdlgandalf
@ytdlgandalf 3 ай бұрын
These times are too good tobe true. Heavy caching through pagecache. He should flush pagecache before every try. 13GB in 1.96 =~ 6.5GB per second. No way in hell with the mentioned ssd. Flushing cache for honest numbers on the same system is benchmarking 101. Did he ever run the java implementation on his own system to set a baseline or did he just take the other benchmaker's results? Do people even know how to benchmark?
@arden6725
@arden6725 3 ай бұрын
why would you want a software optimization benchmark to be limited by your disk speed, that’s literally pointless
@ytdlgandalf
@ytdlgandalf 3 ай бұрын
@@arden6725 why? For reproducibility. His results could now easily be skewed from run to run if for example chrome is having a bad day and is filling his memory and thereby flushing his oagecache during some runs but not others. If you are unaware of this you make wrong conclusions on what changes made your program faster or not. If you want to take ik out of the equation than the benchmark should've stated to use a ramdisk or generate the data in-process
@javierflores09
@javierflores09 3 ай бұрын
@@ytdlgandalf this kind of code isn't meant to be run within a workstation but a server, meaning it'd be the able to take full advantage of the machine. When it comes to a workstation, all of these low-level impl will fall short behind the general impl because there's no way to predict the amount of resources the environment is willing to give this program in question in order to complete it at the fastest time possible.
@ytdlgandalf
@ytdlgandalf 3 ай бұрын
@javierflores09 this is about reproducibility. Doesn't matter if its your workstation or a "server".
@Tony-dp1rl
@Tony-dp1rl 3 ай бұрын
I still don't understand how these BILLION row challenges are not entirely IO limited ... I mean even in JS, how to you spend more CPU time than it takes to read that much data? :/
@Tresla
@Tresla 3 ай бұрын
This is my question. How are they getting millisecond solutions? What are they running on? My NVMe drive tops out at around 1500MBps, so I couldn't even process the file in less than 10 seconds...
@amjad-se
@amjad-se 3 ай бұрын
Could you please do a video on Pocketbase?
@pylotlight
@pylotlight 3 ай бұрын
does flip even watch the videos or just use the markers seeing he misses every cut request ;p
@user-jw9iw2zy1k
@user-jw9iw2zy1k 3 ай бұрын
13GB in one second? I think the ssd couldn't even be that fast, right?
@ismbks
@ismbks 3 ай бұрын
the one guy in your chat spamming "hardly know her" jokes
@qazarify
@qazarify 3 ай бұрын
This cannot be true, the Kingston SSD SV300S37A is not capable of transferring 13Gb/sec
@Yawhatnever
@Yawhatnever 3 ай бұрын
Windows caches file reads in RAM when it can, so it's plausible that not all of the reads are hitting the disk
@sedrakpc
@sedrakpc 3 ай бұрын
How it’s done in Java in 1.5 second? Now you have to read the java version)
@lazyh0rse
@lazyh0rse 3 ай бұрын
they used native GraalVM, it compiles java to machine code
@javierflores09
@javierflores09 3 ай бұрын
​@@lazyh0rsethis wasn't the only reason, sure it reduced the time by removing the startup cost however there are many tricks that led to the 1.5 second (and even, 323ms when using all the 32 cores of the test machine instead of just 8). There is a great blog post by QuestDB that explains the tricks used in the top solutions in detail.
@bhuvya11
@bhuvya11 2 ай бұрын
I want someone to try this in javascript 😂😂😂
@pantsoff
@pantsoff 3 ай бұрын
Flip didn't take it out
@avalagum7957
@avalagum7957 3 ай бұрын
That Go person used tabs (8 spaces)?
@Yawhatnever
@Yawhatnever 3 ай бұрын
All Go code uses tabs. The reason it looked excessive was because the default browser styling for the tab-size property is 8 spaces, and apparently they didn't change it with css.
@avalagum7957
@avalagum7957 3 ай бұрын
@@YawhatneverOh, thank you. I didn't know that.
@michaelgreenberg6344
@michaelgreenberg6344 3 ай бұрын
On his hardware, he's I/O bound and any optimization is useless. Dude has 32 gigs of RAM. Meaning that, on an idle enough system, most of that memory will be used for file system cache, into which a file with the size of 13GB fits quite neatly. I will probably not be too exaggerating if I say that he only read the file from disk once - the first time he ran his program. If not once, then by the fifth run, the entire file would be up in RAM for sure. All the rest of the "I/O" tests were performed against the memory, which just checked how fast memory copy in chunks of different sizes and multiples of allocations can be performed. Had he been performing actual I/O, there's no way he'd be getting >13GB/s (which a time of ~0.98s suggests.) In fact, his drive is rated at 497MB/s (manufacturer spec), so on that hardware, it's useless to play with the buffer size, since you won't be reading the file faster than ~27 seconds, as the first file read test with the buffer size of 1024 would suggest. 13*1024/497=26.78, and i'm pretty sure that all the allocations were done during iowait, so it's safe to assume the file size is not exactly 13GB, but more around 13.3-13.5 :D This article is written by someone who probably doesn't understand storage or operating systems too well (using windows for development - first hint... jk,) but it's a nice experiment to see how well you can optimize such an algorithm if your disk bandwidth is infinite.
@viktorhugo1715
@viktorhugo1715 3 ай бұрын
Renato Pereira is a Brazilian name soooooo... BRAZIL MENTIONED LWSGOOOOOOOOOO BRAZIL!!11!1!1!1!!1!1!1!11!1!1!1!1!!!!1!!1!1!1!!1!
@bluecup25
@bluecup25 3 ай бұрын
15:55 - Ignored
@ytdlgandalf
@ytdlgandalf 3 ай бұрын
nobody is wondering how he can read 13GB in under a second? Really?
@sebastianwapniarski2077
@sebastianwapniarski2077 3 ай бұрын
Can anyone suggest a streamer that is as good with SWE but on the other side of the spectrum - TEMPERAMENTwise. I'm more of an Uncle Bob kind of guy.
@havokgames8297
@havokgames8297 3 ай бұрын
Stalagmite - *might* reach the ceiling one day Stalagtite - holding on *tight* so it doesn't fall
@b0nes95
@b0nes95 3 ай бұрын
how can you read 13GB from disk in 1.5 seconds even :/ I need to watch the rest of the video lol, the timer must've been started while the 13GB was in mem
@Tresla
@Tresla 3 ай бұрын
RAM disk possibly?
@TheRadischen
@TheRadischen 3 ай бұрын
2
@FrederikSchumacher
@FrederikSchumacher 3 ай бұрын
Gopoutine
@jazzochannel
@jazzochannel 3 ай бұрын
how can i insert a yomoma joke here, or an insult involving your mom?
@sebastianwapniarski2077
@sebastianwapniarski2077 3 ай бұрын
There are two kinds of great professionals who show of their skills: 1) will make you inspired 2) will throw you into despair. For me Prime is the second kind. But he's funny. I give him that. And him boasting about how he ruined every ones day when he got that calc test way ahead of others back in his uni times is just a proof of this.
@himbo754
@himbo754 3 ай бұрын
32 GB RAM? So laughably small...
@mkvalor
@mkvalor 3 ай бұрын
Ain't no way you're a Boomer. _Maybe_ Gen X.
@truehighs7845
@truehighs7845 3 ай бұрын
Why windows, that's gotta count for half the slow down, you want to optimise, get rid of windows.
@Murderbits
@Murderbits 3 ай бұрын
Good job, Flip.
@selimpy8105
@selimpy8105 3 ай бұрын
damm so early
@chasep9440
@chasep9440 3 ай бұрын
Or you could just code in Elixir because its just straight better.
I am a Bad Software Engineer | Prime Reacts
26:06
ThePrimeTime
Рет қаралды 155 М.
Golang is BAD for SMART PEOPLE
27:25
ThePrimeTime
Рет қаралды 248 М.
Osman Kalyoncu Sonu Üzücü Saddest Videos Dream Engine 170 #shorts
00:27
The day of the sea 🌊 🤣❤️ #demariki
00:22
Demariki
Рет қаралды 86 МЛН
A pack of chips with a surprise 🤣😍❤️ #demariki
00:14
Demariki
Рет қаралды 54 МЛН
OMG😳 #tiktok #shorts #potapova_blog
00:58
Potapova_blog
Рет қаралды 3,6 МЛН
I Parsed 1 Billion Rows Of Text (It Sucked)
39:23
Theo - t3․gg
Рет қаралды 61 М.
The Stockholm Syndrome of SQL | Prime Reacts
31:21
ThePrimeTime
Рет қаралды 134 М.
Golang is OG?
5:16
Martin Baun
Рет қаралды 5 М.
How Fast can Python Parse 1 Billion Rows of Data?
16:31
Doug Mercer
Рет қаралды 172 М.
Prime React: Fast Inverse Square Root - A Quake III Algorithm
30:19
ThePrimeTime
Рет қаралды 148 М.
The Pain Of Frontend Dev | Prime Reacts
21:42
ThePrimeTime
Рет қаралды 224 М.
So You Think You Know Git - FOSDEM 2024
47:00
GitButler
Рет қаралды 1 МЛН
WHY IS THE HEAP SO SLOW?
17:53
Core Dumped
Рет қаралды 205 М.
What Does Your Editor Say About You | Prime Reacts
22:12
ThePrimeTime
Рет қаралды 268 М.
[UPDATE] Mojo Is Faster Than Rust - Mojo Explains More
52:09
ThePrimeTime
Рет қаралды 239 М.
1$ vs 500$ ВИРТУАЛЬНАЯ РЕАЛЬНОСТЬ !
23:20
GoldenBurst
Рет қаралды 1,1 МЛН
iOS 18 vs Samsung, Xiaomi,Tecno, Android
0:54
AndroHack
Рет қаралды 91 М.
APPLE совершила РЕВОЛЮЦИЮ!
0:39
ÉЖИ АКСЁНОВ
Рет қаралды 4,1 МЛН