I wonder if the benchmark runs share the same threadpool. That'd explain a lot(Like the worst case for parallel 2 mill benchmark being about the same as non parallel)

The thing about multithreading is that there is a cost to initializing it. If the cost of that and joining the result is higher than simply doing it in one thread, it will be slower and require more resources.

it's not "big Oh" it's "O", people say big to communicate that it's the capital version of the letter, but you don't need to say that in text.

Polars is great

Blazingly Fast

@@tardagardovarg you have to read that in the primeagens voice :) It is Blazingly Fast.

What's the joke?

@@SlexisSlacks which joke?

@@codeshowbr Why it is blurred. I thought you said "what you did" as in "I see what you did!", so as if there was a secret joke 😅

Rayon's parallel iterator actually has a chunk(n) method that splits work into chunks of size N. I managed to make my performance be on par with serial solution for 200_000 elements by doing pub fn par_number_of_adults(people: &Vec) -> usize { people.par_iter() .chunks(4096) .map(|p|p.iter().filter(|&&p|p.age >= 18).count()).sum() } Notice that we now have map method that calculates a sum of number of adults per row as chunks changes the iterator to operate on rows instead of individual elements. Also, probably because I messed up, the inner filter method now operates on type &&&Person (does it do triple dereferencing?) As I mentioned, this solution is about on par with the sequential one, and for 2_000_000 elements it's twice as fast, just like in the video running 2 tests test tests::bench_number_of_adults ... bench: 45,952 ns/iter (+/- 1,795) test tests::bench_par_number_of_adults ... bench: 43,662 ns/iter (+/- 5,494) running 2 tests test tests::bench_number_of_adults ... bench: 580,440 ns/iter (+/- 65,093) test tests::bench_par_number_of_adults ... bench: 278,091 ns/iter (+/- 43,477) Also, I don't like this solution in general but use it often. I use Rayon for generating images as a hobby, and if you want to generate a pixel array, chunks() requires you to collect() twice and use flatten(). Here's my function that takes an image loaded from file and calculates distance between current pixel and nearest black pixel. calculate_distance(x, y, &Image, (usize, usize)) takes a coordinate and loops over every pixel in an image let buffer: Vec = image.as_raw() .par_iter() .enumerate() .chunks(128) .flat_map(|row| { row.iter().map(|(index, value)| { let index = *index as u32; if **value == 0 {0} else {calculate_distance(index%dimensions.0, index/dimensions.0, &image, dimensions)} }).collect::()}).collect(); Maybe someone who's smarter than me (not a big requirement) could come up with a better solutions

@@polimetakrylanmetylu2483 Got any idea how to check if the benchmark rounds happen on the same threadpool? Then it might not be able to use more than one thread at a time

ChatGPT now has APIs available, so who knows!

You could code it out, delete everything and then just tab it back with Ctrl+y, no add-ons needed

The filter ran so fast the overhead of parallelism was greater than the gains of doing it in parallel

Using indexes properly and ordering in the query itself is probably like 1000x faster though

​@@Ir0nman55 yes, and no, asuming that proper indexing is already in place, since scalling horizontally is far more easier in the application server than in the database, if the number of elements returned is the same, ordering and doing other types of calculations is better on the application since you release the database early allowing it to perform more work for other queries.