I Made a FAST Search Engine

Рет қаралды 126,836

Күн бұрын

Get $15 free credits with BrightData: brdta.com/conaticus1
BrightData KZfaq Channel: @BrightData
TF-IDF Blog Post: janav.wordpress.com/2013/10/2...
Lemmetization Word Lists: github.com/michmech/lemmatiza...
Crawler Repository: github.com/conaticus/search-e...
API Repository: github.com/conaticus/search-e...
Client Repository: github.com/conaticus/search-e...
Discord: / discord
Github: github.com/conaticus
Twitter: / conaticus
Join this channel to get access to perks:
/ @conaticus
I Made a FAST Search Engine
0:00 Intro
0:20 BrightData
2:10 Inverse Term Frequency & Indexing
6:41 Page Ranking & Lemmetization

Пікірлер: 169

@conaticus Ай бұрын

Start building awesome projects with $15 free credits using BrightData today: brdta.com/conaticus1

@AWIRE_onpc Ай бұрын

@xulaxwtf Ай бұрын

@aryanszone4963 Ай бұрын

@lifeofme702 Ай бұрын

I don't know what this guy said, and still was mind-blown of all the effort this guy puts

@conaticus Ай бұрын

Thanks much so 🙏 It would not be possible without your support

@jaymarksum6542 Ай бұрын

I’m impressed, can’t wait to see you build a multithreaded web server in assembly

@da40au40 Ай бұрын

Why do I find it super funny 😅😅😅.

@ArthursHD Ай бұрын

@@da40au40 Me too :D

@DanskeCrimeRiderTV Ай бұрын

it's not impressive. Of course querying a few hundred or even hundred thousand web pages isn't as complicated or slow of a task than querying trillions of webpages.

@KibitoAkuya Ай бұрын

@@DanskeCrimeRiderTV google also wastes time deciding wether you are allowed to see or not certain sites

@DanskeCrimeRiderTV Ай бұрын

@@KibitoAkuya what does that have to do with anything? Google is still faster at querying trillions of results than this.

@asm_x86 Ай бұрын

That's really impressive, I can't even figure out how to run it.

@ZuperPotato Ай бұрын

Nice username

@conaticus Ай бұрын

Just added some instructions to the READMEs if you're interested :)

@asm_x86 Ай бұрын

@@conaticus thanks, I'll do that

@coderx8634 Ай бұрын

Love your content. You and your quality have really improved. Keep it up ❤

@conaticus Ай бұрын

Thanks so much, your support means a lot ♥

@greensporevalley Ай бұрын

SERBIA MENTIONED 🎉🎉🎉

@europa_the_last_battle Ай бұрын

Now waiting for Russia 🥰

@RealMephres Ай бұрын

@@europa_the_last_battle>goes to comments >sees meme comment >looks at replies >only a LARPer replied lol

@MAXHASS-ph5ib Ай бұрын

@@RealMephres this aint 4chan nga

@jawadmansoor6064 Ай бұрын

that name rings a bell, maybe from some kind of Serbian movie?

@RealMephres Ай бұрын

@@MAXHASS-ph5ib tell that to the LARPer dawg

@ccost Ай бұрын

7:40 flashing those questionable websites in a sponsored video is quite the move

@twitchizle Ай бұрын

You scared of porn?

@coderan5029 21 күн бұрын

This is basically what we learned in my big data class, but we used map-reduce to do the TF-IDF calculations, so it's impressive you figured this out on your own

@rafaelpereiracoias1047 Ай бұрын

Nice video and nice code, keep up the good work!

@ExpandedCuber Ай бұрын

Let's go another conaticus video

@foqsi_ Ай бұрын

Love this dude and his video projects

@conaticus Ай бұрын

🙏

@MySachincool 10 күн бұрын

Subscribed & notifications on :) you deserve more recognition bruh

@GermanTimecrafter Ай бұрын

such a cool video! i love the way how you explain what you are doing :) random question but what is your editor font?

@conaticus Ай бұрын

Appreciate it :) I'm using Jetbrains Mono it's free to download

@polyshrub Ай бұрын

This is very impressive, what was the size of the database when indexing is finished? Seems like it would be quite big

@turb0004 Ай бұрын

Please finish your file explorer in rust fully, because the idea of it is awesome. Love your videos, content is very engaging 🎉

@R_Y_Z_E_N 23 күн бұрын

Google also does the same but with disstributed computing to reduce the overall time . Just scale the database horizontally and mimic googles apporach

@6IGNITION9 Ай бұрын

filter out JS for another 10x bandwidth savings alternatively use an adblocker. (can puppeteer do that? It's just chromium right?)

@devinlauderdale9635 Ай бұрын

The problem is this approach is susceptible to SEO spamming/invisible SEO keywords

@conaticus Ай бұрын

Yeah for sure, realistically it should be moderated based on user interaction as well

@iritesh Ай бұрын

Awesome effort ✨

@SG-kn2jl Ай бұрын

Why did you choose TF-IDF instead of word2vec or any context aware model?

@skorp5677 Ай бұрын

+1 Woule like to know

@madalenaferreira3018 22 күн бұрын

great video, gave me ptsd from my information retrieval class though

@stayhappy-forever Ай бұрын

thats insane, hows this only at 12k views

@alexmoses3215 8 күн бұрын

Programming 🤝 martincitopants…match made in heaven

@Nerdimo Ай бұрын

Impressive, seriously!

@80sVectorz Ай бұрын

3:07 Best pronunciation of Euclidean I have every heard :P

@CrazyDiamondo Ай бұрын

Where?

@80sVectorz Ай бұрын

@@CrazyDiamondo I added a timestamp

@a6gitti Ай бұрын

Supa dope. I would like to use this search engine of yours

@allenfpascua Ай бұрын

Super good editing 🫡🫡🫡🫡

@conaticus Ай бұрын

Would not possible with your breathtaking animations 😄

@dreamsofcode Ай бұрын

🔥🔥🔥

@jugurtha292 Ай бұрын

very nice, built something similar for my info retrieval class. we have to use okapi bm25 formula for the ranking but overall very similar. scrape, tokenize, parse, inverted index, rank

@lonelybookworm Ай бұрын

Well of course it is very fast, it only has like 200 websites

@errplane_ Ай бұрын

oh my fuck i saw this on your github last night

@jsalsman Ай бұрын

I believe it's "inverted indexing", as inverse indexing is something else.

@yorailevi6747 Ай бұрын

how much did you pay for the web scraping service in total?

@maksymilianglowacki1409 Ай бұрын

is this engine oneline or ( wouldt it be abel to be oneline for otcher users ) so otcher also coulst enjoy it? or was it dust a peak or somthing you made cuz ( you where bored or smt )

@user-xl2om2up2x Ай бұрын

W ad plug, it's 100% relevant and actually necessary to fulfill the premise of this vid.

@gaimnbro9337 Ай бұрын

Nice job :D

@MortonMcCastle Ай бұрын

Good! The world needs a new Google Search, one that's more like how it was in the 2000s.

@thekwoka4707 Ай бұрын

How much did the scraping cost if it wasn't free?

@ethanstewart1011 21 күн бұрын

How did you manage to get a node.js memory leak??

@joenutt1232 Ай бұрын

Create your own database engine for shits and giggles

@conaticus Ай бұрын

B+Trees 💀

@mahrezjanati3426 Ай бұрын

first time watching a vid of yours ... i have one question : why are you vibrating ??

@-rate6326 Ай бұрын

Cause he is vibrator

@animeworld4775 Ай бұрын

what is things that i should to know or learn to create like these projects

@GONDWANA-de4od Ай бұрын

HTML for website creation CSS page designing Javascript for making website dynamic and for backend SQL for indexing Rust for fast backend services

@gammongaming9081 26 күн бұрын

yk what would be funny? making the slowest search engine possible without like halting the program for a set time, just with maths

@carlitosdummy Ай бұрын

i love this channel

@datainsight1724 Ай бұрын

Next time use the Common Crawl dataset ;)

@HyperCodec Ай бұрын

Bro managed to memleak in js

@larry_berry Ай бұрын

Lol. Got notif after clicking the video.

@gamedirection_us Ай бұрын

🍎 👀 .. Apple being like "when will it be ready?".

@gopallohar5534 15 күн бұрын

ain't see rust there!

@Raven-fu1zz Ай бұрын

Remember, never return an over 18 site without an over 18 word in the search request

@SlimyFrog123 Ай бұрын

Now make your own email system to go along with it. 😉

@lazarusNoob Ай бұрын

You should host it

@playtatus1758 Ай бұрын

how do you edit your vids

@conaticus Ай бұрын

Allen uses adobe after effects for the amazing animations - I just use Davinci to cut things up 😁

@playtatus1758 Ай бұрын

@@conaticus ok thx

@Serhii_Volchetskyi 29 күн бұрын

🔥🔥🔥 I was looking for that algorithm and didn't know its name.

@fangg194 Ай бұрын

you seem ok

@monotonedevelopment Ай бұрын

If only windows file explorer could do the same

@SandWire 26 күн бұрын

For this we have thing named Everything :)

@binpersonal Ай бұрын

"some fucking genius" lmao

@daemonkisure2952 Ай бұрын

how can i install this search engine?

@conaticus Ай бұрын

Instructions are on the Github repos :)

@humanontheinternet6510 9 күн бұрын

Auto solve captcha you say🧐

@a224kkk 28 күн бұрын

Nice, you re-invented the lucene library

@TheRealMangoDev Ай бұрын

good vid

@AquaQuokka Ай бұрын

Rewrite your genetic code in Rust.

@pyyrr Ай бұрын

i would rather be bug free so i will pass

@_DarkLiquid Ай бұрын

discord clone when

@ALTERRAa8 Ай бұрын

6:08 nahhhhhhhhhhh whats bro even searching 💀💀💀💀

@v037_ Ай бұрын

I found a worthy opponent

@thescratchguy428 Ай бұрын

at a desert

@trolIface_ 29 күн бұрын

hub 🎉🎉

@igrb 23 күн бұрын

nice

@Xanmattauri Ай бұрын

@google acquire this man

@Faeest Ай бұрын

why disallow and user-agent matter? can't you just scrap everything?

@skorp5677 Ай бұрын

You can but it might be illegal

@sleepybraincells Ай бұрын

Why is there Rust in the thumbnail? This was written in Javascript

@conaticus Ай бұрын

Used Rust for the API and TF-IDF matching - decided not to keep in much of the footage for that as it was already explained in the animations

@neologicalgamer3437 Ай бұрын

Bro sounds like WilburSoot

@J0Y22 Ай бұрын

shockedd

@Miluum 29 күн бұрын

1:06 automatically solve captchas? i knew these things exist just to waste our time and energy

@Tech_Code127-76 Ай бұрын

Good

@Ayymoss Ай бұрын

MAKE LONGER VIDEOS

@Macellaio94 Ай бұрын

Liked and subbed

@danielisop3182 25 күн бұрын

What did u mean by the websites u shouldn’t have searched

@dylhack Ай бұрын

da goat

@juniordevmedia Ай бұрын

what TF is IDF ?!!

@neofox2526 Ай бұрын

idk man but watching it makes me feel smart

@jamesbarret4240 Ай бұрын

Term frequency (the number of times a given word or so shows up in total) - inverse document frequency (the number of times it shows up in a specific document). The wikipedia article is pretty good: en.wikipedia.org/wiki/Tf-idf

@chiroyce Ай бұрын

What are the consequences of scrapings sites you aren't allowed to?

@conaticus Ай бұрын

Probably not much on its own as long as you're not violating copyright - however it is curtious not to scrape sites forbidden by the robots.txt

@trollinqu Ай бұрын

wastes their resources and yours

@monkshee Ай бұрын

damn

@user-fj5ts6sz1f Ай бұрын

rust is a real badass❤❤

@susannerudolph8469 Ай бұрын

then brightdata makes captchas useless

@educacionespecialchannel3756 26 күн бұрын

Captcha's effectiveness has been in question for quite some time now.

@AhmedMahmoud-ec4kz 9 күн бұрын

Great video 😊 FYI: bright data is an Israeli company 😮

@_sohom Ай бұрын

Make a better version of VSCode.

@konstantinsotov6251 Ай бұрын

we had a hackathon where we basically had to implement TF/IDF - also a search engine of a sort, but for files. we did the interface in python and all mathematics processing in C++. It would have been a fun experience if not for the time limit. we struggled really hard, on test data our solution worked faster by an order or two than most other participants, but... we somehow failed on the exam data. we failed fucking IO. and won nothing. I fucking hate hackathons since then. fuck IDF. also maybe this happened because i had written 75% of the code, while 4 other members did almost nothing. It was (their) responsibility to handle IO, and mine to handle mathematics and processing. I hate working in teams. I know noone cares but i might as well just burst out all of the rage I have towards that experience. once again, fuck team work, fuck hackathons, fuck my teammates, fuck everything and everyone

@skorp5677 Ай бұрын

skill issue

@konstantinsotov6251 Ай бұрын

@@skorp5677 exactly

@kavinbharathi Ай бұрын

Not to be the 🤓☝️ guy, but "Jana Vembunarayanan" is pronounced 'Ja' as in 'Jarvis' and 'na' as usual. Just fyi

@conaticus Ай бұрын

Thank you, I'll do this if I ever pronounce it again 😂

@ph03n1x_dev Ай бұрын

You made a search engine for porn?! Thats disgusting... is it on GitHub?! 👀

@conaticus Ай бұрын

All open source and ready to play around with 😂

@latrapa918 Ай бұрын

105

@lukamajcenic1172 Ай бұрын

This is just an ad for BrightData. Compared to previous videos very low effort.

@deadshadow759 29 күн бұрын

this result dont make any sense xha... very fast

@vrljk Ай бұрын

SRBIJAAAAAA

@planktonfun1 Ай бұрын

Still not fast and scalable enough. The result is not even relevant, you made bing not google

@LaugeHeiberg Ай бұрын

wow really? Im also surprised one single guy didnt manage to make a product rivaling Google

@avi7278 4 күн бұрын

You need to learn how to sync up your audio and video.

@DanskeCrimeRiderTV Ай бұрын

how is this impressive? Of course it's gonna be faster. You aren't querying billions or even trillions of web pages unlike Google? So this search engine isn't even faster than Google...

@conaticus Ай бұрын

It wasn't meant to be impressive it was meant to be informative and entertaining 👍

@DanskeCrimeRiderTV Ай бұрын

@@conaticus your thumbnail implies it is faster than Google. And I believe the original title did too.