Using Image Recognition to DESTROY Fruit Ninja

Рет қаралды 442,358

Күн бұрын

To try everything Brilliant has to offer-free-for a full 30 days, visit brilliant.org/CodeNoodles/. You’ll also get 20% off an annual premium subscription.
This project has been a long time in the making. I set out last year to automate Fruit Ninja with image recognition, but I ran into a couple of road blocks along the way. But luckily, this year I decided to try it again, and it worked out fantastically to say the least!
This video was sponsored by Brilliant
LINKS
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
Support the channel: ko-fi.com/codenoodles
Itch.io: codenoodles.itch.io/
GitHub: github.com/OfficialCodeNoodles
MUSIC
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
Shoutout to @qumumusic for making wonderful remixes of some of my favorite video game songs! Check out their channel if you haven't already!
• The Legend of Zelda - ...
Roboskater by Audionautix is licensed under a Creative Commons Attribution 4.0 license. creativecommons.org/licenses/...
Investigations by Kevin MacLeod is licensed under a Creative Commons Attribution 4.0 license. creativecommons.org/licenses/...
█▀ █ █ █▄▄ █▀ █▀▀ █▀█ █ █▄▄ █▀▀
▄█ █▄█ █▄█ ▄█ █▄▄ █▀▄ █ █▄█ ██▄

Пікірлер: 345

@CodeNoodles Ай бұрын

To try everything Brilliant has to offer-free-for a full 30 days, visit brilliant.org/CodeNoodles/. You’ll also get 20% off an annual premium subscription.

@deltadeltagrand2842 Ай бұрын

I feel like something that could improve it is if you had the code specifically look for any black spots that would mark a Bomb and label areas around them as dangerous, that could help the code avoid bombs as well as make sure it doesn’t try to hit fruits too close to bombs. I would also make it so that it doesn’t scan too close to the bottom of the screen as a bomb could come up right after or at the same time as a fruit and get hit.

@gaco577 Ай бұрын

if object = bomb is bad else is good

@CodeNoodles Ай бұрын

Basically

@catWithFunkyFace Ай бұрын

if only it was this easy

@IgsihziysPleazure Ай бұрын

That looks like something from scratch.

@cybi124 Ай бұрын

if (object = bomb) { donttouch } else{ destory }

@Duckless137 Ай бұрын

==*

@lukeseaman2994 Ай бұрын

The somewhat unoptimized nature of the program gives it a lot of personality and comedic value 10/10

@CodeNoodles Ай бұрын

Thanks, that makes me feel better 😆

@IntrovertSinceBirth 5 күн бұрын

Programming also makes mistakes like humans do?!

@johnsimpsen5 Ай бұрын

It would be cool if the program waited a while before the fruits were on screen and then calculate how many there are. If there are more than one, it tries to slice them in one go instead of a bunch of slices.

@Rjciralli Ай бұрын

I can already imagine that constantly catching bombs in the big slices

@dylanherrera5395 Ай бұрын

@@Rjciralliperhaps a pathfinding algorithm to avoid the bombs, although this might be too slow

@advance64bro Ай бұрын

The program does what’s efficient

@TheStolenBattenberg Ай бұрын

@@advance64broit uses python…

@inakiorozco881 14 күн бұрын

@@advance64bro the program does what is designed to do, It can be efficent with fewer slides if its designed that way

@rockinggamerdude Ай бұрын

The problem I see is it doesn’t wait for all the fruit to make sure to get a combo for a points bonus

@mateuszpragnacy8327 Ай бұрын

IT is possible to train ai that way just Smart genetic algorithm or other

@advance64bro Ай бұрын

It doesn’t even know how the game works, it’s a program that does what is told

@mateuszpragnacy8327 Ай бұрын

@@advance64bro yes but no. You can train it just genetic algorithm and train 10 ai at time and ai with biggest score replicates like natural selection and there is a chance that ai learns to mąkę a combo

@Cozumii 18 күн бұрын

Isn’t that lost score made up by how many critical it gets?

@cvabds Ай бұрын

This could be better optimized if you didn't based it on color recognition but actually on moving pixels on the grid you created. Based on that principle, you should need to only recognize the black color of the bomb, everything shouldn't be biased on only color. I saw it bug on the background splashed fruit some times

@ryans3979 Ай бұрын

This would add a small delay because he'd always be one frame behind (need to take the current frame and subtract the previous frame pixels that didn't move), but I do agree that the optimization would probably speed it up enough to be worthwhile. He should also not be processing the full RGB image, especially because he's in python, that's obviously going to be slow. Turning it into a grayscale or HSV (lightness channel) 2D array and doing some sort of processing to check for the darkest pixels would definitely be faster.

@cvabds Ай бұрын

@@ryans3979 nice! Thanks for the input! That makes sense!

@ilovecoffee4978 21 күн бұрын

@ryans3979 could he not make it identify the position and type in one frame, wait like 3 frames, and then identify the same objects position for average velocity, then swipe afterwards based on delay?

@madhuchowdary8826 5 күн бұрын

Wont that be pretty hard, as the program needs to calculate a path to slice, a path which does not include any bombs in between

@emanuelemanzoni5191 19 сағат бұрын

how would you detect moving pixels

@zobiah6263 Ай бұрын

This feels like a strong candidate for reinforcement learning imo. Just give it the average color values you have already collected, a reward function based off the in-game score system (obviously make bombs a high negative), and watch it go.

@4_real_bruh 11 күн бұрын

Could also just use a single convolution operation with a filter of 7x7 since the bombs always have a minimum size, then compute the average of the colors as he did and if its not white or black, slice the fruit by going 10 pixels in either direction from the pixel

@sbd2639 Ай бұрын

I have made a program in python to automatically complete tasks in among us and wrote code for almost all tasks for the first map. I faced the same problem in some tasks ( like clean vent, clear asteroids etc) where the image recognition would not work properly due to random rotations of the sprites on screen. Your solution to the problem might be perfect in my program and I am gonna try that soon. It might be even better suited since there is even less chance of false positives ( which was caused by the splattering of the fruits on the wall in fruit ninja ).

@TheFurry Ай бұрын

awesome project! Are you going to do a video on it?

@sbd2639 Ай бұрын

@@TheFurry nah probably not. I just made it because my USB mouse broke and using laptop's mousepad was very annoying. If you are interested, I could give you the code but since I made it just for me, I didn't take the screen resolution into consideration, so it only works on the monitor with the exact resolution of 1920x1080.

@HarshWeave9487 Ай бұрын

Interesting.... Would you mind if I could see the code? If you can share the link, I would be grateful.

@sbd2639 Ай бұрын

@@HarshWeave9487 This is my first time using github so sorry if there is any mistakes.. Also, since I just made it for me, almost every pixel coordinate is hardcoded so there are just random numbers everywhere. If you have any questions, you can just ask here, I am online pretty regularly.

@sbd2639 Ай бұрын

@@HarshWeave9487 I commented the link here but I think it got deleted by youtube. Are there any other way to share the link???

@ryans3979 Ай бұрын

Being somebody who works in the computer vision field, I feel like it would've been simpler for you to convert the image to HSV, take the value (lightness) channel, and then binarize the image by checking if it's less than a certain threshold. From there you can see how many connected pixels there are, and if there are more than say 1000 black pixels, its a bomb.

@McTuber42 Ай бұрын

The rhythm of the slicing syncs surprisingly well with the music after 8:25

@-ShoeCat Ай бұрын

i haven’t heard about this game in a long time, when i saw this, it reminded me about how i got around 1390 while playing this in a daycare

@CodeNoodles Ай бұрын

That's impressive!

@-ShoeCat Ай бұрын

@@CodeNoodles thanks

@shadow_blader192 Ай бұрын

You are writing functions without space between them 😭

@Trevorus1 Ай бұрын

May I ask what’s happening in your profile picture with the mimi sentry?

@aaronking2020 Ай бұрын

@@Trevorus1 mini sentry meets big thing

@trystankitty5393 Ай бұрын

@@aaronking2020why

@romancatholicgameing Ай бұрын

@@Trevorus1 It takes someone with experience to recognize that pfp's image and what happened in it... I don't know if I should be horrified or impressed.

@letter_o_hyphen_letter_o Ай бұрын

is bro's argument invalidated by deranged pfp?

@ElementEvilTeam Күн бұрын

would've been useful 14 years ago

@ShockedTaiLung Ай бұрын

Aw yeah it’s noodling time

@TVDemonstro Ай бұрын

Loved the part where he said "noodling time" and noodled all over the place

@GreatDynamics Ай бұрын

I also loved the part where he said "noodling time" and noodled all over the place

@affegpus4195 Ай бұрын

The color recognition is a clever idea. The fact it is fast it can be used in combination with other methods for increasing accuracy

@JamesTDG Ай бұрын

I feel like this system could be expanded more if used right. Like for example, if the system detects high red pixel counts in a region, have it take a screenshot and use image recognition to see if it can detect a red arc at around 75% completion, use an or function with it so it can also check the image for an X. If either result comes true, it will determine the region to be dangerous for the next x-amount of frames. Should help improve the system by a bit, and even can use fewer resources if it is able to know the danger region before doing the bomb check.

@ODISeth Ай бұрын

Oh wow you've been at this one for quite some time, excited to see how it turned out!

@trystankitty5393 Ай бұрын

@ODISeth Ай бұрын

@@trystankitty5393 hi?

@trystankitty5393 Ай бұрын

@@ODISeth hi?

@ODISeth Ай бұрын

@@trystankitty5393 you replied to me lol

@trystankitty5393 Ай бұрын

@@ODISeth nope

@gummybread Ай бұрын

8:37 8:59 *RUUUUULES OF NAAAATURE*

@lingeprune2373 Ай бұрын

yeah it's swinging at the speed of raiden crackhead mode

@wChris_ Ай бұрын

I thought you would use image recognition with a neural net for object detection, but maybe that could be v2 if you ever want to do that again.

@skmiraj1549 6 күн бұрын

I don't know what is going on but I love it

@SuadoCowboy Ай бұрын

why don't you just instead check if it's a bomb or not? as far as i know the bomb is the most different from them and by only checking the bomb you could optimize this code alot and then you only pass the mouse on things that are on movement except the recognized(s) bomb(s) maybe you could even try using grayscale images or something idk

@melody-v- Ай бұрын

IF ONLY

@advance64bro Ай бұрын

Dude, do you not know how hard that is

@SuadoCowboy Ай бұрын

@@advance64bro do you?

@advance64bro Ай бұрын

@@SuadoCowboy yes

@SuadoCowboy Ай бұрын

@@advance64bro and don't you think it's worse checking each type of fruit instead of just checking if it's a bomb or not?

@vitatreat9037 Ай бұрын

I feel honoured to get this video on my feed. This video made me excited for sure 👍

@CodeNoodles Ай бұрын

Thanks, I really appreciate it!

@CarlosRoxo Ай бұрын

You can try using something like a yolo network for the fruit recognition. Most recent models run very fast with a medium capacity gpu and are pretty accurate

@ofridaniel2127 Күн бұрын

Exactly what i thought

@Jayrenzyx Ай бұрын

I gotta admit that segue was clean

@linkmastr Ай бұрын

2:50 nice explanation of kernels in an image recognition model

@fadlanal-amsi9839 Ай бұрын

Its magical to see a fresh half brick games content these days

@Mossy_Dahlia Ай бұрын

I’m not too savvy on machine learning, but would it be possible to extract the UV maps from the fruit and use that to differentiate them from the bombs? The UV map has their whole texture so would it be possible to use that to understand and recognize a fruit at all angles?

@mateuszpragnacy8327 Ай бұрын

You can lower the screenshot resolution so network can have whole screen as input and train it if game is laggy while training just make game run slower

@ranarehanqaisar2266 Күн бұрын

Bro is going to be a genius one day.

@nasirgrant4271 2 күн бұрын

Never thought I would see TAS for Fruit Ninja

@jordancrouse714 Ай бұрын

Yay! Excited to see you post 🥰 only place I'm interested in code 😂

@KirbyCoder Ай бұрын

Now I wonder what it’d be like to train an AI to play this game

@harkiratsingh3967 2 күн бұрын

It made me remember a scene from tge dark forest where the droplet is destryoing the the space fleet , its drscribed as God 's scribbling as the droplet is ramming objects taking sharp turns which are zero in accordance to our aerodynamics

@midtech168 Ай бұрын

If I had this back in 2011 I would have seen so popular in school lol

@yanglightheart Ай бұрын

Now that's an actual fruit ninja, my personal best is 694 though... in the arcade mode.

@gdclemo Ай бұрын

Very good. Next project: build a robot arm with a samurai sword that chops up fruit you throw at it (but not bombs)

@notchpoodles5864 6 күн бұрын

0:45 I️ remember this from CodeBullet (an angry Australian man who also programs robots to play games for him)

@I_am_Itay Ай бұрын

I think it would be fun to optimize it, here is sime idea that would maybe improve it- Waiting with slicing and calculating where each object is not just frame by frame so it would include the speed, direction, where the object would get in the next frames and I think it would give better accuracy + would give the compatibility to plan combos with smooth slices instead of just spamming

@advance64bro Ай бұрын

It does what is efficient, don’t be that harsh just because you’re dissatisfied

@I_am_Itay Ай бұрын

@@advance64bro I am not harsh, I really like it and the video and I just want to add suggestions so maybe he would do another video and improve it or to inspire someone else who interested in this for example I started building my own version to this bot

@advance64bro Ай бұрын

@@I_am_Itay doing something like that would have to make the fruit identifying program to be more complex which you already saw was really hard to do

@I_am_Itay Ай бұрын

@@advance64bro Of course it would be a bit more complex but it's well within the capabilities of code noodles, he did much harder things. The preprocessing he did would really help, it would be more work on how the bot play than on the identification

@ramadhanisme7 16 күн бұрын

The chopping is unreal 🤣🤣

@krischarles4326 2 күн бұрын

How is your IDE not SCREAMING at you…? PyCharm would be kicking in my door at the first lack of whitespace 😅

@cinnamonsugarcourtney6073 Ай бұрын

Oh hey, I was wondering why you didn't just use color averages at the beginning, looks like you figured it out though! :3

@user-js4xl7pw7l 20 күн бұрын

You can change the background, your blade color and splashes on the dojo in Srttings, as well as try to instantly revoke any area containing black pixels or i dunno, just (if any of this place in a screenshot has black bad No black Good, slice)

@equalent 15 күн бұрын

your naming convention in Python should be illegal

@GimmeMyHandleBack Ай бұрын

Could you edit the color mixing program to favor black more heavily than other colors? That should help with the bomb identification

@memeboi6913 Ай бұрын

Finally, the first Fruit Ninja TAHS (Tool Assisted High Score)

@kelvinluk9121 4 күн бұрын

Would it be easier and more accurate to fine tune an image segmentation model for this task?

@racingfarmer053 Ай бұрын

Would it be possible to code it so if it detects any black on am object at all to steer clear of it. It not because of other dark spots on the background and such

@charlierogers98 10 сағат бұрын

Make a bot to destroy Candy Crush and win every level without doing any single micro-transaction.

@silver2797 22 күн бұрын

your fruit killing skills are remarkable

@pierpaolopinna6301 Ай бұрын

considering an impunt based on 2 succesful screenshot (instead of 1) would make the program too slow? good job thought, nice project

@cosmilitebar6772 Ай бұрын

For screenshots i tent to use mss library, its much faster than pyautoguis screenshot method. Maybe you could do a v2 with machine learning, yolo 5 or really any version of it would probably perform much more accurate. And also extremly fast as that stuff runs on the gpu. Nice Video, its cool to see games being automated.

@cruncyart Ай бұрын

bro update your profile picture that's an extremely old cosmilite sprite

@konothomasda7430 21 күн бұрын

Maybe implementing sound extraction might help?

@Ramzi65ok 16 күн бұрын

Combos would be so cool

@collincatmull1369 Ай бұрын

Code noodling time :) I always have loved your videos.

@CodeNoodles Ай бұрын

Thanks!

@collincatmull1369 Ай бұрын

@@CodeNoodles Wow, Ive never had a youtuber near this popular even get close to talking to me. Thanks :)

@jasonorjoshlee7607 Ай бұрын

The bot even got a 5 fruit combo😮

@niji4894 4 күн бұрын

The most real part in this video. 5:20 the naming

@ThegreatLynix Ай бұрын

Now,Do it with snake eating fruit game 😈

@lukraniom8731 2 күн бұрын

You could also just hold your finger in one spot and if a fruit passes by it it splits

@elicooper5233 Ай бұрын

we need code bullet to do this

@usptact 13 күн бұрын

You can try to use computer vision algorithms for object detection to detect and track objects. Look into detectors like SIFT for example. Warning: it’s a DEEP rabbit hole.

@feffy380 Ай бұрын

I'm thinking the color sampling code could have been massively simplified by just scaling the image down, which is a highly optimized operation in most image processing libraries

@ontoya1 Ай бұрын

You should tag the dude that created Fruit Ninja, he's on KZfaq!

@user-px2we6di1w 2 күн бұрын

using A* pathfinding can be good choice for finding the right path for the mouse

@piercegames4577 Ай бұрын

even in the image that had a blur effect I could tell you were using Visual Studio and Python

@robertstevensii4018 Ай бұрын

7:55 watch the left side of the screen and imagine a very angry Beatrix Kiddo

@Happy_Koopa_boi Ай бұрын

You should do an image recognition thing but for suika game

@noahnash7265 Ай бұрын

Would there be a way to remove the fruit juice in the background to make it perform better?

@inciampare9083 19 күн бұрын

8:27 funny green splatter

@lonelyPorterCH Ай бұрын

Has been a long while since I played this^^

@danygeo7365 19 күн бұрын

Best thing in the video was this at 8:37 :) (joking great job u did👏)

@ExploringNew1 15 күн бұрын

How did you get a video of what the program sees?

@CodeNoodles 15 күн бұрын

I wrote some code to create a video file where I could draw objects on top of the image to display information.

@fishygaming2554 Ай бұрын

One of the best games.

@mrcooltree3566 Ай бұрын

Why not have the ai focus on the background color and bomb color and when thag color is over the screen it does not attack there? Sure you couldn’t get combos but I think theoretically it would work easier faster and longer since instead of picking out a fruit it is just picking out a difference in what is normally there

@makxs4818 Ай бұрын

what is the font You use on 0:16

@wewuwewu 13 күн бұрын

fruit ninja says "G太ME OVER"

@paradoxcorporated2906 Ай бұрын

Bro is code bullet 2.0

@real1cytv 16 күн бұрын

5:23 this code is my absolute nightmare... Why didn't you put newlines inbetween the functions but put them around the while loop?

@godofall332 22 сағат бұрын

Can we use Ultralytics YOLOv8 for this? I understand the processing time would increase, and it would need some program optimizations (what you did at 6:11), but it might be able to avoid bombs completely, making it better at play. Creating a dataset for this is easier than normal; let me know your take on this. By the way a great video, it helped knowing how you approached this problem from a problem-solving point of view 😊.

@EvaleAxolotl 16 күн бұрын

Fruit ninja my beloved

@kmyau_2583 14 сағат бұрын

2:44 The background music sounds familiar 🤫

@some_guy1990 Ай бұрын

Toriel got stuck in the dummy room

@angelfire9622 Ай бұрын

if you have a good enough pc you could have used YOLO for object recognition and have a better accuracy more easly

@bachoulekikoune Ай бұрын

yes, small yolo models can even run on crappy pcs pretty well

@strawsEXE Ай бұрын

Can you make a video about using image recognition to get the best possible scores on the human benchmark website?

@CodeNoodles Ай бұрын

That's a good idea!

@lynxiegrande Күн бұрын

keep up the good work 👍🏻

@user-fh3wg8oj6q 16 күн бұрын

Indeed, there is no need in ML. Pixelperfect analysis is enough

@nicolasanderson4146 Ай бұрын

On the topic of older games, imagine coding for jetpack joyride...

@DizzeeAfterHours 2 күн бұрын

THERE WILL BE FRUIT…JUICE!

@00_ontop Күн бұрын

The animatronics do get a little quirky at night…

@rbbr08 8 күн бұрын

Bro, just use deep learning. If you want it to be fast you can use some mobilenet network along with some rpn detection heads... Faster R-CNN or something like that. Like a 2 stage network.