Python + PyTorch + Pygame Reinforcement Learning

Python + PyTorch + Pygame Reinforcement Learning - Train an AI to Play Snake

Рет қаралды 355,454

Күн бұрын

In this Python Reinforcement Learning course you will learn how to teach an AI to play Snake! We build everything from scratch using Pygame and PyTorch.
💻 Code: github.com/python-engineer/sn...
✏️ Course developed by Python Engineer. Check out his KZfaq channel: / @patloeber
🎨 Art by Rachel: rachel.likespizza.com/
⭐️ Course Contents ⭐️
⌨️ (0:00:00) Part 1: Basics of Reinforcement Learning and Deep Q Learning
⌨️ (0:17:22) Part 2: Setup environment and implement snake game
⌨️ (0:40:07) Part 3: Implement agent to control game
⌨️ (1:10:59) Part 4: Create and train neural network
🎉 Thanks to our Champion and Sponsor supporters:
👾 Raymond Odero
👾 Agustín Kussrow
👾 aldo ferretti
👾 Otis Morgan
👾 DeezMaster
--
Learn to code for free and get a developer job: www.freecodecamp.org
Read hundreds of articles on programming: freecodecamp.org/news

Пікірлер: 236

@edwintjoa6099 Жыл бұрын

Thanks for the awesome video! Really fun to see agent improving during the games.

@HRDmonk Жыл бұрын

Thank you so much for this tutorial. I have always wanted to introduce snake with ML. and now looking forward to learning more about pytorch

@willchantemduang5871 2 жыл бұрын

I’ve always wanted to do this, thanks alot for the tutorial

@adrian46647 4 ай бұрын

Awesome, so hard to find that type of explanation of dqn. All clear, great balance between theory and coding part for beginners in rl.

2 жыл бұрын

Always very high quality videos when I just need it :) Thanks a lot! Looking forward to finish the video 😎

@dabyttamusic7308 Жыл бұрын

Super great video! I run it and after more than 2000 games it seems that the average score reached a plateau of 30. It Avoids boundaries very well. But it hit itself when the tail is too long. But this movement should be predicted before.

@user-fb8tr3kb8j 9 ай бұрын

To avoid the self collision we have to add snake parts positions to the game state somehow, in the current model it doesn't know where's the body so it will never learn how to avoid it

@yavarjn2055 3 ай бұрын

Lot of the code needs more explanation. There is a disconnect from theory and implementation. We copy all the parameters to this and to that without understanding why. The memory part is not well explained.

@30DaysMonkMode-ft1kf 3 ай бұрын

Yeah. He isn't explaining, just repeating what he is writing.

@mihailmihaylov988 Жыл бұрын

A nice video. My only critique is that the presenter kept writing code for almost an hour without running it. That's not giving a good example.

@user-de3oj1xw8u 6 ай бұрын

Need to be patient 😊

@DroopTheSnoot 5 ай бұрын

Debugging code should always be done every function or so

@pedroklain9375 4 ай бұрын

Bro is just too good at coding, he runs it in his head 😂

@mihailmihaylov988 4 ай бұрын

@@pedroklain9375 It's not about what he can do or cannot do. It's about what example he sets. After all, this is an instructional video.

@electricspeedruns6121 4 ай бұрын

friend ai and programming code is a bit different @@mihailmihaylov988

@khalidelgazzar 7 ай бұрын

Watched the first 4 mins and the game and the learning process is fantastic! 🎉 Will go on it with the rest hopefully soon.

@khalidelgazzar 7 ай бұрын

Part 2 - 17:24

@khalidelgazzar 7 ай бұрын

21:30

@Oncopoda 2 жыл бұрын

Excited to try this!

@veronicasalazarpereira9933 Жыл бұрын

I did it!!! Thanks for showing this.

@Hiyori___ Жыл бұрын

Very interesting tutorial. I was familiar with Snake code implementations, Tensorflow basics and RL principles and I could follow along. For some reason the plot doesn't appear, and I only get a small blank window where the plot should be. Probably some bug while typing the code down. Other than that everything works for me. Thank you very much!

@lostsoul4393 9 ай бұрын

just incase anyone else has this problem, you need at add plt.pause(.1) to the bottom of the plot function

@stillrunning9841 7 ай бұрын

Hey there, same happened for me! Did you find the error? I'm still trying to find it :)

@lillymoreau7721 6 ай бұрын

@@stillrunning9841 this code worked for me: import matplotlib.pyplot as plt plt.ion() def plot(scores, mean_scores): plt.clf() # Clear the current figure plt.title('Training...') plt.xlabel('Number of Games') plt.ylabel('Score') plt.plot(scores, label='Scores') plt.plot(mean_scores, label='Mean Scores') plt.ylim(ymin=0) if scores: plt.text(len(scores)-1, scores[-1], str(scores[-1])) if mean_scores: plt.text(len(mean_scores)-1, mean_scores[-1], str(mean_scores[-1])) plt.legend() plt.draw() plt.pause(0.1)

@radiatian4908 6 ай бұрын

Super late reply, but i had this issue and it turns out the video is missing two lines of code. In his files he also has plt.show(block=False) plt.pause(.1) as the last two lines. Hope this works for u too

@niklaskohler3242 2 ай бұрын

@@radiatian4908 Thank you very much!

@dollarfree531 2 жыл бұрын

Very cool video! Highly useful!

@kevas777 10 ай бұрын

nice algo, but how to solve a problem with self destroy, when the closest cell to move is "inside body circle". I think a state must be all field with each body part,head, food etc, but its endless states and all of them unique and it will never learns or may be i wrong?

@techarchsefa 4 ай бұрын

That is so smooth bro, thanks

@vikramganesan 2 жыл бұрын

how to perform UCB, optimal initial values and dynamic programming approaches in this model?

@sergiomollo 4 ай бұрын

Thanks for this video, I have solved all my doubts

@python0tutorial100 Жыл бұрын

Thanks for the video it was great but I have only one question if u can help after I run the agent and after a few try like 3rd game the game stuck and in the console suddenly start to count the number of game without the game really play

@pactube8833 2 жыл бұрын

Thanks for freeCodeCamp for making this possible.

@user-de3oj1xw8u 6 ай бұрын

A video as valuable as a playbook👍🏻👍🏻👍🏻

@johnpap4237 Ай бұрын

Perfect video!!Congrats! Is there any implementation with DDPG, PPO or SAC?

@Fr4nk4000 Жыл бұрын

Following this since I wanna make a pygame project of mine that I poured a lot of time into and have no idea what to make the game about play itself. Wish me luck.

@jaibhagat7441 4 ай бұрын

have you made something

@devnull711 Жыл бұрын

Incredible work, thank you Patrick! PS: It is very funny to spot the typos/bugs before you do :)

@Coding_Destini Жыл бұрын

What to install before creating an environment ,iam confused

@aliengineroglu8875 9 ай бұрын

Thank you for your great work. I couldn’t understand the equation you created in the video about the simplified Bellman equation, Q_new = reward + gamma * torch.max(model(next_state)). In this equation, model(nextstate) gives us a probabilistic action prediction. I couldn’t understand why we added one of the action probabilities to the reward. This is totaly different than the Bellman Equation. I would be very happy if someone could explain how the original Bellman equation was simplified in this way. Thanks in advance to everyone.

@alanlam6302 8 ай бұрын

Same here. Appreciate if someone can provide a reference to this implementation

@wangking7427 4 ай бұрын

The Liner_QNet model gives three actions' Q value vector . It's not the probablistic of each action but the QValue of each action. We take the action which has the highest QValue each step. The QValue in this tourtorial is the reward of the action.

@goldenmarketing3179 Жыл бұрын

Does this tutorial teach how to write "inference" by myself, not using library?

@filoautomata 7 ай бұрын

What about using GA to train the NN itself ? Will be a very interesting comparison no?

@stefanb4340 Жыл бұрын

This might be a stupid question, but how would one go about saving this trained model and accessing it for further training?

@brayanfernandes371 Жыл бұрын

you can save the weights

@sanghoututorial3878 Жыл бұрын

@@brayanfernandes371 how

@segovemoc4776 Жыл бұрын

@Biglyp I actually wanted to ask a similar question. I save the weights and load them through: self.model.load_state_dict(torch.load(model_saved_path)) and it works, but the problem is that the snake is then underperforming. It learns then much quicker than before, but still it is significantly worse than during training.

@segovemoc4776 Жыл бұрын

never mind. I found answer in the later comments - it is necessary to adjust the epsilon value to something very small after loading the model

@gmancz 11 ай бұрын

@@segovemoc4776 It works, but you need to set the Agent.n_of_games to something big value (eg.: 280). self.model.n_of_games = 280 As you remember, there is exploration and exploitation in the get_action() function. We are updating the epsilon value using the self.n_of_games variable and then get a random number between 0 and 200. This line makes your snake so bad.

@lukasgamedev 9 ай бұрын

Hello! Is there a way to save the state of the neural model? So I can load later a trained enemy AI, ready for being the player opponent? Thank you for the video!

@TanmayBhatgare 7 ай бұрын

i think u can you torch.save(model.state_dict(), 'rl_model.pth') to save model and model = YourModel() model.load_state_dict(torch.load('rl_model.pth')) to load it. Hope this helps.

@soraaoixxthebluesky Жыл бұрын

i just want to know after training, how can we load back a trained model?

@codingfan Жыл бұрын

Really interesting course !

@pyshine_official 2 жыл бұрын

Nice effort and seems like the convergence is achieved upto some extent!

@smkzachatac 9 ай бұрын

So, my array for state is apparently returning NoneType. Anyone know a fix?

@ismailmatrix1 2 жыл бұрын

What's your VSCode theme? It looks gooood

@vespervenom2343 2 жыл бұрын

I copied it code for code and it doesn’t work. It gives me no errors but just runs and then ends with no result🤦🏽‍♂️🤦🏽‍♂️🤦🏽‍♂️🤦🏽‍♂️🤦🏽‍♂️

@markkiryaflex 2 жыл бұрын

I think this project is awesome

@sunyog 2 жыл бұрын

What are the prerequisites of the video ?!

@eugenmaljas7237 Жыл бұрын

I want to modify the program: how do I make 4 outputs? I would like to integrate the snake length.

@arnavgupta8376 Жыл бұрын

I think you can do so if you change the line 20 ie 'self.model = Linear_QNet(11, 1024, 3)' change the 3 to 4. and in line 90 'final_move = [0,0,0]' add another zero. Maybe there are some more things you need to do.

@cesarortegonavacerrada9065 Жыл бұрын

Very good video. I need help with the homework. How can you avoid the loops?

@Miyuru_ Жыл бұрын

same problem it's getting stuck in the same loop

@hcc3904 Жыл бұрын

@@Miyuru_ overfitting... just add another condition point like if u dont eat apple in 10 seconds, minus -10 point

@manojkothwal3586 2 жыл бұрын

Brilliant 🔥🔥🔥🔥🔥🔥

@vinayakbhandage8319 2 жыл бұрын

You never fail to amaze us✌

@gattorwichar3984 2 жыл бұрын

This instructor is the best

@planetoday9169 Жыл бұрын

can someone help me how to create a multiple snakes?

@cadewzan 8 ай бұрын

Theres no need of waiting a lot of time to train, on the game script you can just change the varaible from 30 to 1000 so snake goes much more fast and trains intself on less time.

@MyArtsProduction 7 ай бұрын

He actually mentions that 1:38:00

@gukeskl3671 Жыл бұрын

How can I get this terminal? :o

@simpleepic 10 ай бұрын

Great tutorial

@sakshirokade6321 Жыл бұрын

WHICH ALGORITHM IS USED IN THIS

@Ragnarok540 2 жыл бұрын

My snake became self aware, any ideas to stop it from taking over the world?

@matthewwisdom426 2 жыл бұрын

😂😂😂😂😂 Mine has hacked Russia, it's now impersonating Putin and is about to start world war 3.

@sidheshwartiwari9834 2 жыл бұрын

🤣

@sidheshwartiwari9834 2 жыл бұрын

@@matthewwisdom426 lmao 🤣🤣 i am drying 😂😂

@carolinab9945 6 ай бұрын

Is it reinforcement learning even if you give some instructions about the movements?

@angelosorte5464 10 ай бұрын

Is there a limit to when it stops learning? I mean the quality of the intelligence will stay the same at some point, or will it improve even more and more after those 12 minutes? Thanks.

@damienlmoore 8 ай бұрын

At 34 mins, it breaks out of the game and starts gobbling up your filesystem. At 762, it breaks out of the machine and comes for you. 😅

@joekanaan2548 2 жыл бұрын

Hi I have a question. I followed along and wrote everything. When I run it in the terminal it works but the plot doesn't. It gives me an empty minimized white screen. If someone else experienced this please help.

@AlexandruTunschi1 2 жыл бұрын

There are 2 lines missing: plt.show(block=False) plt.pause(.1)

@RANDOM_DUD-qj3jd 6 ай бұрын

Where do these go?@@AlexandruTunschi1

@S3R43o3 2 жыл бұрын

So far so good... but how about load the old 'brain' if u exit the game and want resume later ? Oo iam bite confused tryd with state if os.path.exists('model/model.pth'): self.model.load_state_dict(torch.load('model/model.pth')) self.model.eval() in the Agent init. but that dosent work

@vikramganesan 2 жыл бұрын

did u find out a way to load the previously trained model?

@S3R43o3 2 жыл бұрын

@@vikramganesan nope sorry

@RORoMiguel Жыл бұрын

it seems like we need to remove the epsilon to load the model, otherwise it will keep doing random moves

@jeremiebouchard6950 Жыл бұрын

@@RORoMiguel You're right, it works for me :) thanks! first game : 33 score

@polllloigyv 11 ай бұрын

i think it doesn't work because you you wrote / instead of this\

@Miyuru_ Жыл бұрын

someone tell me how to save trained data?

@deepakpaira0123 4 ай бұрын

how to save the model...? where to know?

@Radu 2 жыл бұрын

Really nice!

@sidheshwartiwari9834 2 жыл бұрын

Look who is here, none other than mighty Radu sir🙏🏻

@Radu 2 жыл бұрын

@@sidheshwartiwari9834 haha :-) funny!

@feliuswyvern7189 Жыл бұрын

so i had the program run over 2000 games and it couldn't get past the 80s mark. how would i get it to improve past that?

@taraskhan475 11 ай бұрын

there is no way to improve it unless you got coding knowledge

@fitybux4664 Жыл бұрын

<a href="#" class="seekto" data-time="2293">38:13</a> "It has to be [0,0,1]"

@ckelsel8198 Жыл бұрын

thank you very much.

@pranjalshukla8096 Жыл бұрын

<a href="#" class="seekto" data-time="1138">18:58</a> for me it is not showing as (pygame_env) ... .... i am a windows user (edit) it worked for I had to open a CMD outside of VS CODE and then follow the steps of conda pygame_env now I am getting "(pygame_env) C:\Users\........."

@bobjeff6779 4 ай бұрын

how did you get it to say pygame_env

@pranjalshukla8096 4 ай бұрын

@@bobjeff6779 when you are creating a venv (virtual environment) you can name it so to make venv with a different name you can do that by python -m venv myproject

@murtazabadshah8747 3 ай бұрын

Everyone's commenting that its an excellent video but IMO this tutorial is awe-full! The instructor does not explain the process, hes all over the place going back and forth and just rushes through the concepts. If you want to blindly follow an online tutorial watch this video, if you want to actually learn the concept I would look somewhere else....

@30DaysMonkMode-ft1kf 3 ай бұрын

Exactly.

@ShtBall5 2 ай бұрын

any video recommendations ?

@30DaysMonkMode-ft1kf 2 ай бұрын

@@ShtBall5 You can check this video: "Learn Pytorch for deep learning in a day. Literally."

@murtazabadshah8747 2 ай бұрын

@@ShtBall5 Yeah I ended up watching this video that helped me a lot to understand the basic concept of the Q-learning algorithm and the basic concept of how the state and agent work kzfaq.info/get/bejne/pteorKeGmMzYk6c.html&ab_channel=Bits%26Neurons

@SahdevSingh-io3qp 2 ай бұрын

Some WHERE else??

@dexterbarrot5226 Жыл бұрын

What text editor

@spyrav 2 жыл бұрын

thanks!

@enginturabk817 15 күн бұрын

In <a href="#" class="seekto" data-time="2310">38:30</a> part comment line of 138 is # right turn r -> d -> l -> u but line 141 is # left turn r->u->l->d should it be like this: # left turn l -> u -> r -> d ? or (written) r->u->l->d Please somebody explain it to me I do not understand

@GenkiKuri 3 ай бұрын

Awesome!!

@luthermillamuculadosreisec3844 Жыл бұрын

Thank you

@canerunafraid9491 10 ай бұрын

The snake moves smoothly, but when it hits the first wall, the interface closes idk :/

@Lenslyfe 5 ай бұрын

It's not meant to wrap. That's the correct behaviour.

@Sibixpur 2 ай бұрын

you can understand the code given right??

@user-zo4cx8yi3g 6 ай бұрын

thanks, intresting!

@VietnameseBoii 8 ай бұрын

I use your code and train it with speed 60000 (just modify the "game.py" file)

@joe_hoeller_chicago 4 ай бұрын

Cool vid 😊

@kabrailchamoun8974 2 жыл бұрын

Could you bring us a crash course of blender (3d modelling program)?

@slamsandwich19 Жыл бұрын

Python has nothing to do with 3d modeling though

@muhammadnaufil5237 Жыл бұрын

@@slamsandwich19 it is. Blender has a python library called pyblender. I am looking forward to a good blender tutorial and how to integrate this with python

@googlyeyes3237 2 жыл бұрын

Can we speed up the game soo fast so ai can play faster and learn faster. Basically it can finish 100 games in a minute or two.

@sidheshwartiwari9834 2 жыл бұрын

Absolutely, you just have to increase the render rate. Good question though.

@tlefevre2918 9 ай бұрын

it does that when it is drawing on the chart, I speed it up by making it only update the chart every 100 games.@@guillaumelotis530

@keethesh2270 9 ай бұрын

Just change the tick speed to make it a higher one. It is at the top of the script and is called "SPEED"

@usernumber334 2 жыл бұрын

true

@enginturabk817 15 күн бұрын

<a href="#" class="seekto" data-time="3519">58:39</a> I dont understand #danger straight right left and 4 line codes :( please help me

@marcbringas1006 2 жыл бұрын

where is he typing the commands? is this a CMD?

@science_horizonts 2 жыл бұрын

Do you mean the source code of the program or the command to run the program If to start, then yes, this is the command line(CMD), but you need to go through it to the folder where you have the project, CMD is somewhat similar to the explorer that goes through the folders

@mtijohn9274 2 жыл бұрын

yes

@fizipcfx 2 жыл бұрын

why not just give the ai full board information as the state

@piyushsrivastava7636 2 жыл бұрын

Can we create AI bot to play dice game?

@sidheshwartiwari9834 2 жыл бұрын

Well, it's already done. Almost every Ludo game has a bot. In fact you don't even need AI and ML for it as the conditions are so few that it can be hard-coded

@RANDOM_DUD-qj3jd 6 ай бұрын

no windows opened when I ran. not even any errors. how do i fix it?

@bobjeff6779 4 ай бұрын

did u get it

@philtoa334 2 жыл бұрын

Crazy.

@hihi-kv1hu Ай бұрын

How many Qvalues do you have?

@sureshsingh9880 2 ай бұрын

Great content,,,❤❤❤

@nescreation 2 жыл бұрын

Wow

@yarinh8417 7 ай бұрын

somoone knows hot to imporve the snake so he does not collide with itself and loop over itself?

@aliabbas-xs6qm 7 ай бұрын

Discard the AI an use a hamilton cycle kind liek how codebullet did in his video, It wont be an AI but it wont loop or kill itself

@cryptomoney6901 2 жыл бұрын

There is a small bug where the snake will eat itself if you go in the opposite direction that its going in. To prevent this from happening change this line of code: if event.type == pygame.KEYDOWN: if event.key == pygame.K_LEFT : self.direction = Direction.LEFT elif event.key == pygame.K_RIGHT : self.direction = Direction.RIGHT elif event.key == pygame.K_UP: self.direction = Direction.UP elif event.key == pygame.K_DOWN : self.direction = Direction.DOWN to: if event.type == pygame.KEYDOWN: # 2nd conditional to prevent snake from eating itself! if event.key == pygame.K_LEFT and self.direction != Direction.RIGHT: self.direction = Direction.LEFT elif event.key == pygame.K_RIGHT and self.direction != Direction.LEFT: self.direction = Direction.RIGHT elif event.key == pygame.K_UP and self.direction != Direction.DOWN: self.direction = Direction.UP elif event.key == pygame.K_DOWN and self.direction != Direction.UP: self.direction = Direction.DOWN

@slamsandwich19 Жыл бұрын

He fixed that later in the video

@ashishbehal5903 22 күн бұрын

If someone can help me understand <a href="#" class="seekto" data-time="91">1:31</a>:34 it would be appreciated thank you

@omidelahi6196 Жыл бұрын

I got this error: ValueError: not enough values to unpack (expected 3, got 2) for line 118 in agent.py

@ericb9056 Жыл бұрын

I got that as well I forgot to type out the first value of the associated return in the game file. check that you have, return reward, game_over, self.score at around line 82 and 96

@omidelahi6196 Жыл бұрын

@@ericb9056 Thanks dude

@xccds Жыл бұрын

very great course! thanks! A little question is , in class QTrainer maybe target = pred.clone().detach() ?

@superfact8751 Жыл бұрын

hi bro kaha se ho

@godswillhycinth9809 2 жыл бұрын

I'm a JavaScript Developer that knows nothing about python, but I must say I'm jelous 😭😭. This is really cool

@Ragnarok540 2 жыл бұрын

Python is way easier than JavaScript. Give it a try.

@Loug522 2 жыл бұрын

Yeah, if you already know other languages(especially OOP) then Python shouldn't take more than a few hours to lean most if not all of its basics.

@delete7316 2 жыл бұрын

I’m jealous of you. Python is super easy to learn.

@bipinmaharjan4090 2 жыл бұрын

Found new hobby

@serverautism2268 Жыл бұрын

after 6000 games it averages at 33.4

@pepeCastillo 2 жыл бұрын

Jobs Is alive?

@Agesilas2 7 ай бұрын

set video speed at x1.25 or x1.5, thank me later.

@chandramoulidasari3946 2 жыл бұрын

What was the theme?

@Peaceful-er4vf 2 жыл бұрын

@mohammedismail6872 Жыл бұрын

dumb question ik, but what is the box he is usint to code, is it CMD, or windows power shell etc ?

@melvinliew2426 Жыл бұрын

It might be cmd,but for me is anaconda prompt

@washyb 4 ай бұрын

hes using some CMD on a Mac, I think.

@enginturabk817 14 күн бұрын

<a href="#" class="seekto" data-time="67">1:07</a>:18 why did you write 80 ?????????????????????????? explain please

@user-wz2xn4ni2r Жыл бұрын

I want it to play minesweeper

@aoebb5021 Жыл бұрын

@__________________________6910 2 жыл бұрын

omg patrick wow

@patloeber 2 жыл бұрын

yeah :)

@aoebb5021 Жыл бұрын