Thanks for watching! Some additional details not explained in the video, which might help to better understand the irregularities observed: - I didn't show all the AI inputs in the video to keep things simple. In reality it has access to more information, such as x,y,z velocity, velocity rates, roll-pitch-yaw rates, etc. But maybe it's still missing some crucial information, it's hard to know. - The irregularities observed are not due to hardware or framerate issues. All the things I'm showing in the video are made with a tool called TMinterface. This tool allows me to inject action commands into the game at precise timings, in a way that is 100% repeatable. The same sequence of action on the same map will aways lead to the same outcome, even on a different computer. It's completely deterministic. And it's used a lot for Tool Assisted Speedrun (TAS), you can find many examples of that on youtube. I have a bit of extra footage that I couldn't fit into this long video. I plan to post some of this extra content on my Patreon in the next weeks. Any support there is a great motivation to keep making these videos :) patreon.com/Yoshtm

I came here to see AI destroy humans on Trackmania, but I got a 37 minute essay on how pipes in Trackmania are an exercise in chaos theory instead. 10/10

So you’re saying my world record fails are actually the butterfly effect’s fault…

The editing, the music choice, the compactness of the information while still remaining coherent, the pacing. This video is a masterpiece! Even small things like using the butterfly-shaped pinhole transition when bringing up that topic for the first-time, or how the hue of the cars match up with what generation they're from, so many little details that I could write up my own essay detailing every little thing you put extra effort into. I was enraptured the whole time. Bravo!

I love your video style and how you're displaying the learning process. Thanks for the great video!

PhD Student in Deep RL here. The behavior at the end seems mathematically chaotic and might lead you to the conclusion that you cannot predict deterministically the behavior of the car. However, that does not mean you can't improve the performance by a long shot with a few simple tricks. You are doing model-free reinforcement learning, which basically means that you don't need to predict what is going to happen exactly (and extremely hard here), you just need to figure out what actions are the best in this situation. In most environments, this is actually much easier to do (and a reason why learning an accurate model is often harder than just straight up optimizing for performance with model-free). Second problem is that you are using an RL algorithm and RL assumes you are in a Markov Devision Process, ie that you have full observability of the world, ie that the probability of taking the next action ONLY depends on the current information that you have, and not the past, ie that adding information about the past does NOT help ou make better decisions. HOWEVER, you are NOT in an MDP (Markov Devision Process) here since you lack critical information about the state to take decision, as many pointed out. In practice, what you can do will most certainly improve performance: - either manually add all the missing info (speed, rotations, ...) - or more simply, create a concatenated vector of the previous N past states and inputs and use that as input to your NN (LSTM/RNN works too but is more complex and often not necessary when we are dealing with a few steps of history). - Use more layers (deeper network = more complex function can be learned by the network) - Sticky/random actions: introduce manually randomness in the action of the agent. For example repeat randomly the past action with a small probability. Do NOT decrease this probability with time. What this means is that the agent will have to learn the whole time in a now stochastic (ie random) environment and will have no choice but to cope with the inherent randomness that you added, making it largely more robust than an deterministic agent. This is a common flaw of fully deterministic agent in these kind of environment btw. - Look at extensions like Max Entropy RL (Soft Actor Critic for instance), where you both maximize reward and entropy of the policy. Some papers proved that it makes the learned policy more robust to out-Of-Distribution perturbations, ie perturbations that were never seen during training. In your case, this will help the agent lean "recovering behaviors" to recover from deviations caused by physics bugs/randomness/whatever. - Max Entropy RL will also help with exploration a lot, which might help the AI be more creative in finding solutions. - Look at other tricks that improve performance "for free". I don't know how much you implemented already, but a basic start is looking at Rainbow DQN, PPO.... But you are already using it probably..?

The song that played when you showed Wirtual's WR was just peak video making lol

What an insanely good video! I haven't even played Trackmania in many years... but the way you explain the details and visualize everything just makes this soooo interesting! 😌👏👏👏

Incredible video, one of the best I’ve seen in a long time, thank you !

22:48 "THE AI GOT THIS RUN" omg even the song, that's great man

Roboticist here. Some regions of configuration space are more chaotic than others; the demo you did where you perturbed a car that was about to fall off demonstrated a smooth increase in success rate as you moved the perturbation further away from the point at which outcomes diverged, suggesting that the failure was in a fairly narrow chaotic _region_ and moving out of that region made the system more stable and enabled more consistent success. This is what unnamed did to complete the jumps track: Found a strategy that consistently avoided highly-divergent regions of configuration space. One problem is that, if the display of the inputs is representative, the AI doesn't have the right sensors. Specifically, the AI has its position and speed but not its _velocity_; it doesn't know how much of its speedo reading is in each component of down the track vs. across the track vs. away from the track. More critically, it also doesn't have roll-pitch-yaw rates. It only knows its current orientation, not the rate at which its orientation is changing. I suspect that the car also needs to know the difference between a simple pipe elbow, a tee, and a cross. Even if the extra geometry isn't in the direction of its travel it'll almost certainly still interact. Finally, consider increasing the numerical precision of your network. This is one of those cases where running floats vs. doubles might actually matter.

That ending sequence was an absolute joy. Prefect syncing with the music

This is one of the best videos I've ever seen. Amazing work!

28:50 “Now let’s have the AI drive upside down” had me rolling. For a hot second I really thought you were about to blow my mind once again 🤣

28:40 was a comedic masterpiece, the editing, the pacing, incroyable!

Apart from the excellent research, I'd just like to comment on how amazing the editing was for this! That kept it interesting AND funny!

Amazing production, your videos are always such a treat!

Yosh: "I am going to lock your breaks and acceleration so you cant slow down" AI: "Pfft check this out"

Your analysis about chaos theory is exactly what is going on. And TrackMania's chaotic physics are very familiar to me from another TrackMania project. The best way to control that chaos is by going slower, as the deviations caused by the bugs are less severe at lower speeds. Perhaps giving less reward for speed and more reward by having a high chance of getting far would help with consistency.

Possibly the best video I have ever watched on KZfaq. Well done!

Man, you have the best vidios on AI reinforcement learning on the whole internet, kudos!

Waiting for all the Trackmania KZfaqrs to react to this.

I love how the AI on the 65km pipe track looks like it's just blissfully skipping along while the human records look like they're terrified. it's so silly.

Fantastic study! Thank you so much for sharing.

What an extremely high production value we have here. The edit, the music, the time and dedication oooozzzzee excellence in the art. Well done to all involved. This is a patrion worthy production for sure! Keep it up and hope to see more at this level. And yes it strengthens understandings in both chaos theory, and agnosticism lol. Mahalo

Wow, this was not only informative and entertaining but a cinematic masterpiece with perfect pacing and music selection and everything, I didn’t even realise it was 40 minutes long until after I finished watching it!

Mathematician here, I fully agree with your chaos theoretic conclusions, and I think the pipe experiment at the end strongly supports this (as you realised). Why not give the ai control after a random amount of time has passed at the beginning? I think it is really interesting, I would like to see if the ai could somehow abuse the chaos theoretic nature of the situation - I remember a video about a Pokémon ai that figured out some elementary RNG manipulation so that it wouldn’t get grass encounters :D If you want to talk actual mathematics, then I think perhaps the KZfaq comment section is inadequate - I cant answer every question either, I only took one chaos theory class as a grad student, but perhaps we can try to understand some things.

Thank you for the visuals, they are incredible! I understand AI and the process of learning a lot better now!

Dude this was an amazing video bro well done

your visualizations are just amazing, thank you for all the editing!

I like how yosh's attitude towards the learning model went from "We shouldn't have to sacrifice consistency for speed" to "Fuck it we ball"

Amazing video! I don't even watch or play track mania, but you managed to maintain my attention for the entire duration!!!

Incroyable ta video, ca ta pris beaucoup de temps comme tu la dit bravo pour ton travail de fou

Thanks for using my map :D I have heard people saying that Trackmania is a deterministic game hundreds of times. I think from now on I will prefer to use the word "chaotic" :D

16:32 and there he is, introducing brownian motion in a virtual world to isolate entropy. Good job!

one of the most entertaining videos ive seen. good job!

I’m not a TrackMania player but I love watching TrackMania videos. I also love game AI vids, and this video really did it for me. Being fascinated by chaos theory I’ve been screaming at my screen throughout the whole video ”bro it’s chaos theory”, and then you even mention it yourself. Top tier content, it’s what I’m here for. You just gained a new subscriber 😄

Unironically one of the best track mania / AI beats things video I have ever seen, the visual representation of everything and your clear explanations made this so satisfying to watch. Like the showcase you did when explaining the randomness of tiny input changes was so well done and it is really clear how much time you put into this topic and video. Also your humor ist just amazing lmao

The butterfly transistion at 17:27 is a pure masterpiece! The many small things u do while editing dosen't get enough appreceation! Congrats on a beautiful video!

Best video seen this year. i really enjoyed it thx

This is the pure essence of youtube, why this platform is amazing. This video is an absolute master piece, I am forever impressed, congrats

yooo such an honor that you used my map for the 3rd part ♥really enjoying your AI videos :) keep up the great work!

This video and others are so well-made, Yosh! Bro WTH

Amazing how complex the simulation in Trackmania is... Never knowed. And your work ist mind boggling to me... Purely wonderful work.

I didn't expect there to be micro fluctuations in a still car, but its equally fascinating as it is frustrating to learn the game is so sensitive that milliseconds can, in niche scenarios, determine your success. It may not be truly random, but it's close enough that you can't account for it.

I loved how it just randomly turned to a Wirtual documentary for a while at 21:55, even the Norwegian flag is here.

Amazing video! I love the editing

RL in trackmania make it very visually interesting. Thanks for creating video of this quality followed by all your thoughts over the projects.

Your humor with "the song" and "the AI got this run" is just too damn much man. On top of being a cinematic masterpiece, it's also funny as hell. Insane video, genuinely hope you blow up beyond belief.

"In the Hall of the Mountain King" playing while all the cars rush to their demise like a cascade, is so fitting. Your videos are so interesting and it was worth every second spent.

I’ve never played track mania. Only watched a handful of these videos and all I have to say is WOW! this video was amazing! Please keep making them.

Incredible video. You have gained a fan. And I love maths and AI. So can't wait to start studying it

I have no idea what's going on but it was really satisfying watching huge cascades of race cars falling off pipes, thank you!

Even deterministically chaotic systems can be controlled, in practice it is when Lyapunov exponent is small and the agent has quick enough reaction time (and steering capability) to keep control. But the touch at 34:06 looks similar to a landing bug. A sudden deviation from usual behavior. What you can do is to add another (part of the overall) deep neural network to act as a predictor. That would detect such "buggy surprises", and you can focus the score on surviving such surprises. As to making the competition with humans more fair, what you could do is to make AI commit to its actions some time (say 0.3 seconds) into future. This would lower its control, so it would have to drive slower in order not to lose control. (Edited typos.)

Thank you for including the music track list at the end, so many bangers in this!!!!!

Amazing work. This was a great watch.

25:55 was the funniest part, no questions asked. Long but great set-up to a satisfying slapstick punchline.

This is really stellar work. Not just the AIs but the videos. I'd equate these with the likes of Summoning Salt who are just very good at not only doing stellar work themselves (speed runs in his case) but also present it in an enjoyable and captivating fashion. I have not binged watched your material, but from the videos I have seen this is some of your best work. Keep em coming.

Thank you so much for this. I really enjoyed it.

Thanks for explaining the theory! This was fun watch

OMG i was litreally watching the previous ai video and this JUST DROPPED ive been BLESSED. Actually huge timing

What a wondeful video, the quality, the effort, the scripting, even the joke about driving turtle, totally got me, piece of art!

Vidéo très pédagogique, j'aime bien ! Merci pour ton travail

Chaque vidéo je suis comme un dingue, alors que je ne joue pas au jeu de base. Continue c'est très fort 💪

Wow. The way you paint the picture between about 17:00 and 20:00 is so captivating, wonderfully done!

Great video. Ive totally watched it after 44sec release

The music editing at the end was excellent!

what an informative and interesting video. Thanks !

This definitely is your best video ever! I've enjoyed the previous ones a lot, but this is a masterpiece! Keep going at it; the humor, the theorycrafting, your thought processes... it's all just one giant jigsaw that fits together perfectly.

babe wake up, new trackmania ai video

Amazing channel, great content!

Intéressant, intriguant, étonnant, fascinant, et même émouvant... Quel travail d'immense qualité, merci de tout cœur !

Software Developer with focus on ML here. Just from 32:03 you can see a lot of variance in the game engine physics calculations. This is most likely due to floating point precision error over time. To a human player, this variance isn't preceptable. Eg. one hundred thousandth of a velocity change looks the same in reality, but the different input may cause the AI to make a different decision. The AI may be over fitting into the region of noise in the signal. To make the AI more consistent and human like, I think you need to actually DECREASE the precision of these values fed into the model. That should help reduce the chaos being injected into the system. This is a balance because reducing the precision too far is bad too. A simple trick to try is to convert inputs to int8 or int16. That will give lower precision input and perform faster with aimpler math calculations. Or you can just truncate the float to the first 4-5 significant digits.

Every time I see these videos appear, I can't help but watch. It's super interesting to see the steps required to train the model and the steps you take as you iterate. On top of that, the way you've visualised all of the attempts and with the use of colour to demonstrate how close the AI is to ideal for the scenario is just pure eye-candy. I love it!

Great video, keep up the good work :)

Great video! Love the little homage to Wirtual with the sound track and "and then the AI got this run".

This was such a masterpiece video! I’ve been following you for a long time but it is incredible on how far you have come. I enjoyed every minute of this video. Thanks. ❤

A race without 🥕 = chaos Congrat's Yosh ! Love watching this little car grow 🧡

Brilliant work. Astounding.

Merci / Thanks for your work on this video. ❤

Fellow NN dabbler here, I have a few thoughts / suggestions: There's a degree of unpredictability from floating point bit precision and, as it's obvious to all, the system is inherently unstable. But I don't think this is insurmountable. Firstly, there's a limit to what you can achieve with just two hidden layers and no temporal context. The lack of layers will translate into a lack of abstraction. All you'll get is "this high, that low" kind of reaction. Worse still, the NN is meeting decisions based on the instantaneous state when the most important thing is momentum which has a time component. Therefore you've given it an unsolvable problem, like trying to play chess without knowing where your opponent's pieces are. You can test this hypothesis by calculating angular and linear momentum and feeding it in as an input. If I'm right, you'll see an instant improvement. Much better though is to allow access to the past. Try a simple recurrent NN first to see if it improves matters. Although you'll have to watch the expense. Finally, just like the pipe it rides, the instability of the system becomes critical when you approach loss of control circumstances. Right now that's not being trained for, the change in the input just alters which corner fails. So the loss is not factored into the reward meeting that you stay in the same position in latent space. You need to find the bell curve before the failure is most recoverable. Say it's 3 secs before the fall. Find a failure point and mass spawn there with a random time variable of say 0.5 seconds. Train the AI that, on average, has the least falls. Fundamentally though, an input / reaction AI is insufficient. I would probably look at using an attention vector where I'd look to get it to recognise certain situations and contextualise its response. Just a thought. Very fun vid btw!

"I don't want to dream of this pipes anymore" 😂 Love the video

such a high quality and well made video damn

La vidéo est très clean, le taff est fou, c'est intéressant, fun, chapeau, c'ets du très beau taff

You make the bestvideos on Reinforcement Learning! They inspired me a lot to want to learn how to do this, keep up the great work :)

Yo man was just gonna say that you inspired me to dive deeper into reinforcement learning for my master thesis after seeing your amazing work. Keep it up!❤

The editing is actually amazing

Your videos are so good that I make popcorn and then watch them on the living room TV 😊 they are pure poetry 🤓

Incredible editing, story telling, and informative! Well done!!

The tiny turn to make a "butterfly effect" was very clever man.

Editing this video must have been insane. Nice job!

Fun and subtle little introduction to computer science, chaos theory and butterfly effect. Good experiment!

I never comment on KZfaq but this was an absolutely amazing video! Content and script/editing were both perfect. Thank you for making it!!

First video of yours I've seen, and it's an instant subscribe. I know nothing about Trackmania and I'm not much of a gamer generally, but this was absolutely fascinating, and the quality of your shots and editing is outstanding.

this has to be one of the best videos ive ever watched!

incredible videos...! it looks like it's some kind of new art form..

You've completely nailed the editing, pacing and comedy in this video. Keep it up, and thank you for such a masterpiece!

7:58 Feels like watching Wiggler from Super Mario walking on a pipe lol

This video is so insanely well made. Being able to create AIs like this is no small feat, and the fact that chaos theory crops up from this is a testament to the bleeding-edge nature of this technology. I say that with absolute seriousness too.

This was fascinating in terms of both content and presentation. I studied chaos theory in detail through graduate school and think it's amazing that we've progressed so much in recent years that we're able to train models to control themselves with high degrees of freedom to achieve complicated objectives "with ease" like this. Your presentation was really well done and I'm frankly impressed that you introduced the ideas to us with such clarity. - I saw a comment mentioning precision of the network, where floats vs. doubles might matter. I might even raise that and say that using quadruple precision may be helpful, especially for very long tracks where the outcome of control decisions early on have more room to accumulate and "stabilize" before getting jostled around by typical double machine precision noise. - It's possible that Trackmania indeed has a deterministic physics engine, but the result of a floating point operation depends on computer hardware. I think it would be interesting to run the same models on different computers and race their best products against each other to see if there are any significant differences due to pure numerical error in carrying out the game's computations. This might also be able to prescribe a quantitative number telling us how chaotic a segment of the track actually is which could be interesting to know as well. For example, to be able to tell the model that *this* segment of track is actually easy and there's nothing to explore, or if in *this* segment it should expand on the way it learns by 1) using much finer controls to perfect the local extremum it has found, AND 2) using much broader controls to explore the control space more rapidly without burrowing itself in one particular local extremum. - The fact that the state fluctuates so wildly (to the 5th digit) when no controls are given seems to me like it implies that the state is also fluctuating while the car is being driven. Perturbations to the state in regular chaotic systems are normally way smaller (at least 100 or 1000 times less), but the fact that it's so large and is always changing (with zero mean) makes this seem like a stochastic system to me.