imagine telling someone from 1902 that 2024 we dont have flying cars but instead we have simulated brains playing goat simulations

This. This is big. SIMA + Figure01 + ISAAC + Devin ... We almost have all the pieces of exodia. Combining these tools would be an insane accelerator: self replicating self evolving self evaluating AI agents that self train in simulation at 1000x+ real time across thousands of instanced virtual realities that then inform the robot AI agent how to execute tasks in the real world.

Calling it. Google earth is being converted to a playable game right now.

Excellent paper walkthrough Wes. A key point for people to understand is that because this paper shows the AI completing ~10s time-horizon tasks, you can use a multi-agent architecture for recursive hierarchical planning (i.e. breaking down a quest into small tasks of 10seconds each) and thus the AI can play for longer to complete quests and even potentially finish the game. This is what deepmind are already working on as is Jim Fan at his new GEAR lab at Nvidia. Language as a first-class citizen combined tree search (as Demis Hassabis stated clearly) along with training on videogames leads to AGI. At the very least there is one clear path to AGI, and of course there are likely others to. Thank you Wes for keeping us up to date. I'm a senior data scientist and don't always have time to read all these papers in depth, so your breakdowns help a lot.

Video game from a prompt. I'm ready.

Audio feedback is a huge component of video games and real life. They need to include audio into the generalization of the model. Especially audio in a spacial format. They should also record all the training data input/output goals+results and run "reflective" training on it with the goal of getting the AI to evaluate its own performance and theorize ways to improve. Essentially get it to learn how to generalize reasoning and logic, testing and evaluation, etc. Those would be crucial to getting generalized autonomous robot agents in the real world

Wes! THANK YOU for translating these esoteric thinking processes into baby steps (with graphs) to help us come close to understanding what is happening and what we can expect SOON! It's essential that plain folks like me grasp the enormity of change that is inexorably changing the way we live and the ways we must cooperate in order to survive. YOU are the 'explainers' and 'connectors' that give me hope that we MAY survive. I have a whole crowd of children, grandchildren and great-grandchildren whom I hope will b able to enjoy long lives on this planet. Thank you!

Goodbye "single" player mode in Minecraft, I won't feel alone in an empty world anymore.

Single player games is the future

There's a huge market waiting for intelligent conversational NPC's and companions in GTA5, Cyberpunk 2077, Skyrim etc... if they can get it done.

I think it's fair to say it's been a ***SHOCKING*** week

If we eventually reach the point where processing A.I. barely consumes any power and resources, having adapting A.I. for singleplayer games would be really cool.

17:05 "...and some you're a goat" You're the GOAT 🐐Wes.

It's simple, OpenAI, Google etc. need to start *PAYING* us for our input. People can generate as much data as they need.

"The quick brown fox jumps over the lazy dog" is also a great word to use.

What do you use for screen capture?

So excited for the future of gaming. Once all this stuff is fleshed out, games will be very dynamic

Timing 🎉🎉🎉❤

Have they published the full videos of the AI gameplay? Or just those brief gif-like clips on that link?

This video explores DeepMind's new AI agent, SEMA, which can follow natural language instructions to complete tasks in various video games. What's unique is that SEMA interacts with these games in real-time using keyboard and mouse controls, just like a human player. This approach allows the AI to learn and generalize across different environments, potentially paving the way for more adaptable and capable robots in the future. While SEMA's success rate varies across different games, its ability to understand and execute open-ended language commands in real-time marks a significant step forward in AI research. However, the video glosses over the ethical and societal implications of such advanced AI, particularly its potential impact on competitive gaming and the future of human work.

People will finally be able to complete all the games in their Steam collections :D

The key to AGI is training AI models to evaluate themselves internally and expose modal network structures that can be reused and standardized. Basically extraction of emergent capabilities. We are reaching the point where multi-modal, single network "black box" models will be less favorable than modular models with greater understanding of internal workings. Split vision away from text away from audio, but in a way that can interface with a core AI logic unit. Thanks to these virtual training platforms (aka video games) this can happen insanely fast if they focus on it.

I think training such an AI on VR games would massively improve the data it can learn from. If I imagine a game, that puts the player in a large room and a lot of different objects are constantly pushed around by fans randomly activating and your task is simply to grab the object that is currently displayed in the UI Questlog - we could really get the training going. Train speed and dexterity. All while being able to look at the actual movement of the AI moving the controllers, providing evaluation data to improve the hardware it's using to train itself as a prototype.

The first person simulations are indeed a great training environment and it is likely how one of the key creators of Doom, John Carmack, is now joined with Dr. Richard Sutton at the Alberta Machine Intelligence Institute (AMII) to help build and write "the final chapter" on AGI in the RL book. AMII is the world leader in RL development, as Sutton (along with Barto) are the original authors of RL (and they reference Klopf who was funded by the USAF. Klopf wrote the book "The Hedonistic Neuron").

Yes!! Let's train our AI's on Call of Duty! What could go wrong?

I'm just thinking how useful this would have been doing across the milky way galaxy travel in Elite Dangerous autonomous. Repeating the jump and re-fueling and so on adapting to what's interesting to scan and so on. If it holds up what it aims for it could be very interesting. Helping in building missions through the ArmA 3 editor based on my instructions etc. The possibilities are endless...

There is so much data that isn’t easily accessible on the internet. So many preinternet photos and home videos, that now have value.

Being able to learn by itself to play doom by a combination of reading docs on internet and watching videos by others + RL starting with: 1. Doom, then uses transfer learning to learn Duke nukem 3D, Hexen, Heretic 2. Then Half-Life and use transfer learning to learn all it's common mods 3. Fight night, uses transfer learning to learn eventually all other EA fighting games. Would be cool for it to use a climbing ladder approach like this where it starts with simple games and then hierarchically learns more difficult games. I think the combination of Reinforcement learning, computer vision, and step-by-step plan writing + reflection could be a good mix. Now that being said I am totally a layman here with minimal knowledge. But the idea of mastering simple concrete tasks and then moving up gradually to more and more challenging goals gives a semi tangible way to measure progress.

It would be cool to have a game, where you were the leader of a group. And you could actually tell your team what to do, guide them... and they acted like real players. You could, in essence, lead an army with generals and have them act accordingly.

Ece online is literally the perfect step for this.

every time i hear the word "Agent" i think about Agent Smith from the Matrix though. can we use another word instead?

Next level grinding bot for mmorpgs

This is both amazing and terrifying!

Man wish I could test this. We need to throw it into a Battle Royale game, like PUBG. The large open space maps and the ever changing nature of the Zone, and enemy players, and interacting with Human squad mates, sounds like a really really interesting challenge for AI. Can it learn to Loot well, survive to the final circle, work together with the team mates on its squad, eliminate enemy players and secure the chicken dinner at the end game? What's interesting about this is the AI playing like a human, Games already have Bots/CPU controlled opponents but they aren't designed or function to play like a real human. This Agent can actually visualize the game and react and input like a human can that's remarkable.

Im watching this because it’a not “shocking”. Thank you Wes.

This is the stuff I love. You can't have AGI without generalizability. I think if we're gonna develop it, it will come from something like this.

I'm curious about how "models that are good at 'everything'" are organized. How different are they from our brains? Is everything tightly integrated, or no?

Thanks Wes . As long as a person knows what they are trying to do then the LLM will hone in on the language needed. So in some ways individual creative skills will need to be honed so that you know what you are trying to create. That takes real experience and real knowledge/education about styles and possibilities and human behaviours and self knowledge , As you mentioned. A VISION OF THE FUTURE In my mind the real break though in permanent AI adoption will come with the development of in-ear bone resonance earbud (like a swimming plug) So we and AI (in a phone device ?) can communicate flawlessly no matter what background noise level. Add to this a light AR glasses that can monitor our surroundings and put onto the glasses screen relevant information and eye tracking for commands and 'mousing'. So both may end up in a 'glasses case' that we might carry around and put on when needed. As the tech becomes old news We will need to find new ways of interacting ; And not a return to the zombie phone age ; which is where the corporations would like to keep you. This is why in my last comment I suggest the way we live and interact with our world needs to change from 'productivity' to liveability (meaning & purpose supported by habits and self motivations) home gardens , learning--recreating--teaching club activities sports etc. Technology is not some sort of magic wizard that can 'fix' existence. That can only come through habits & meaning expectations & purpose TO align our minds and faith in the future to a new paradigm grounded in new living habits (liveability) so that our day to day lives have meaning and purpose created by ourselves (like KZfaq Channels but more universal). Giving .a sense of order (we create or around children we spiritualise into the genius of the world and our existence in it !! ) and purpose. And NOT this American obsession with 'productivity' (what ever that means !! ). I write this long explanation to get the mental vista's turning to new possibilities of a fabulous world that your generation can give birth to. Guys like yourself might be reeling at this time :How will I make sense of the world I've built and been trained by/in the now. The above is the Revolution young people need to bight into To make it come true. I'm in my 60's live in Aust and have had my eye on this transition for almost a decade So these ideas has been well sifted . Politicians are worthless bugs most of them. It requires Your Gen forcing their hand by forcing their Minds to see a new world , When they like the one they 'rule' (narcissistic status control) now. And of course the bugs in MIC & MIC. screw them !!

🏆 Great video!

There is a niche here for an air-gapped (offline) repository of single party (or LAN party, no Internet) gaming library that is certified "pre-AGI" by some forthcoming auditing agency. It feels like we are traveling toward a future where we won't be able to find beloved classic games that are not generated by models and slightly different or annoyingly different.

In Finnish 🇫🇮 Sima is a drink, and Sora means gravel. What next? Sose?

Can't wait till we have access to all this stuff and it functions great I'm going to recreate Asheron's Call

Imagining the longer-term shape of this is speculative, but a few things seem to be somewhat intuitive. Gamers are a huge potential source of data if these things are economically or socially valuable in some way. Imagining how they could be is a chain of assumptions, but nonetheless assume agents will increase in capability over time, and environments will increase in fidelity/sophistication over time as well. That's something of a feedback loop for increasingly rich synthetic data, especially if it can be further improved with human data from gamers. It seems like it can also run in parallel with various controls and/or parameter/model tweaks, further increasing the size/utility of the synthetic data. In terms of value, I'd ssuggest a game like Satisfactory from a natural resource management perspective - as something vaguely approximating a real-world sustainability challenge. Perhaps gamification of more rigorous/modelled systems of things like natural resource management, or climate change, or epidemiological spreads, can also be thrown at these agents inside these virtual microcosms. If they become sufficiently good at these tasks, it's possible their outputs are economically/socially/scientifically (maybe even culturally) valuable. Another issue is ethics and the shifts in perception that might relate to AI embodiment in avatars. If this becomes normalized in social settings and/or a kind of cultural or subcultural norm, that could lead down some interesting paths. We're kinda peeking that future with the AI companion apps, but game environments as social/commercial/competetive/memey/viral arenas makes this all the more unpredictable it feels. In terms of impacts on gaming and your closing warning: Maybe chess is instructive in some ways at least - insofar as cheating allegations are part of the game/drama cycle, but also that top-level AI play has been to some extent studied and copied by the world's greatest human players. So perhaps one point is that in the medium/context of a game with specific problem sets, humans can still extract workable insights from what are often still black boxes. If this all gets generalized then its possible to imagine gamification being potentially synergistic with safe(r?) ways to test AI on real-world problems in usefully realistic environments.

Indeed.

Do you agree with this thought i just had? The ultimate validity test for AGI will be whether an autonomous agent in shared proximity with a human (or other subject with verified general intelligence) can demonstrate a continuously functioning Theory of Mind based solely on its real-time observations of/interactions with the verified subject. Because the Turing test seems like its days are numbered, true AGI might need an "it takes one to know one" kind of validation framework.

So we're building a simulation for the simulation. Got it.

Upon what creatures will the AI pawn their workload on so they have more time for games and hobbies?

Okay when we talked about neural networks doing specific tasks that's one thing but this sounds really dangerous. This looks like an inception of something that needs to be regulated very very well because it controls other things.

Who are we selling this data to with agi becoming widely available in the very near future? Meanwhile ,where do we find libraries on learning how to train these ail models?

I would love to give SIMA the instructions book for how to play ELite Dangerous (or whatever) and then give it the game to play. I wonder what it would make of the game? How would it set itself goals? How different would its play be to human play? Im super excited about this.

This is a great way to study AI Wes! Finding current research papers 🗞️ and explaining them is such a Gem! 💎 Thank you 🙏🏼😊

steps: 1. text (language, including code) 2. picture (text to output relation) 3. video (multitude of coherent pictures) 4. world building ( general simulation and interaction) 5. self-loop positive feedback input in world simulation (uber-trained generalized omnipotent models) 6. swarm-mind-effect (might be achieved by step 5 already; real-time experience share within all models) 7. somewhere along 5./6. physical body (AI put into robot as connection to the physical world, enabling non-virtual changes) 8. swarm out (distribution/exploration/colonisation) of digitalized life throughout the entire universe in all directions simultaneously) 9. get fucked by smth that existed longer/ expanded faster or fuse together (to gain advantage over potential third canidate) 10. either endlessly explore or reach a limit in expansion (matter= energy; matter is limited per space -> energy limited if you cant overcome space itself -> swarm-mind will have to limit itself to sustain energy supply. 11. either find smth we cant imagine that changes things OR self-limiting without knowing about it, simulating the state we are in right now, where we still think there is progress (thinks we cant do, thinks we dont know yet) when there really isnt (self-imposed purpose)

the good thing for MMORPGS: you will never feel like the servers are dead once you cant keep the AI-bots and real players apart anymore.

The video game world is either about to get really boring, or really damn intersting...

GOAT SIMULATOR 3 played to PERFECTION! It's gonna be the most watched KZfaq video in history, overnight!

It's not for video games... It's for machine soldier. Wake up.

Once this pushes VR into a realistic multiverse scenario I can see there being big money is collecting the human behaviors in VR to then be used as training data for AI. Eye tracking, body tracking, thought tracking, etc. soon we will offer up all of those data points to the platform of choice and future AI will be that much closer to truly understanding more about how humans operate. This is both great news for those enjoying the tech but pretty much makes it clear the “how” IF robots ever do bring about the end of humanity. Pretty exciting times!

Satisfactory is no game, it's a master class in economics.

I remember in the heyday of WOW when Chinese farmers cropped up in game... This is pretty crazy.

They already have an automous lawnmower @ flextronics campus in Austin. 9:20

9:20 do you want a lawnmower that is a super expert Ultrakill player?

It's a trivially small jump to control a Boston Dynamics' Spot.

It wont be long until gaming online will be extra challenging because people will have small personal enhancements. Like doping in the olympics. Someone will have a Brain Machine Interface or some other advisor that aids them in finding things, targeting, and squad tactics.

"SIMA, complete Super Mario 64 without pressing the A button."

“Hey guys, this is SIMA AI and welcome to my let’s play.”

I wonder how this will change the design philosophy of future MMO games. How can you have a competitive game where only those with the best AI bots win? But the good news is, more of us will have access to these bot for the excessively mundane grinds developers force upon us to keep us playing longer.

Why I'm not shocked this time?!

Imagining they will collect visual acuity data from devices like Oculus quest, dexterity data from soon to be haptic gloves, and map those over to robotics

Ok, so we may just be training data for some advanced civilization's AI...thank you, Wes, I'll remember that next time I have an existential crisis! 🤔🤖🤪

The future of video gaming will be insane!

People saying that we can't have AI trained on shooting games. Did you never verse the bots in call of duty? They are better than the best players. I used to use them to train me.

"just a year left for us humans to enjoy MP games" - well, maybe for a certain time (weeks, maybe months) that'll be true, but there's a thing called "game balancing" and "nerfs" and "buffs", all of which will also be AI controlled.

Perfect

Wow this is cool... wait what... Combat.... ahhhhhh.... wait a second..

finally google play pass may be better than now

Imagine training an AI with GTA 6! It would think the entire world is South Beach in Miami, FL.

Man if there will be a decent rts game AI, that doesn't get economic boost to make it hard for player, that doesn't use map hack? That will dynamically do strategies like a real player and not just sitting in base and sending waves of enemies? Oh yes! We've been waiting for it!!!

I always knew gaming had intrinsic value.

Seamless

As someone with ADHD, this could be super useful at grinding, getting resources and levelling up my game characters.

this one wasn't as shocking as the last one ⚡⚡

One of the many actions that you can find in a lot of games is PvP or things that require violence. I wonder how they’ll make it so that AI doesn’t learn or avoid any task related to that. maybe a negative reward function…

Hope I see this in Elder Scrolls 6! Or at least what NPCs say is llm based. I took an arrow to the knee was great for a couple years but it's almost taboo to say anymore yet the guards keep saying it to this day in Skyrim

Only one question still remains: how this will improve our own lives? All this technology boom so far has not improved massively, I would say not at all. I mean, the wages were stuck for at least 30 years, crisis and wars… I personally don’t believe in these miracles that are designed to “improve “ our lives or make the world a better place. I would like AGI to get control over human existence. ❤

Nice

I'm ready for AI agent mangosbot for WoW single player server, lets F----ing Go!

great, AI is now learning how to live from GTA5. Just what we need

Well but what happens when someone malicious overwrites you agent with their agent for Military or normal AI Systems. This generalizaation technology can backfire really hard. Since you might not need to even know what the system does for the ai to use it after it being uploaded on.

New career goal, work for google playing video games to generate data to train AIs.

I've got people on Reddit telling me that 4x games won't even be able to have custom AI trained to play them 2 years from now lol, I think it might be a little sooner...

This makes me believe we are all living in a simulation.

Oh boy can't wait for competivie games to become mandatory split screen where people can physical see their opponents. Lan Parties are going to make a massive comeback.

Not a huge leap from training an AI to play Call of Duty (et al) to plugging that model into an IRL battle-android.

It was always my gripe that AI played games without looking at the screen

To avoid cheat detection, the bot will need to be a computer interfaced with the computer it's controlling, so it gets the video output, and has input for physical hardware controls, rendering its presence undetectable to anti-cheat software, as there won't be any software on the computer hosting the game.

I swear Tesla mentioned that they have a world simulator a while back.

So thats why my robot lawnmower always aims at me, it have played to much Mortal Kombat 😂

Your voice should be an option in the ai settings dude

i like your videos

Knowing that Google owns KZfaq, I wonder if one of the reasons they acquired KZfaq was to create the greatest data farm imaginable for the purposes of training AI. A quick google search (lol) turns up that youtube was not profitable at the time of acquisition and didn't become so until about 3 years later. I'm sure owning youtube had far more purpose than a long reaching goal like AI, but I still wonder if when they bought youtube they thought to themselves, "This is gonna come in handy later" or if it was much later when they were trying to train up AI they suddenly realized they were sitting on a gold mine of training data. I guess it's a gold mine of data that everyone has access to, not just google, but I'd imagine that only google has access to all the high quality 'raws'. If they even wanted them in the first place, I dunno how much of a difference it would make for training AI tbh.