"Don't worry I'm not gonna crash into them" 💀💀

It's kind of cute how the AIs always place some "special" block as the top of their pyramids instead of just making the whole thing out of one material

That tower joke caught me off guard 💀💀

"See those towers over there? Let's fly towards them"

You gotta admire how versatile and full of possibilities Minecraft is. The game has been out for about 15 years, and people are still doing unique and interesting things with it.

6:05 The sun being at the exact parallel angle making the geometric shadow was a fantastic coincidence

Wise word to remember about A.I. "The more creative powers you give them, the more destructive powers you give them. There is no way around it."

the best pyramid turning out to be surrounding previous failed pyramids feels like a metaphor for something, but i'm not sure what.

The AI doesn’t even know what that tower looks like IRL, you should get it to describe its own design first so it realises what it’s even building. I’d also recommend something like snippets or get it to make its own codeable rulesets to building before it builds.

You can do "/gamerule commandblock_output false" to make it so that it won't display every single block that they place in the chat

9:17 that would be sick having a ancient city inside of a pyramid

I’m really curious to see where AI inside of minecraft goes. I imagine in a few years we’ll have a way to train them on good and bad builds, give them a better concept of what constitutes a “building”, but what I’d REALLY love to see is some kind of simulation between them of an SMP, giving them access to baritone and allowing them to work out how to get things done in an open environment. Let their decision making guide them to the best path. Have them identify problems and come up with contextual solutions. E.G. I’ve been attacked by monsters outside my home at night, the best course of action would be to light up the surrounding area and build a wall to keep monsters out. Have them break that down into simply steps that don’t have to be ai powered but can just be automatic, craft torches, go strip mining to get resources for the wall, etc. I saw a paper about a similar project a while ago where they put down LLM agents in a simulated town and the “people” in the town had a small memory where they remembered previous experiences and had some agency on what they wanted to do, one of them planned a Valentine’s Day party and invited guests to it entirely on its own with no input. Very interesting stuff.

6:00 omg that actually caught me off-guard holy moly

What happens if you ask the AI to build esoteric concepts? Like "build the meaning of life" or something bizzare and conceptual like that.

I wonder how it would handle a prompt asking it to make a maze or labyrinth

There’s something really amazing and unique the even “incomplete” buildings are. Very symmetrically interesting patterns! Keep on making more of these tests and videos!

It was a good choice giving the bots cheats.

as I've been watching this series I've been thinking about how cool it would be to have a mod that adds a ton of alien-like AI generated structures, it really feels like aliens looked at human architecture and tried to re-create it. I can just imagine stumbling upon those pillars at 4:35 and following them all the way just to find an empty, strange looking building with seemingly no purpose. gameplay wise I imagine players would be disappointed but personally I'm just here for the vibes, and the vibes these buildings give off are immaculate.

I would love these AI bots building liminal spaces in minecraft. It would totally feel disorienting.

Are the models integrated with vision, or is it literally just "blindly write code and hope for the best" ? I know it'll be expensive, but imagine if you supplied the model with periodic screenshots.

9:38 the music combined with "built it on top of failed pyramids" is so emotional.. makes me want to cry. Stellar video !!

You should see if this can be used to help with the build the earth project. This might actually make it possible with further developments.

Survival test! It would be the best thing to just have a survival and have these slaves to mine and get wood. Would be so funny.

I love videos like this that just explore what AI can currently do and treat it as a novelty to play with, rather than making any grandiose promises or predictions. The societal implications of this tech are huge of course, but we can also just slow down, take it in, and play some video games.

As someone who is tinkering with writing AI and loves Minecraft I thank you for making these videos.

"The more creative power you give them, the more destructive power you give them." - Foreshadowing our future AI overlords.

These kinda remind me of the feeling of the original backrooms with these structures that are almost human but are very clearly not human at the same time

This is a much bigger deal than the narrow domain of minecraft. You’re showing how we can augment the capabilities of the models with better design choices for how they navigate their task. This is a multiplier on algorithmic / scaling improvements for future models.

You can also use /fill command to make it even faster if you are doing cuboid etc or use worldedit for more complex stuff like spheres.

this project is so cool, it's so interesting to see the different LLM interactions in the game, i'll keep tuning in ! great work !

Loved the pyramid build. Thanks for sharing your progress/experience with Mindcraft

this is awesome dude, im so invested in this project

I like that some of these look a little alien, that one listing pyramid with all the gaps feels so strange. That Parthenon, colosseum, all have a strange, nearly otherworldly aesthetic, vaguely recognizable but still strange. I like it.

i LOVED the first build at the beginning. it's so beautiful! it's a real work of art.

I just started to watch the video and immediately he says “See those buildings, I’m gonna fly into them” lmao

11:08 lmao. When I was 9 in 2011 I got operator to a small server that had world edit. So I wanted to build a huge hollow glass sphere in the sky... but I set it wrong and the server crashed. When I re-logged, parts of the world for hundreds of blocks were made of glass shapes that weren't quite fully rendered into the giant sphere I had tried to create.

Love the new approach and transcendental build sophistication - gotta love that!

its cool to see that you keep making videos on this its pretty interesting

I wonder if it would help to use a multimodal model and automatically take screenshots of the building for feedback so the model can iterate until it judges it to look good.

i've watched all of your mindcraft videos, and they are very interesting. I have been interested in AI for a while now, so its interesting to see it implemented in a game as complex as minecraft, and seeing the way that AI creates code in a more visual way. I would like to see if there was a way you could get the AI to build something complex, such as a mansion, by having specific parameters about where the AI can build and build each room separately, until you get a final product. And also potentially having different AI models work on the same more complex build, so we would be able to see a more obvious difference in the way they generate code. Overall, these mindcraft videos have been great :)

Genuinely one of the best things I've seen done with AI, this has actually given me build ideas

Imagine the possibilities. An interesting idea would be to see how he can change the world, ask him to delete everything and create something else to see his capabilities... it would be incredible

I believe for the /setblock command you can use keep and it'll only change the block if it's an empty air block - then, if something goes wrong while the bots are building, you could tell them to just replace everything they just built with air as a sort of "undo" feature

the tower joke was funny but seeing what the ai built blew me away. what a ridiculous improvement. I love these videos

The sphere at 6:05 is honestly impressive. Especially the shadow it casts. I honestly never thought GPT could make something so intricate.

I love this kind of vids so much, ty

Damn, that's a huge improvement 🤩🤯I hope more people will see this channel 😊If in the future we can get the models to understand us more carefully and take longer to write the proper code, it'll be awesome. Right now, every AI from every field is trying to make stuff as fast as possible, so they make a lot of mistakes and we have to iterate multiple times 😅I wish they would just take their time and do it nice, i don't care if i have to wait 2 min instead of 10 sec.

I love these I in Minecraft video's! Hope to see even more in the future :)

This guy is a legend, in less than 30 seconds in he already made a two tower joke that includes planes. (You know what i mean)

Great video, as always! :3

I'm fascinated with all your work. A very different but understandable way to show the comparative potential of the various models. And one can only imagine how videogames are going to look like in 3 years time. We are truly at the doorstep of a new Era.

This is really cool, we can't lie about that. Have you discovered anything new with the AI's since this video? When is your next one? I'm really enjoying your videos, your showcasing of their abilities are fun to watch.

You: hmm interesting its made out of concrete, glass and glowstone Also you: please build it out of concrete, glass and glowstone

this is by far my most anticipated AI projects. I canot wait to try it in the future when it's a bit more stable and easy to use

Love these videos, keep them up!

Looking forward to the setup tutorial!

If anyone sees this, I need an explanation. I still don’t rly understand how the AI is kinda just building in Minecraft. It’s just a language model. Is it just coding and that code is put into “mindcraft” then it translates that code into the characters doing something? And how does it kinda just automatically build something? I don’t get how AI is using cheats. And how is the code that AI is creating to build different from mojang’s code that is used to create terrain and villages and stuff?

This is so cool because it sort of offers us a way to almost visualize the way the neural networks formed the "understanding" of concepts.

If you want to fill a place with the same block just use /fill command instant of placing every block with /setblock .

LOVE THESE VIDEOS! KEEP EM UP!!!!

16 seconds in and you earned a Subscribe... Worth it, and valid.

You could use a greedy-meshing-like approach (but with volumes instead of faces) to post-process the AI's output into chunks of the same block type. Then you can use fill commands to place multiple blocks at once, which would make it much faster. Btw I believe you can also disable chat messages for commands, so the chat is not spammed with setblock messages.

I love these Ai building stuff in minecraft videos

wow another upload! so quick!! 😁

This is amazing. I must try it!

Very cool and fun to watch!

I find it cute that the AI can be proud if itself, like Claude saying that it's happy with how its builds turn out

A while ago I tried to get this working, but they were blissfully unaware of their code not working, or that they had previously written code. I might be able to get it working now, but I"m 100% demotivated. I'll just watch your videos instead. Also, you thought about experimenting with instructing the agents to build a fully functional vanilla neural network?

Cool stuff!

I am currently doing a similar thing with AI. I am writing a python script to nearly mimic the functionality of chatGPT, with my own added features such as file creation, txt file reading, and internet searching. I also created a makeshift memory for it, that allows it to recall past conversations even if you restart your computer or the model. All of this runs locally. The issue im running into is having it write commands to execute these different functions.

It's quite impressive how well the larger models do

You could possibly write code manually to check if the block being placed is more then a certain distance away from the bot to prevent accidental griefs, maybe make horizontal and vertical different values though in cases like its tower

my favorite series on youtube

Good video dude keep it up

10:45 Have an undo function that logs the coordinates of all placed blocks (as well as the block it’s replacing), have it delete all blocks that replaced nothing then swap the blocks with what it was before! (Reverse the array of swapped blocks and set replaced lava and water to the bottom of the array) You should have it create a new log every time the AI triggers the build action. (Of course it won’t take into account damage caused by placed TNT, lava etc. but it’ll cover most of your cases.) Or if your computer is really fast you could just save the state of the game before the ai does an action.

Are you able to get it to add on to a build it already created like with the panthenon build ask it to add a floor to the bottom of it or is that not possible? I’m so excited to see how advanced the AI builds can get.

Imagine in the future making new survival words each with unique structures made specifically for that world

this series deserves more views

this is my new favorite genre of mc youtube

AI in Minecraft would actually be terrifying. Like imagine never being able to prove that you made your own creation without recording the entire process.

We need an AI redstone building contest where they build a lot more complicated redstone builds

Cool video!

With the multimodal llms, you can probably get it to take an automatic picture of the build, ask the llm to critique it's build, then iterate!

Love this series

Hello, @EmergentGarden. How it would be setting an AI Minecraft bot in wild there and making it learn by itself from its mistakes via machine learning? This idea fascinates me. Great videos, by the way.

Super cool!

So satisfying how quick and dirty a.i. this can be used to develop new worlds for Minecraft. Definately next gen gaming for minecraft.

I think it would be interesting if you added one of those mods that remove the building height limit, could make for some cool big towers

i could watch these guys build random nonsensical architecture for a while. looking forward to the tutorial video, want to be able to do this myself

could this be used as some kind of dynamic custom dungeon generator, or enhancing other minecraft structures such as villages?

Still my favorite series on youtube. Keep it up please :) I prefer the survival mode attempts to building than this "cheaty" way.

Hey, you should look in to the /fill command, and the /clone command. I guess they can really help speed up your process if you integrate them as optimization

0:20 light looks so nice

for some reason, looking at the inside of the glowstone tower with the glass floors got me thinking that, if you're inside a room where the walls, floor and ceiling are completely covered in sources of light, BUT the light is relatively weak, could you make the room so large that you couldn't see anything but the silhouettes of things placed in the center of it, or would there always be enough light to see clearly?

Amazing work. How do the models react to more abstract requests, such as, "Build a gloomy house"? Or how about "Build something that demonstrates the Banakh-Tarski paradox" (or, more simply, "Build something that proves the Pythagorean Theorem")?

i think one way to improve this is by, making the AI make relative positioning instead of world position. since the ai wont have to do more calculations, then you take his numbers and convert them to world cords. that way, you can also better control where the building is gonna be.

Feels like giving it the ability to revise its work with a group of expert AI would help improve the output

You should attempt self reflection: after every single prompt, prompt it to refine its previous output (you can automatize it). I wonder if the results get better.

I wonder what was different about the one that centered on 0 0 0? If they're only running setblock commands maybe you could filter them so the coordinates are all relative to the set building location, and if the coord is too large it doesn't work because it's outside the building zone. like whenever they try to send setblock 3 6 3 stone or whatever, detect it via regex and change the coords before sending it

Can't wait to see how Claude 3.5 sonnet will do!