Unity Sentis: Create AI-driven XR experiences

Рет қаралды 4,052

Күн бұрын

Hi XR developers! In this video we look at how to install and use Unity Sentis! Sentis brings the rapid pace of AI model innovation into game development. Import your model via the ONNX file standard, create new gameplay experiences, and publish on all Unity-supported devices. As always I prepared several use cases and sample codes for this tutorial, which you can get on my Patreon!
Resources:
🔗 Unity Sentis Presentaiton at Unity 2023: • New AI-driven gameplay...
🔗 Unity Sentis Product Page: unity.com/products/sentis
🔗 Unity Sentis Documentation: docs.unity3d.com/Packages/com...
🔗 MNIST Model in ONNX Format: github.com/onnx/models/tree/m...
🔗 Image Texture: drive.google.com/file/d/1ktF7...
🔗 Unity Sentis Samples: github.com/Unity-Technologies...
Other Videos:
📹 Get started with visionOS development and Unity’s PolySpatial: • Develop for Apple Visi...
📹 Create Believable Haptic Feedback with Meta’s Haptics Studio & Haptics SDK: • Create Believable Hapt...
📹 Get Started with Meta Quest Development in Unity: • Get Started with Meta ...
📹 Meta Hand Tracking Pointer & Pinch: • Meta OVR Hands | Point...
Join the Discord!
👾 / discord
Support the Channel:
🫂 Patreon: / blackwhalestudio
🐦 Twitter: / xrdevrob
Need professional XR & Multiplayer development for your project?
👨‍💻 Get in touch with us: bento.me/blackwhale
Chapters:
0:00 Intro
1:06 Unity Sentis installation & basics
2:32 Model download & import
3:13 Digit Classifier Script
6:20 Add Softmax to Classifier for better readable outputs
7:20 Finger Drawing Script (Rough Overview)
11:35 Setup scripts and canvas to draw numbers
13:05 Final testing & tips
13:51 Outro

Пікірлер: 28

@jumprabbit8946 2 ай бұрын

Thank you so much!!

@YeahThatGuy7 6 ай бұрын

Can you make a tutorial that shows how to create an AI that is capable of having a dialogue with the player. They showed that at Unite but have 0 examples, very frustrating.

@betterwithlasers 7 ай бұрын

Thanks for the great tutorials! Really helpful with navigating the SDK changes. Quick one - how do handle moving the camera (walking around) with analog stick? Wondering if you directly change the transform of the OVRCamera, give it a parent object or something else. Cheers

@blackwhalestudio 6 ай бұрын

You can give your OVR Rig a Character Controller component and move it that way. Using the analog stick would be the preferred way yes!

@DevDunkStudio 4 ай бұрын

Nice guide! I wonder how the texture converter converts texture, 28, 28, 1, into 1,1,28,28 Shouldnt it be in the same order?

@ajit5626 4 ай бұрын

Great Video! Would love to know which asset pack that background world is from please!

@blackwhalestudio 4 ай бұрын

I generated the skybox on Blockade Labs!

@JackHeslewood 4 ай бұрын

I did not know that you can add layers within Sentis, I thought it was purely inference. Would you suggest for performance reasons to add the layer within the ONNX model? Thank you for the video!

@blackwhalestudio 4 ай бұрын

I would suggest to do as much as you can outside of Unity already, yes!

@zappist751 6 ай бұрын

Can you please make an object detection video using it?

@blackwhalestudio 6 ай бұрын

Hi, nice idea, I will look into it for sure!

@zappist751 6 ай бұрын

Thanks! @@blackwhalestudio

@crazyfox55 5 ай бұрын

One small thing, why not use the built-in "using" keyword, which will dispose of the objects even when an exception is thrown. Also can you dispose the input tensor within the execute function either at the end or after the model has already ran? Thanks.

@KP-if2rm 4 ай бұрын

Good point, thanks!

@abderrazakschannel4491 5 ай бұрын

Please can you tell me what is the sprite used in the first object detection game(car....), transparent but has soem kind of boudries around the objectc. Thank you

@blackwhalestudio 5 ай бұрын

This is an object detection sample from Unity. They haven't released it yet unfortunately!

@carpineti93 7 ай бұрын

Very interesting. I think I will try to do that. Can you tell me if you can use meta quest 2 or do you need quest 3?

@blackwhalestudio 7 ай бұрын

you can use Quest 2 and 3, both works! :)

@mq6802 5 ай бұрын

Would it be possible to do this but in passthrough mode reading a book or words on physical object?

@blackwhalestudio 5 ай бұрын

Yes, absolutely

@mq6802 5 ай бұрын

@@blackwhalestudio how? i been trying to find the script that gives you access to the camera?

@blackwhalestudio 5 ай бұрын

We dont have access to the camera of Meta Quest!@@mq6802 What exactly are you trying to build?

@mq6802 5 ай бұрын

@@blackwhalestudio thank you for your quick Reponses. I would like to use the quest 3 cameras for real time number classification like in your video. For example a math problem hand written in a paper would be solved by being in passthrough mode and and a combination of neural networks. When having the quest on would display the numbers in your headset.

@blackwhalestudio 5 ай бұрын

Ah, I understand now. Unfortunately, this is impossible right now, since we don't have any way to access the camera.@@mq6802

@SaschaRobitzki 5 ай бұрын

Do you also get a bunch of warnings of the following kind? `Shader warning in 'Compute.Shaders.Broadcast.gen': integer modulus may be much slower, try using uints if possible. at kernel ElementwiseFModInt at Compute.Shaders.Broadcast.gen.compute`

@blackwhalestudio 5 ай бұрын

I haven't experienced such errors before

@SaschaRobitzki 5 ай бұрын

@@blackwhalestudio It disappeared for me as well. I only got it at some of my first runs, even though I didn't change anything after that but the warnings disappeared.