Realtime Animation Production with Immortal Babies! BEBYFACE PART 5

Рет қаралды 76,578

Күн бұрын

This episode is all about an exciting announcement surrounding Bebylon and Project Bebyface! It also doubles as the latest Bebyface test focused on realtime animation production using the iPhoneX & Xsens suit while capturing, editing & rendering all within the Unreal Engine!! It makes production ridiculously fast & fun!
My goal was to spend 4 days and sketch out a narrative type piece from start to finish, testing the realtime pipeline in a more production scenario. I basically wanted to see what unexpected pitfalls I would run into.
iPhoneX (face), XSens (Body), Unreal Engine for everything else (realtime capture, edit, cameras, lighting, rendering, etc...)
You'll have to forgive my ghetto acting and the long test. It could use some serious editing but testing time ran out ; P
Take Aways...
- Its cool how much content a single person can make & how fast.
- The iPhoneX face capture continues to hold up pretty well (High five to the face shift guys!)
- The face capture expressivity can still be improved quite a bit
- Realtime rendering is mega awesome and a huge net win despite a few issues *at least with our project
- Still lots of pain points within the workflow but should get worked out over time.
- Lots of fun still to come!
Related News...
Joe Graf from Epic has been making some really nice improvements to the ARkit integration coming out in 4.22, which I'm excited about!
/ facial-animation-captu...
Context:
These are a series of test I'm doing to find a fast pipeline for performance capture (Face and body) that we can use for our up and coming game Bebylon: Battle Royale. bebylon.world
#vtuber #animation #unrealengine #3d #ue4 #iphone #bebylon #gamedev #arkit #apple

Пікірлер: 176

@Eppideem Жыл бұрын

Yes chap! Thank you for all the detail. So happy I'm finding all this stuff.

@mr.anderson8987 4 жыл бұрын

Thank you for sharing your pipeline! You did an amazing job! Please make more videos!

@Megasteakman 5 жыл бұрын

5 Days is absolutely for this level of quality! Keep killing it! It was so amazing seeing you on stage at Real Time Live, and I'm super happy your work is being recognized!

@malcriado3d 5 жыл бұрын

andddd you just did it again! raising the bar and showing the world what can be achieved without budget and with a lot of balls! Congrats Cory, keep it coming! Felicidades!

@SahilSanganis 2 жыл бұрын

Here after watching xanadu. You have come a long way buddy way to go!

@asianfunythings3942 5 жыл бұрын

so cool man! Great animation,great set up

@ZukunftBilden 3 жыл бұрын

I want to be able to do this with my character. That would be a freaking dream.

@lawjackson7376 5 жыл бұрын

Wow!! Congratulations on the support! Your work is a really promising signal of the future of storytelling

@chrisduwe 4 жыл бұрын

Great and inspiring work on the technical stuff!! It would seem your biggest challenge is to have a message. What would you say to a passer by in your dying moment? What movies touch your heart? What do you want written on your tombstone, to be said in your funeral? What do you want your children to leave your house with as they grow up? Looking forward to see what you will make!!!

@maghata 4 жыл бұрын

really amazing work . every one week i come to your chanel to see this videos and i feel so happy there is some one thinking like me

@SpeedChar 4 жыл бұрын

Man, when will you post something new? Some updates on what are you currently up to?

@lovedancexrsingleman8711 5 жыл бұрын

Hi, how do you rig your own charactor blend shape and skin weight in maya, rig should be as same as apple's skeleton and blendshapes ? All you have to manual do?

@ZeroControl 2 жыл бұрын

Brilliant. I am a fan now for life.

@TexasRy 2 жыл бұрын

What did it take to learn all this? Looks like a great hobby and lots of fun, but do you need lots of background in this tech to get anywhere? Great video, love the material and funny as hell...bugga douggie!

@MrAndreisousa 2 жыл бұрын

thanks for showing how is made. great channel. i want to learn this so much

@everybodycallsmesir 2 жыл бұрын

This is beautiful.

@alhusseinali 5 жыл бұрын

Great job bro, keep going 👏

@kingsmustrise 2 жыл бұрын

the babies just vibin' in the back hahaha

@keneukwueze 3 жыл бұрын

man, I just found this channel... where have you been?

@Chaosfoxx 5 жыл бұрын

yo!..this is the coolest shit i ever watched! ..how did you make the face expressions and the hands work?

@viralbible1629 5 жыл бұрын

wow! AMAZING!

@samfisheragent 5 жыл бұрын

Youre a genius brother I hope you go far. If you ever need any help in your projects let me know im here for you

@slevingee3338 5 жыл бұрын

anyone has an idea how to do that myself?

@RuataLungchuang 2 жыл бұрын

This feels like a fever dream.

@sheraixy 4 жыл бұрын

When are New Updates are coming? :o

@glennevangelista 4 жыл бұрын

Cool. Noob here. Does the ARkit data go straight to unreal engine? Or does it have to go thru like another program like iClone?

@SeanLake3D 4 жыл бұрын

Thanks man! Question, what video setup are you using to record these streams? Love the quality of your videos

@LifeDesignPodcast 5 жыл бұрын

Amazing Cory! This is perfect, you need to get back to filmmaking pronto! Cannot wait to see this film take shape.

@trickdiggidy 5 жыл бұрын

Im stoked dude, I can't wait!

@tunerhun 5 жыл бұрын

Awesome. Very good job

@zhoudongyu0225 5 жыл бұрын

How much is the Xsens suit? I want to buy one to do the mocap.

@miagifodder5599 5 жыл бұрын

looked pretty cool good face animations and movement

@busesart 7 ай бұрын

soo inspiring man. you make my chronically depressed tired 3d artist ass 3d laugh so much and intellectually stimulated

@xanaduBlu 4 ай бұрын

😂

@RobbertScholten 5 жыл бұрын

I can't figure out how to use live mode from Facecap, can you?

@virtual_intel 4 жыл бұрын

Look it up for Blender 2.8 or Blender 2.79b it will surely help to find such info. Good luck!

@abumusa4625 4 жыл бұрын

what is that gadgets name for full body mocap on iPhone

@albinosgambit1877 4 жыл бұрын

Xsense have so many solutions, so whats the price of a xsense suit? I have Iphone 10 but use iClone. Can it be done animation there as well. Nice work!

@oshamikamichaelo.3145 3 жыл бұрын

Hey man, your facial gear, does it come with the MOCAP SUIT?

@alanwagner9359 3 жыл бұрын

How did you create that helmet? you didn't have an amazon shopping last handy for creating one of our own do you? 😬

@ArnoldVeeman 5 жыл бұрын

8:20 yes we can, and once again: I want to do the music! 👶🏾🙉

@davidmonteiro6822 5 жыл бұрын

Fantastic stuff. Did you do some testing regarding the real time performance of the Iphone x hardware (polycount, dynamic lighting, Unreal shading networks, dynamics)?

@trickdiggidy 5 жыл бұрын

Thanks David. I haven't pushed the iPhone rendering side of things much so Im not sure how close to the GPU ceiling i am. My main priority is getting clean data from the phone so I've purposely kept the rendering minimal but I think there's a lot of headroom still.

@davidmonteiro6822 5 жыл бұрын

@@trickdiggidy Thanks a lot. I will do some testing soon myself and let you know

@dragonmusicstudio6946 4 жыл бұрын

WOW! Very Nice!

@harisshlunanii507 2 жыл бұрын

Great work Man, will support you - from India.

@JohnJewel 4 жыл бұрын

Do you have a website ? I can totally see myself trying this for a live someday !

@fliperman22 5 жыл бұрын

Hey man! I am interested in this and would love to learn how you do this. Can you please give me some advice? Thanks a lot! Love this!!!!!!

@trickdiggidy 5 жыл бұрын

Thanks! I would download the unreal engine + the ARKit face capture sample project: docs.unrealengine.com/en-us/Platforms/AR/HandheldAR/FaceARSample

@notalpaka8251 5 жыл бұрын

Wondering if you could use this for a live podcast with animated characters?

@kendarr 5 жыл бұрын

Probably but it's very very expensive, that suit ain't no pocket change

@kehindea 4 жыл бұрын

Nothing to say but wow!

@NightSpyderTech 5 жыл бұрын

Really interesting what you guys are doing with the technology. That's enough to get me to follow along. I'm really surprised you guys don't have a patreon up. This would be perfect for it and help with that issue of hiring your dream team.

@user-mh2bh1tb8q 4 жыл бұрын

Incredible. Please release the app for purchase.

@kineticmotive4466 5 жыл бұрын

Your workflow inspired the hell out of my colleagues and I. Been saving up for an Xsens suit for a while, sucks that the Xsens software isn’t perpetual, but Ikinema Orion will suffice in the meantime. Also, I wish UE4 supported grooms/hair/guides, until they somehow add that feature, I’m stuck rendering in Houdini or Maya :( Anyways, I appreciate you sharing your knowledge for free

@trickdiggidy 5 жыл бұрын

Thanks man, i love hearing that! For the hair, I feel your pain. i've been lucky to avoid that so far with the beby characters but its something I might have to tackle soon.

@kineticmotive4466 5 жыл бұрын

Cory Strassburger well maybe NVidia Hairworks will be a good solution. But game engines always resort to those annoying 3poly collision spheres, and we can’t add other objects for collision, only the characters body can get the spheres. Only thing I can think of is creating poly hair strips, simulating it and exporting via abc. Tedious workflow lol, would rather use a renderfarm but fortunately for us, we only need better collision support and we can progress with the hair

@saikrish655 4 жыл бұрын

Oh my god..... U r genius man..... Hope u find out a fast and smooth workflow...... Without pain points... Anyways... Awesome 👌🏼

@ErnDroid 2 жыл бұрын

Dope!

@MTHALO. 5 жыл бұрын

Dude keep it up but you need amazing story it is very cool man!

@MechAArma 3 жыл бұрын

That's impressive

@rendermanpro 4 жыл бұрын

Sick idea and technically perfection at the same time :)

@mostras33 5 жыл бұрын

Awesome. Can we get an Antz in Da Pants music video?!?

@lukethedukea380 4 жыл бұрын

the goddam possibilities for indie filmmakers now...

@blackleg13 4 жыл бұрын

how did he moves the fingers, it's not inclueded with the suit he wears. is it by rigging? if yes, how can you rig that mocapped model?

@xanaduBlu 4 жыл бұрын

Sadly have to keyframe the fingers and still on the hunt for affordable gloves

@iluminousvj 4 жыл бұрын

insane dude!!!!

@myrhev 5 жыл бұрын

Good luck on perfecting the facial expressions. I know I would be interested in buying something that could mimic facial expressions exactly. Would love to do an animated talking head.

@crazyAlexander23 4 жыл бұрын

can you release the iPhone app for the face capture feature of course you will need to have a prize tag og it

@dileeshaabilash5562 Жыл бұрын

how do you do it on 5 days

@shanezucker1756 5 жыл бұрын

Wowzas. Strassburger knocking it out!

@trickdiggidy 5 жыл бұрын

Dude, i've been meaning to hit you up... I'm email'n ya.

@matrixate 3 жыл бұрын

Great mouth articulation. Not exaggerated but not subtle either....just right to keep it interesting.

@DAVANAMSUMANTH 3 жыл бұрын

how many blend shapes did you used for face character

@trickdiggidy 3 жыл бұрын

@DRdee2018 2 жыл бұрын

Ya masterpiece

@gsadmtp 5 жыл бұрын

Real time animation is future, I cant wait to see what's next in 2020 and after, As a animation 3D/2D student, I'm so hype, I mean ... Wtf ? We will be able to STREAM 3D RENDERING LIVE ON !!!! Do you imagine ??? Looking at a show stream inside a amazing 3D scene with amazing realistic render, with the liberty of 3D cameras, it's fcking awesome, so hyped .......

@Reyana929 3 жыл бұрын

And there is a potential new jobs, realtime 3d actor

@VfxBlender 5 жыл бұрын

Did you use Ikinema to correct the facial animation or was it all done in unreal?

@trickdiggidy 5 жыл бұрын

the facial animation is uncorrected and all done in unreal. Basically It's just the raw stream of data from the iPhone being captured into unreal then played back within a sequence.

@VfxBlender 5 жыл бұрын

Cory Strassburger thank you for your help. I really think this is amazing and you are very inspiring!

@freenomon2466 5 жыл бұрын

Im speechless.. awesome awesome achievent in 5 days!! How many facial wrinkle maps are using for the baby faces?. Is it a custom blueprint?.

@trickdiggidy 5 жыл бұрын

Thanks! I think I only have around 8 or so wrinkle maps and nothing special in the way they are setup. Each one is driven by its associated blendshape. I definitely want to add more wrinkle maps

@freenomon2466 5 жыл бұрын

Cory Strassburger, you are my hero.. I currently use Ikinemas Orion and ipisoft for mocap and I've been doing facial mocap with iphonex/facecap and custom apple facial blendshapes, but I cant link the wrinkle/normal maps to blendshapes in UE4 the way I do with sdk in maya/redshift. Can you show us how you linked the normalmap/wrinklemap inside ue4?. Btw, your lighting in ue is just beautiful! Did you know ue4 has a dev rendering 4.22 version in github that can render softshadows and realtime refelctions? Keep it up and looking forward to your workflow pipeline!!

@trickdiggidy 5 жыл бұрын

Its pretty easy to link up the wrinkle maps... there's several ways/tutorials on blending normals which can be driven by the blendshape by simply making a float parameter in the material, then name it the same as the blendshape you want to drive it... i.e. 'browDownLeft'. Then go to your blendshapes (AnimCurves tab) within your character mesh and theres a column called "type" between the "name" & "Weight" This is where you tell the blendshape to work as a material parameter, so check the box next to 'browDownLeft' and it should just work!

@trickdiggidy 5 жыл бұрын

I'm excited for 4.22! I've been tempted to grab the dev rendering build but don't want to get suck into that vortex just yet!

@3RDEYELOVE 5 жыл бұрын

So is it best to capture audio during mocap and import that into logic for further sound design?

@trickdiggidy 5 жыл бұрын

I would say so. It saves a ton of time and UE4 makes it really easy to record the audio right into a sequence along with the mocap data.

@3RDEYELOVE 5 жыл бұрын

@@trickdiggidy 🤙ty

@REALVIBESTV 3 жыл бұрын

What the price for the realtime that made this video

@SolomonJagwe 5 жыл бұрын

Congratulations Cory!! :)

@trickdiggidy 5 жыл бұрын

Thanks Solomon! Congrats to you too man on your scifi short, awesome job!!

@jovontowns591 5 жыл бұрын

I have a perception Neuron Suit, I wonder if it will live link with high quality. Will the new iPhone XS Max work with this?

@trickdiggidy 5 жыл бұрын

Hey Jovon, im quite sure the XS Max will work

@jovontowns591 5 жыл бұрын

@@trickdiggidy I wasn't sure if they planned to use the same RGB True Depth Cameras and that Faceshift technology in this phone also. They make such drastic changes sometimes, so I'm happy to hear I'd be able to follow UE4 documentation when it's time to test this. Thanks for pioneering and sharing this with us.

@Allinone-np5jm 3 жыл бұрын

Is software ka kya naam hai

@LisaMurphy 4 жыл бұрын

Hi there! What an interesting and unique project. I am very interested in the process you're using here because the results are impressive - and I have a story of my own I'd like to bring to life, in 3D. Can you tell me how well you like the Xsems mocap suit? Is it easy to use and trouble free?

@xanaduBlu 4 жыл бұрын

I love the xsens suit. It's very easy to use and super robust. (knock on wood) I've never had to mess with it in the 3 years I've been using it. Just throw it on and start capturing.

@rajendrapansari4229 2 жыл бұрын

@@xanaduBlu sir can u provide the application that u used in iPhone Please help me

@femisuccess124 2 жыл бұрын

@@rajendrapansari4229 faceware 2 I think

@DRAWKCABLLA 2 жыл бұрын

"They liked it"

@MAMAvsGOD 3 жыл бұрын

Hey brother can you show Tutorial on how to create VR environment i can use Avatar to live stream from like Codemiko. I have Occulus Quest 2 and 3 vive trackers with 2 base stations.

@RANInkTV 2 жыл бұрын

That's how I wanna get dressed

@mondoghulam3050 5 жыл бұрын

Congratulations and very excited to see what you come up with. You inspired me to develop my own setup and content. Still developing the pipeline but share your sentiment about what is possible with this tech. Between this and Unreal, it's one of the most exciting film-making tools around.

@trickdiggidy 5 жыл бұрын

couldn't agree more!

@AAa-tn5rz 5 жыл бұрын

Are you going to be releasing your software for free or cheap related to the mocap animation pipeline since you guys got dev grants or will all this tech stay within your company?

@trickdiggidy 5 жыл бұрын

I'm going to try and put out as much as i can. The difficult part is so many facets are tied deeply to our game character assets which we wouldn't post so it creates a lot of work to separate things. The other factor is most of the core pipeline is readily available right now just by downloading some of the sample projects.

@talismananimations9572 5 жыл бұрын

@@trickdiggidy Hi Cory. You mentioned about the sample projects, what repository are you referring to?

@NathanielGreenVFX 5 жыл бұрын

Cool

@ca_quixote 2 жыл бұрын

Full Blu environmental tutorials???

@janjezz 5 жыл бұрын

This is amazing..so does it record all this directly into Sequencer? Including the audio?

@janjezz 5 жыл бұрын

Also, how much does the suit cost and does it come with gloves?

@trickdiggidy 5 жыл бұрын

Yup, Sequencer nabs everything including audio and drops it all neatly into a sequence ready for playback. The xsens suit doesn't come with gloves... I'm still searching for a solution there. For the suit cost, I would check their site... i know the software features you need play a role in cost (we got ours years ago and I can't remember all the details.)

@janjezz 5 жыл бұрын

@@trickdiggidy cool..and how are you able to capture the iPhone facial data and body mocap at the same time? Did you create a custom pipeline for that?

@jungchoi5834 5 жыл бұрын

There is a way to connect hi5($1000) gloves to mocap in UE. Some guy is doing it with Orion and Hi5...I can get you his link if you need it.

@trickdiggidy 5 жыл бұрын

I just setup the bebies animation Blueprint to use the iPhone Live link for the face (Blend per bone) and the Xsens LiveLink for the body.

@jadendupree 5 жыл бұрын

Would love to help out in some way...if i could

@Ponk_80 5 жыл бұрын

I can’t wait til this technology is built into mobile phones, where we don’t need to put on a suit :)

@virtual_intel 4 жыл бұрын

Not gonna happen, you'll always need a suit for full body mocap, or bare minimum a XBox Kinect like 3D depth sensing camera.

@HumanInteract 5 жыл бұрын

Awesome! If anybody can do this it's K&L

@oshamikamichaelo.3145 3 жыл бұрын

Would love to see more stories and perhaps contact you for a critic of mine which is currently in development. Thanks big man.

@borisnoa773 5 жыл бұрын

is the book ''Film Making For Idiot Bebies'' already available ?

@trickdiggidy 5 жыл бұрын

lol

@srinavin 4 жыл бұрын

Amazing 😍 teach me master

@yourbestfriendlawbringer917 4 жыл бұрын

Get back to us man i want this made

@lamardwingster1825 5 жыл бұрын

Brother I love your work, and the baby characters OMG!!! Ok, I'm trying to get where you are as far as the animation part. How can I make one of those animation suits to do my animation? I do have the iphone X but not the rest of the get up. Can you do a tutorial on a DIY, and all the soft ware I may need to get started. By the way I using Daz3d.

@paulgrl5227 5 жыл бұрын

Awesome, exept the futuristic look, i hate it, it might be good for a game but not for a film,... i think.

@philippdrflr 5 жыл бұрын

Can this ever be widely available to human lifeforms? Or do you have to be a genius like you?

@trickdiggidy 5 жыл бұрын

Haha thanks Philipp! Thought its all widely available if you have some 3d/tech chops... and money for an iPhoneX + some kind of capture suit!

@philippdrflr 5 жыл бұрын

@@trickdiggidy aha... I understand. So I guess it's definitely failing due to my missing 3d / tech chops ;-) Sh...t

@oshamikamichaelo.3145 3 жыл бұрын

It's easy to run mad in 'em suits.. fast-forwarding made it real funny...Big tops

@creativemotiondemo6243 5 жыл бұрын

Guys i need help is there any ios developer using unreal ?? i would appreciate anyone's assistance by replying my comment so i can ask him a question

@felipeborba932 5 жыл бұрын

Film?

@trickdiggidy 5 жыл бұрын

lol, I wondered if anyone was going to call me out on that! (In my warped brain) it was a very obscure reference to FightClub which would have made slightly more sense if I didn't forget to add the other FightClub references into the video. I'll get'em into the next one!