BlendArMocap is getting wild - custom transfer, freemocap and more

Рет қаралды 12,621

Күн бұрын

Get BlendArMocap
github.com/cgtinker/BlendArMo...
Read the Docs:
cgtinker.github.io/BlendArMocap/
Get Freemocap
freemocap.org
00:00 Basics and Detection Features
05:20 Freemocap Import
06:30 New Transfer!
08:30 Transfer Concept
11:40 Value Mapping
14:47 IK Chain
18:03 Value by Distance

Пікірлер: 122

@SquaresToOvals Жыл бұрын

This looks incredible. I remember spending a lot of time with a kinect in mikumikucapture years ago, and have been surprised to see no good models releasing with face, body, and hand tracking all-in-one. I hadn't known about mediapipe holistic until now. The best of luck with this project! I think I will learn blender now :)

@ssanyeux 11 ай бұрын

dude i love you. thank for creating an alternative. flexible enough for diy'ers (specially those that lack all the best equipment) and professionals.

@obeycelestia Жыл бұрын

Man this is SO exciting, especially shaking hands with the FreeMoCap project is really cool to see! This is PRIMO

@HeelHeatTV Жыл бұрын

Crazy, I was messing with this last night and thought it was amazing! Thank you!!!

@marcosaltamirano8776 Жыл бұрын

Your job is amazing, the pose and hands mode are the best ones.

@allimille Жыл бұрын

Worked great, installed everything. Here are some suggestions: a preview of the video detection progress and the option to be able to choose from which frame the video detection starts and ends. Congratulations on the excellent work!

@soothingillustration50 7 ай бұрын

This is really impressive! You deserve a lot of support and donations 👏

@4U4U7 7 ай бұрын

You are a legend! Appreciate your work, thank ever so much. Keep well and keep up the excellent work.

@s2mann 4 ай бұрын

I use this with Blender 3.6. I haven't used mocap before and this made it incredibly easy. The tutorial is a must but after that, pretty smooth sailing. Thanks!!

@hbarone Жыл бұрын

Incredible job !! Looking forward to testing

@Leukick Жыл бұрын

Subscribed! Look forward to updates like Auto Rig Pro compatibility

@jamesmcandrew7860 Жыл бұрын

I believe it's an AirTrack. BUT Holy crap! This is awesome!!! Great work!

@kttdestroyer 11 ай бұрын

Just found this few days ago... Just wanted to say thank you 🙏

@gaudenzin Жыл бұрын

Awsome work!

@Ready-Assets Жыл бұрын

Thanks for update! Super

@SimpHarderPlz Жыл бұрын

This is a magnificent update Thanks

@NandukDaProot Жыл бұрын

I dont have any more words than just beatiful amazing and amazing.

@themineurs 10 ай бұрын

you are creating open source full motion tracking solution it is insane thank you so much

@Bordres Жыл бұрын

Amazing! Was wondering when you could use videos for mocap, as my webcam isnt the best in quality haha Looking forward to more updates on BlendArMocap!!!!

@cgtinker Жыл бұрын

Videos can be used for a while by now, I made it the default as it has been overlooked by many people :0

@MrGamelover23 Жыл бұрын

@@cgtinker The fact that you were able to make real-time motion capture work in blender is amazing! Any chance you could take this tech and turn it into something that works independently of blender, so that I could be used, for instance, with other animation software, or with video game engines or into a tracker for vtubing software? Because if you could make this work in a video game engine, this could change the game for amateur v-tubing.

@realjames1 Жыл бұрын

this looks amazing since I'm planning to do an animation project

@apachon 10 ай бұрын

This is amazing !

@jihamih1219 Жыл бұрын

nice, thank you!

@TheRealJerseyJoe Жыл бұрын

Awesome !

@danyalghani7421 5 ай бұрын

Amazing stuff, truly! And the option to import from freemocap translates into multicam goodness for us DIYers! Question: Have you considered doing your own multicam setup BlendArMocap within Blender?

@cgtinker 5 ай бұрын

don't plan to develop it further for now

@danyalghani7421 5 ай бұрын

@@cgtinker Oh thats sad, but understandable, keep up the good work!

@cgtinker 5 ай бұрын

@@danyalghani7421 thanks :) hope you doing well!

@mb345 11 ай бұрын

Can you place your video in another part of the UI so we are able to see what options you select on the right side of the screen? Awesome tool you built!

@patricki2071 9 ай бұрын

You need more folowers thats amazing and with the gnu license thats a very helpfull mocap addon thank you

@Vidyut_Gore Жыл бұрын

This looks amazing. I read the docs. I didn't understand why blender needs to run as admin.

@gutzimmumdo4910 Жыл бұрын

amazing

@vivianwarran8804 Жыл бұрын

Thank you for your video. How do you get the meta rig into a model? Thank you.

@DanielPartzsch Жыл бұрын

Thanks. Is it not possible to also use rigify rigs with the new face rig? Even if I just would like everything but the face mocap? Also does transfer mean it's gets linked via drivers or does it bake them movements to the rigify rig then? Thanks again!

@cgtinker Жыл бұрын

For the new face you'd have to create a new config file - I think renaming targets should do. Linking is via driver's and constraints, this intended. This allows you to modify the data more easily using the driver settings in the constraint panel.

@MrGamelover23 Жыл бұрын

The fact that you were able to make real-time motion capture work in blender is amazing! Any chance you could take this tech and turn it into something that works independently of blender, so that I could be used, for instance, with other animation software, or with video game engines or into a tracker for vtubing software? Because if you could make this work in a video game engine, this could change the game for amateur v-tubing.

@cgtinker Жыл бұрын

should work but guess I don't have the time to maintain stuff like that (kinda capped with my current tools..). On which video engines u thought? just curios

@isaacwardmusic 6 ай бұрын

What should be done if you're mesh overlaps with itself after applying mocap data?

@zoruken_4w4 Жыл бұрын

Thanks for this amazing update!!!! oh, also blenderARtrack will have any update? the app is cuting the framerate at 15fps unfortunately... Also wasnt unable to set up a 1920×1080p resolution, but isnt a big issue compared to framerate. (even with lower settings) also it have some small tracking displacements when one uses different angles or target the camera to up into sky or turn around. A external stabilizer gimball fix a lot of the camera shake for lucky ☆

@cgtinker Жыл бұрын

planning to switch the video recording subsystem in the future.. will take time - but lets call that a yes^^

@mbxymail 11 ай бұрын

I have a suggest it seem the nonlinear function cannot be use to make adjustments? Right? Could you create a button that would take the final rig and create keyframe for the rig and release /delete the empty and delete constraints and just have keyframes . That way the nonlinear blender function cluld be used. That would advance mocap. THANKS

@ArpittheFact 11 ай бұрын

Hwy can we put animation in other rig also because when i start animation translation it stop other armature animation please reply 🙏🙏🙏🙏🙏

@cgtinker 11 ай бұрын

I don't fully understand sadly. Do you want to "retarget"? Also, due to the nature of the add-on, driver objects get deleted when a new config gets applied, the add-on is not intended to animate multiple rigs in one scene. kzfaq.info/get/bejne/kMx8iJV_vb3XhXk.html

@ArpittheFact 11 ай бұрын

Thank you so much for your reply As can I export a animation armature and import it again in blender so that I can use animation in different rig in Blender

@cgtinker 11 ай бұрын

@@ArpittheFact you can use this rigify extension I made to make the export easier: github.com/cgtinker/rigify_gamerig_extension link the metarig to the control rig, then bake some animation(s), unlink it and export it as fbx by example once you are done

@isaacwardmusic 6 ай бұрын

Is it possible to remove the mocap data after transferring it?

@liweilin6649 Жыл бұрын

when i done transfering driver, how do i adjust Rigify pose? i cant keyframe or move rigify rig

@marcosaltamirano8776 Жыл бұрын

Is great, but i have a little problem the face have a bit issues , the markers show a precise movement o the face, but the face rig doesn't move the same way.

@cgtinker Жыл бұрын

checkout the distance driver setup, you can modify the results and maybe even enhance the transfer config.

@showhuiy933 Жыл бұрын

Android's blendarTrack cannot compress files for sending to a computer. How do I set up the application?

@wbcast9698 Жыл бұрын

Can you do a demo of face with video and sound in blender. I loaded BlendArMocap and a audio but it seems the transfer of mouth speaking is not working. Please thanks

@dpersona 7 ай бұрын

How to combine retargets of blendartrack facial data and mixamo animation?

@hata_0 3 ай бұрын

After I reassigned the animation to rigify in blender, it did not capture the center of mass information. How can I solve the problem?

@Wolv21 Жыл бұрын

Hi! I'm running Blender 3.4.1 and I have session data from freemocap. But when I click Load Session Data I get the error message "Session directory not valid. D:\videos\mocap session\session_2023-03-11-10_29_10\" Wonderful work you are doing here (and Jon). If I can redirect more information to you let me know!

@cgtinker Жыл бұрын

Thanks, uhm, does the folder contain the folders "DataArrays" and "SyncedVideos"?

@Wolv21 Жыл бұрын

@@cgtinker I see the folders "annotated_videos", "calibration_videos", "output_data" and "synchronized_videos"

@cgtinker Жыл бұрын

Seems like folders have been renamed on the freemocap side? Are in the output folder .npy files?

@Wolv21 Жыл бұрын

@@cgtinker yes i see in "output_data" mediapipe_body_3d_xyz.npy mediapipe_face_3d_xyz.npy, etc. in "synchronized_videos" Camera_003_binary.npy Camera_004_binary.npy

@Wolv21 Жыл бұрын

also running the freemocap-gui to do the calibration / capture / sync / export to blender, so i'm not sure if that has something to do with it.

@cartoonforkids-usa725 5 ай бұрын

My webcam doesn't work in Blender 4.0.2 Mac OS 14.2.1, what should I do? Installed everything you need.

@tobiasb.7516 Жыл бұрын

hi there, is a full tutorial planed ? like for a stickman or whatever i have problems to transfer it to an character -_- and i already tryd to understand your videos

@Hepworks 11 ай бұрын

Hi - is there someway to see the frame rate of the motion capture data? I could see on the BlendAR app that it seemed to vary. I'm trying to figure out how best to sync it to audio.

@cgtinker 11 ай бұрын

for blendarmocap with video files that should basically work out of the box. I also synched audio with blendartrack (iOS) successfully - using a clapper helps a lot... as long the frame rate is consistent there shouldn't be an issue

@Hepworks 11 ай бұрын

@@cgtinker for the blendartrack does it allow you to record the audio? I ended up making weird faces to use as my clapper!...and was recording my audio into OBS - my problem was partially for sure having different frame rates between blender and premiere...and OBS for that matter...I just need to smooth out my workflow. Follow up question: is the data from blendarmocap as high resolution as from blendartrack? What a great set of tools you've created!

@cgtinker 11 ай бұрын

@@Hepworks you're welcome :) oddly enough I'd say blendarmocap is somewhere in between! arcore (android) < media pipe (desktop) < ARKit (iOS) think I'll soon switch to shape keys, I think that will improve transfer to faces a lot, point based is not to great. I consider to make the most tracking stuff standalone in a separate exe as it's easier to maintain for me.

@dawitadmasu-vo6mu Жыл бұрын

it is really cool!! could you share me the way, to fix the problem of installation of mediapipe failed, check the system console output when I try to install install dependencies?

@cgtinker Жыл бұрын

create a GitHub bug report and give me some info's there (hard to debug in YT). Let me know the following things: 1. Operating System, Blender Version, copy logs from the system console and either attach them as .txt or paste them github.com/cgtinker/BlendArMocap/issues/new/choose

@dawitadmasu-vo6mu Жыл бұрын

thank you my brother, now it is safe!!!!

@PecoraSpec 7 ай бұрын

i noticed that all my recording creates uncanny results where the face is so badly shaky! What should I do?

@mica.motion Жыл бұрын

I dont understand how to edit the animation after the transfer is done. Help please.

@cgtinker Жыл бұрын

all the data inside the collections uses keyframes. (besides the objs with a .D suffix (which are just drivers which is based on it). You can modify driver values using the mask presented in the video, besides working with the keyframed data is gd swell

@kingskye7110 Жыл бұрын

Can you provide a separate video to introduce the general algorithm from MediaPipe to bones? I'm not very familiar with Blender

@cgtinker Жыл бұрын

there isn't a general algorithm. I've started working on a pypackage but it's not done yet github.com/cgtinker/mediapipe_rotations

@geraldbryan7249 Жыл бұрын

Hi, there is a problem when i transfer animation, the arms of my rig doesnt move, but everything else does. Do you have any idea why, and how do i fix this? Thanks!

@cgtinker 11 ай бұрын

most likely the rigify rig has not been generated

@sarfarajsaiyyad4057 Жыл бұрын

It's great cg tinker Can you please make video on How we can transfer ther capture data onto another model ? Or how we can export this data in fbx format for macap animation?

@cgtinker Жыл бұрын

Guess I'll make a video about exporting soon as many seem to have issues doing so. The video should actually give you the knowledge on how to transfer to another model.

@shoaibakhtar4934 Жыл бұрын

@@cgtinker so if we select the armature of the model and apply the mocap to it will it automatically work

@cgtinker Жыл бұрын

@@shoaibakhtar4934 the config is for rigify, you can change the config to support other / similar rigs. so if the armature is a generated rigify rig - yes

@Kozlov_Production Жыл бұрын

Sorry, but I didn't find a way to transfer face animation to HumanGenerator models

@cgtinker Жыл бұрын

In the video is everything you need to know to create a custom configuration to do so. It's some work though. However, once a config has been generated it can be shared so hopefully the transfer config pool grows over time

@Sgirlie755 11 ай бұрын

Have you figured out how to transfer face animation to Human Generator models? I am stuck at that part of the transfer too.

@Kozlov_Production 11 ай бұрын

@@Sgirlie755 no

@drugdog-qp8ww 6 ай бұрын

How would i bake the motion from driver to keyframe

@cgtinker 6 ай бұрын

baking works the same as usual, just bake to visual

@juvenilenayem 8 ай бұрын

Does this system work with IClone character?

@soitk1825 11 ай бұрын

Will BlenderArMocap work with Mixamo Rigs?

@cgtinker 11 ай бұрын

technically yes, the tools are there to do so within the add-on but I don't create custom configs

@manuelbarrios2732 Жыл бұрын

Is it possible to transfer the data to another armature? like auto rig pro or mixamo for example?

@cgtinker Жыл бұрын

should work, yes, for autorig it may be sufficient to just change targets without touching the drivers. (I don't have autorig but it seems to use rigify).

@misha4sculpt Жыл бұрын

@@cgtinker hey...thanks for the update...I want to try to create a target rig for the metahuman rig...let me know if you have an outline to get started or any other suggestions....if I can follow an outline for the rigify rig, let me know...

@cgtinker Жыл бұрын

@@misha4sculpt the rigify config delivers are fairly nice outline to do so. Most should work by just switching targets

@misha4sculpt Жыл бұрын

@@cgtinker where do I find that...

@misha4sculpt Жыл бұрын

..never mind...found it

@hrishis Жыл бұрын

Just curious. What linux distro is that ?

@cgtinker 11 ай бұрын

manjaro

@ok-hc4he 18 күн бұрын

How do you add an actual model to all this

@GrantMiller 21 күн бұрын

the app is gone from the iphone app store :( what happened?

@haganji Жыл бұрын

How can I export fbx file include animation from the rigged file? Actually I tried it, but every time I failed...

@cgtinker Жыл бұрын

You got to bake before exporting. Where you want to export to?

@haganji Жыл бұрын

@@cgtinkerHow can I explain you where I tried to expor to... I want to explain you with picture, because I cannot explain you exactly. Let me know your email address please...

@cgtinker Жыл бұрын

@@haganji hello@cgtinker.com

@haganji Жыл бұрын

@@cgtinker I have sent you email. Check it out please.

@cgtinker Жыл бұрын

@@haganji I'll :) give me 2-3 days

@kendarr Жыл бұрын

When I use it to make face animations, the animations is too smooth on the character, why idea why? it moves very little

@cgtinker Жыл бұрын

probably you have to increase the influences. you can change up the settings of the mapping objects and should be fine

@kendarr Жыл бұрын

@@cgtinker Will try that thanks, I scaled the empties to a large scale and that 'fixed it', but the tracking doesn't really convey the lip motion super well (on my model atleast)

@cgtinker Жыл бұрын

@@kendarr sometimes it requires some cleanup / change of settings - planning to improve the face transfer results in the future. If you go in the "cgt_face" collection you will find empties named something like "cgt_lid.R.L" - in the object data props (where the constraints live) you'll find custom settings which help to improve the mapping