9 years later and this video is still amazing

I would think there would have been actual 3d animators watching this and posting comments, but I don't think anyone here sees the mess they would have to clean up if given the raw output. It would probably be the same amount of work if not more work then doing it from scratch. The advantage, is with the amount of work input, you have the potential for far greater output because the broad strokes are done. This is a BIG win for 3d modelers, and an outstanding tech and intellectual achievement.

That real time colouring is the coolest thing ever. Physics in 3D camera mapped area also amaze me. These talented guys should be working for Microsoft.

I have to say that the most impressive part was when they took the tea jar away and the modeled one remained in the real-life footage. That was mind blowing! This is HUGE; my best congratulations to this team; This is amazing technology

Imagine this integrated with high end Movie cameras. I have no doubt that special effects would take a giant leap forward.

reminds me some of the things that the 3ds could do, but that was only scratching the surface. This is way more in depth

I have nothing but admiration for the people who coded this. Fucking insane.

This is the best video I've ever seen. I'm getting chills. THIS IS THE FRIGGIN FUTURE

That's incredible, congratulations!

Incredible! Congrats

this is some pretty futuristic stuff

Very impressed. Look forward to seeing more of this.

I think the RGB data could be used to acquire besides color of the surfaces, also the shininess (by analyzing how the color and brightness on a specific spot changes when the camera is in different positions) and perhaps even produce a rough estimate of the size, position and shape of the light sources (by analyzing the bright moving areas in otherwise non-moving surfaces in different camera angles and project from the different surfaces to find the most consistent model for the lights sources)

i think that this is the step to new era, where the car could drive alone and interpreting the volume of the roads, is only the beginning.

awesome awesome stuff... the fact that all this stuff is real time... real wow factor...

That's fantastic! :D

If augmented reality kicks off, say with google glass and they use this software with their camera(s), couldn't we build a virtual copy of our world down to extreme details..feed that into google earth..along with oculus rift = travel the world virtually. Urban areas would be mapped in no time, but even nature would get modeled gradually. Cool!

Mindblowing!

Two words. Mind. Blown.

This could do wonders to autonomous vehicles in terms of anticipating dangers or at least warning systems for human drivers. (Oh, and of course this is one step closer to that construction interface Tony Stark has in his tinker cellar :) )

Awesome!

Absoultely awesome :D

this is great. Now I can model myself working and go play xbox all day. Kinect FTW!

@cruxphotography , I agree with you that this is no replacement for 3D modelers. But it can help designers/artists/engineers model real objects much faster and more accurately. Imagine if I want to place a car in a game I am designing. Instead of designing the car from scratch, I can scan a model of my toy car and import into the game in no time. That amounts to saving 2 to 3 hrs for modelling a car.

@Bireno This could do a ton for modelers - any model in there could either be retopolized by hand or by software (FYI Modern automatic retopology/polyreduction software lets you interact with how and where details is reduced) But as you say for tracing; The kinect model could easily be used to transfer a perfect RGB/diffuse and normal map to an optimized object/UV. This could all greatly help modelers, it just wouldn't be very suitable for anything not realistic though.

Insane

@Bireno FYI: 3D modelers often use incredibly high-poly models to create normal maps, which do not affect the poly count (the polys come from the underlying model that is actually used and then normal-mapped). ZBrush is a common tool for creating these high-poly models, but this (I assume) may give yet another alternative. Just a FYI for your FYI.

Amazing chance to experiment with virtual reality

insanely cool :)

just awesome, unbelievable somehow. Robots will soon be much more advanced than us.

imagine: scanning of an entire city with kinect only! fly through them, in real 3D. amazing!

Lo que kinect va a lograr en el futuro es algo muy superior a lo que podemos imaginar

Watching the part with 3d objects w/ physics superimposed onto the real world made me think of only one thing: 3d zombie survival FPS in my own neighborhood....

Very impressive.

omg the texture mapping !

Godlike!!!

This is great

amazing!

@Bireno I do think it would be extremely usefull for people compositing 3d in liveaction. Less rotoscoping work. And it would be perfect reference scene for 3D animators.

And this was 4 years ago ladies and gentlemen. I can't wait for Q1 2016.

exciting :D its really getting there.. now we just need to learn how to apply this in many interesting ways

this is amazing

FYI: This isn't going to do much for 3d modelers. The model geometry isn't going to be efficient enough. Basically, models for games are carefully created with a polygon budget in mind, detail where it needs to be, less where it doesn't. Proper modeling on joint areas for animation, etc.. While awesome, this software isn't going to really help streamline games aside from offering something to 'trace' in 3d with a more efficient mesh.

@brainiacgames it models everything it sees. If it never sees the back, then it never models it. It builds up a 3D image in it's mind as the kinect moves and rotates arbitrarily around the scene, much like your own brain's visual system does. The technique isn't technically new, it's called "SLAM". What this video shows is subtle new techniques to improve how SLAM works.

Impressive. o.o

@arashsammander The bacon sandwich won't be useless; it will be an excellent high detail model from which to bake normals, ambient occlusion maps and albedo onto a low detail bacon sandwich that you model yourself. Standard game industry software like Z-brush has good facilities for re-topologizing(by hand) a low definition model from a highly detailed mesh. Going the other way is much harder; starting with a low poly bacon sandwich and sculpting a high detail bacon sandwhich.

Why does it capture the DELL logo but not the cables or the bars from the monitor's heat exhaust?

@GreatVomitto Make this technology work with video glasses that have a camera. Then wherever you're looking is scanned and recreated digitally. CG characters could be added that move through the environment similarly to the particle ball effect. You'd be looking at live video of your house but suddenly a cg girl appears that you can see interacting with the real environment and your "multi touch". : P This is really impressive tech and a big step forward for virtual reality.

Most of the time the video shows surface normals which are just depth derivative and lighting which is just a cosine with these sruface normals. One can write these operations in shorter amount of time than you spent on showing them. Also, the segmetnation based on ICP outliers requires a separate registration beforehand. People are more interested in real-time registration nowadays, so it will be nice to see more of that.

Holy CRAP!

Its called re-topologizing the mesh, and their are algorithms for that. The artists will just need to clean shit up afterwards...

Please, post to vimeo! That's where all the technical directors and researchers lurk!

In a couple of years, compress this technology into a couple of glasses (and hardware unit, if needed) and you got yourself constant AR! (Though chances of getting run over by cars that the software didn't properly update while running around town zapping AR aliens = fairly high)

@dillzy1986 Actually you will get a mesh with millions of polys that you will have to spend forever to clean up so modeling a bacon sandwich from scratch is easier, and also more efficient.

Now you can create a virtual mess in your living room Nice! :)

The future: Put on some virtual reality glasses, connect them to your smartphone, play augmented reality multiplayer shooters with your friends. Imagine playing Left for Dead in your own house, actually running around with guns that appear to shoot real bullets. Fuck keyboard + mouse. Or how about dining with your friend who is on the other side of the world? His 3d model appears in your vision, you can hear and see him. Or playing card games that actually come to life?

I think people are missing the point of this kind of work. It's not meant to be a replacement to 3d modelers, it's a break-through in how computers make sense of OUR environment. With known surfaces, walls, objects, etc...we can start to extend the computer interface beyond the mouse & keyboard and into a new world of computer vision where we can detect the movement, position and rotation of any arbitrary object, turn surfaces into an input devices and finally get away from our desks and chairs.

I had a mind organism from watching this

Hello! Is there any way to do 3D reconstruction width Kinect Fusion in MATLAB? I already obtain RGB and DEPTH image from Kinect in MATLAB. The reason is that I have to obtain thermal image too, with a FLIR Thermal Camera I have here, and also works in MATLAB. Thanks!

this is exiting! i wonder, when the book and lid was moved 3D reconstr updated but didn't show the movement. is it something that can be solved?

Hi, I'm going to start a pretty big project and I was wondering if it would be possible to place a 3D object and have it display over the real world? Like what you guys did with the yellow balls but "permanent" ? I was also wondering how this would be done? Thank you very much :)

@GreatVomitto i thought it was clear by firing particles into the scene... you can constantly be bombarding the actresses with various fluid simulations throughout the duration of the movie.

Hey man, this is some video! I am looking to start a project with the fusion but I am not sure which graphics card I should use. Which one did you use?

I'm presuming this is outputting a voxel system rather than mesh geometry. Then perhaps there's some auto-cleaning to select the best 'surface' voxels', hence optimised speed. beyond that, it's possible to consider building a mesh, but I don't know whether that's been made feasible in real-time here?

I totally understand this tech. This can open up manufacturing on a whole new level. Antiquated parts can now be 3D scanned and replicated in CNC for a fraction of the cost before. I just wish someone would get around to making it work with microsofts bastard child, vista. It would be nice to have this work on all windows platforms from XP and up.

next thing for it will be to recognize a kettle-like surface is the kettle and start to interact or animate it…

@arashsammander lol, outrageously simple to reduce the poly count in 3dsmax

Reminds me of playing rainbow six veagas

@MisterTVenables Microsoft research in Cambridge is one of the contributors to this project.

Dude! The applications for the are endless

@wonmeanlee this is going to drive Ikea nuts. All these people running around their stores, holding kinects and laptops. RELEASE THE OTTOMANS!!

Think of this for crime-scene reconstruction / investigation.

this will change the games and de internet chating...

Wow, I laughed at the augemented reality joke apps but this video is just scary lol. Gz to you guys.

love that comment and it is, still difficult for us poor soles with imagination, only limited computer skills, when microsoft will not release anything. You would think after the massive success of all the open source so far.

thank you

@XantheFIN Oh, I seemed to miss the opening intro.

Is this scene generation and physics being done real-time or is the input saved and we are looking at something preprocess?

@wonmeanlee "A boon to game makers. :D :D" Yeah and I can imagine that this could put a lot of 3D artists out of the job!

any chance of including the rgb data for real time texture? im looking forward to the release in future sdk i can think of a million uses

Be Kind, release it.

I have a Kinect v2 and I am using the Kinect Fusion sample on it which is included with the SDK 2.0, but I don't get so good results as in the video. My reconstructions are all messed up if I move my Kinect. It is always nice from one point of view, but it gets messed up as soon as I move. Any ideas?

Other than a kinect, what do you need to do this? I'd LOVE to reconstruct my environment in a similar way!

This is not technology, its magic :-o

un. fucking. believable.

Does anyone know if this software is open to the public like some of the other kinect stuff shown in the past?

@denilsonsa Data takes space; point cloud data takes huge freaking gobs of space. If you are content with a small number of models repeated infinitely you can render "inifinite" point-cloud data. If you want a real game; something you'd actually want to play rather than a tech demo, you'd want at least to have some sort of hybrid approach that uses polygon models for animated objects, which point clouds don't deal well with at all.

quite nice. does the kinect 3D work in sunlight?

this is a fucking win for 3d modellers. get a lazy susan, put your kinect next to it, spin round object. BAM. you have just modelled a bacon sandwich.

сдаётся мне всё же не миновать нам терминаторов и матрицы !

Any updates on the scan resolution of Kinect 2?

That's really impressive. Is there an essay or thesis about this topic, its development or the results of the demonstration? I'd really like to read it!

@Bireno Naysayers gonna naysay.

@Bireno I dont see why you couldn't retopologize any of the objects captured with the Kinect easily. There are plenty of professionals in the industry using sculpting software to make and edit high poly models with poor/temporary topology (similar to the output in this video), then retopologize them for animation and poly budget later.

Is it not just doing the front shell? Hence the resulting images not being spun.

This is amazing dude! Great work, are you realeasing the source? Or do you have any project page?

@kennethsonnek my muscles and glasses say yes

is this some kind of technology for assassin robots.

omfg is there possible to make reconstruction of my home and put it into gta ?

i'll be impressed when i see the wire meshes. bet the geometry's a mess.