Painterly real-time rendering

Painterly real-time rendering | Useless Game Dev

Рет қаралды 51,069

8 ай бұрын

Join me in this disastrous adventure as I try to make a real-time painterly rendering filter for a video game, there will be brushes, there will be compute shaders, there will be tears. Fun times all around.
More reading/watching on this subject:
- Interactive Painterly Stylization of Images, Videos and 3D Animations: gfx.cs.princeton.edu/gfx/pubs...
- GDC Talk A Living Painting: The Rendering and Art of 11-11 Memories Retold: • A Living Painting: The...
- Game Dev Guide on getting started with compute shaders: • Getting Started with C...
- Painterly Rendering in Blender: • Turning 3D Models into...
Music:
"Wholesome" Kevin MacLeod (incompetech.com)
Licensed under Creative Commons: By Attribution 4.0 License
creativecommons.org/licenses/b...
"Beauty Flow" Kevin MacLeod (incompetech.com)
Licensed under Creative Commons: By Attribution 4.0 License
creativecommons.org/licenses/b...
"Wallpaper" Kevin MacLeod (incompetech.com)
Licensed under Creative Commons: By Attribution 4.0 License
creativecommons.org/licenses/b...
"Montauk Point" Kevin MacLeod (incompetech.com)
Licensed under Creative Commons: By Attribution 4.0 License
creativecommons.org/licenses/b...
Other Credits:
- 3D assets by Kenney: kenney.itch.io/kenney-game-as...
Support the channel on Patreon to get extra content, and access to the source code of all projects!
/ uselessgamedev
Follow me on Twitter, not that I tweet that often: / uselessgamedev
Kappa Avatar art by Boskoop www.artstation.com/boskoop
And as always, have a good one!
#Painterly #Rendering #Unity #GameDev #IndieDev #devlog

Пікірлер: 103

@RooMan93 8 ай бұрын

I imagine the balance between temporal coherence and artistic style is the biggest challenge.

@uselessgamedev 8 ай бұрын

Yes, that's a big point in the GDC talk. Their final style doesn't even look that painterly at time but I guess that's the price you have to pay for consistency and readability

@AnmAtAnm 7 ай бұрын

May not have been your target, but I love the look of the scene repainting after the camera moves. Would be great for a game that encourages slowing down and taking time to observe.

@Eishous 8 ай бұрын

I like the approach of splitting the image up into different layers of detail. It's kind of reminiscent of the actual painting process where you'd start with midtones blocking out the rough shape and then paint over them for the details. As always great video!

@ChristinaCreatesGames 8 ай бұрын

A really nice looking effect! I study color and the first time I realized that I could actually turn the values I used day to day into number values to do math with, it certainly opened up a whole different layer of understanding what I chose as one of my majors :D

@MattyDoesGameDev 8 ай бұрын

I think this shader could work really well when paired with a depth texture. You could use large, maybe even softer, brush strokes for the background, and increase the fidelity of the strokes for foreground objects. As always, phenomenal video! Thank you for sharing

@embergamedev 8 ай бұрын

This is incredible for only a few weeks of work! Definitely inspired me to start exploring compute shaders.

@gamesplusjames 7 ай бұрын

This is such a cool effect! I think the way it struggles when moving at the end would actually be fantastic for a dream sequence of some kind in a game. Great stuff as always!

@jebstalp 8 ай бұрын

Love those videos. Really like the artistic aspect of those rendering methods. Continue the good work ^^

@DroolRockworm 9 күн бұрын

Dude you are probably the best video game channel on KZfaq and that's saying a lot given the thousands of copycat, trash clickbait unoriginal buffoons that post on here. Thank you for all your hard work and you deserve 100x more subscribers

@lucariette 8 ай бұрын

Glad to see you didnt stop making videos, love your work

@snarfum 8 ай бұрын

I really adore your experiments and find them really inspiring. Thank you for sharing you explorations.

@uselessgamedev 8 ай бұрын

Thanks that's very nice!

@circamagoo 8 ай бұрын

amazing! this marks a new era in game/interactive art dev

@danilucky1188 8 ай бұрын

First! This is genius! I love your vids, continue like this :)

@DanaTheLateBloomingFruitLoop 7 ай бұрын

This is exactly what I was looking for! One thing that might make moving the camera less painful to the senses it to break up the painting into smaller pieces that are angled to the world geometry, which can stay inplace for a while. Distant objects could be combined into one painting while the closest objects get a little painting each. When the view changes enough a redraw and re-combining may be necessary. I hope this makes sense...

@whud99 8 ай бұрын

I have zero experience with shades, but for finding the angle: Could you use a depth map, then have the strokes try to minimize the difference in depth beneath them. For example a long stroke on the horizon if placed vertically would have a huge difference in depth from top to bottom but placed horizontally it would have very little difference. love your videos btw and I'm glad you are keeping up with them ♥!!

@uselessgamedev 8 ай бұрын

That's an interesting idea but maybe it would work even better using the normals? With a 3d rotation?

@AlexKDev 7 ай бұрын

Looks beautiful! Great work :D

@HeyGrouch 7 ай бұрын

I love this!! Such a cool effect!

@GabeFrank 7 ай бұрын

using a sobel filter to find image directionality is really close to a solution I see a lot of shader devs using: if you take the results of a sobel filter (X gradient and Y gradient), you can rearrange it into what is called a structure tensor: X^2, XY XY, Y^2 for which the eigenvectors describe the perpendicular and parallel directions of that texel's gradient, and the eigenvalues correspond to the magnitude of those directions. Acerola's video on Kuwahara Filters uses this technique, which is where I first learned about it

@uselessgamedev 7 ай бұрын

Interesting. Thanks I'll check it out!

@gnomoverdissimo 8 ай бұрын

Awesome video, thanks for sharing! Now i really want to try implement it!

@n00dles4 7 ай бұрын

Far from useless! I found this extremely interesting and I appreciate the work you put into this

@Youtubershacksshifat 8 ай бұрын

this really looks good 👍

@GonziHere 8 ай бұрын

Yeah. I've seen that talk way back and it got me thinking... It could be maybe used to something else entirely? IDK what, but it's applying "shader look", but in 3D... Random example would be smoke effects. IDK if it would be better or worse than normal smoke effects, blood splatter effects, etc. but it would surely be unique. Love that you are toying with it :)

@qwqeqrqtqz 7 ай бұрын

I recently saw a video by Cody Gindy who used a complely different approach to get a painterly look in blender. It's his only video so far, but the results are very impressive. His method relies on painting over object space normal maps and the base colour, then using realistic lighting. This effectively clamps the normals to discrete values in the shape of brush strokes, but the strokes interact with the lighting. I think it should be possible to use this for real time rendering in games as well. Maybe your next project?

@a52productions 4 ай бұрын

The process of splitting the image into different layers of detail, and then moving those patches around, reminds me of how video compression works. I wonder if Fourier analysis would be helpful at all for this application...

@heavygaming6596 8 ай бұрын

Amazing video

@Leviathanshadex 7 ай бұрын

You did really well with the filter, even if you weren't able to get it to update as quickly as you would have liked.

@anon_y_mousse 8 ай бұрын

That looks pretty cool, and I like the larger strokes. The only thing that needs improving is when you move the camera. Definitely something to consider as I try to decide the art style for my own game as I'm definitely not going to want realistic graphics.

@TJGameDev 7 ай бұрын

Loved the way you built off the sobel filter from the previous video. The explanation of compute shaders was good as well. Ive always heard of the cpu being a university student, and the gpu being a classroom of middleschoolers. Who would get done a task faster depends on the complexity of calculations

@uselessgamedev 7 ай бұрын

That's a nice analogy! Although I assume the GPU is just as "smart" and capable as the CPU, so it's probably more akin to a class of university students except they have to wait for everyone to synchronize after every word they write

@matthewmathis62 7 ай бұрын

I wanted to do this task once, as well. I actually like the effect of just the small strokes at 6:36. Maybe because it is consistent between frames. Now, I haven't finished the video yet, UGD, but I think to make this effect really worth using in a widely-distributed game, I think you should make the math for how the strokes are placed very accurate, and based upon the geometry of the surface, so that no matter where you are looking from, the texture always stays in place. This way, it wouldn't move around as you move around, and it would be beautiful to look at. Now, I don't know if my comment will offend you, because maybe this is difficult to do and you don't want to do it. I don't know. But I think it would make for a beautiful effect. But what you made in the video (at 6:36, because I have to leave at the moment) is cool! Great job, and Good luck! 🙂

@cappntone 7 ай бұрын

(I am not a programmer) A potential solution that immediately comes to mind for me is heavy usage of shader mapping. I think using a depth map (or z buffer??) + an outline map (like in cell shaded games) + a detail map, you can create the effect you might want. The outlines can define the edges of objects and let the shader know where to stop painting to make sure that detail is retained, and you can tell objects apart even at a distance! It can also define the 'paint strokes' to make with your brush! Maybe you can try making the strokes follow the outline and offset to the center of the object to create a more 'painted' look and less 'paint splat' look. You can then use the depth data to draw the strokes from furthest to nearest in order to make sure proper depth is retained. Then, with the detail map, you can let the shader know when to draw with smaller strokes when you need more detail. The idea of using the depth data to create less detail for further objects is cool! The biggest thing in my eyes causing this to fail when in motion is the use of noise. Also! It may be okay to leave gaps in the shader, make the shader transparent and draw it over your scene in order to fill in any gaps and add a little detail back into the image. Again, not a programmer, so I apologize if I got anything wrong, I just had some thoughts that I hope might help!

@GodEmpressZoe 8 ай бұрын

Heyyy I have been watching you for months now and you know what? I will stop just watching and start doing! I'm starting with this one and I'll make a small narrative game with it! Thanks for inspiring me!

@uselessgamedev 7 ай бұрын

Oh no please keep watching Jk have fun experimenting it's very fun Good luck!

@GodEmpressZoe 7 ай бұрын

HSFAGFIAGFIUA. dw I'll still keep watching and since I'll be taking ideas from your stuff probably more so now

@morgan0 8 ай бұрын

something else to consider is painted normal maps, like the absolute normal for the vert not the change from the surface. basically you paint brush strokes of flat areas, which then react to light normally. i saw it done by hand, but that could probably be done automatically, and each brush stroke could have a random value in a map used for color offsets and other stuff. it wouldn't stick off of stuff, but it would at least make each object already look a lot closer to the style, and then maybe hand placed (or again using code to figure out where the strokes are to place on the surface) quads which contain a brush stroke texture that sticks off of it.

@danielrachbauer2159 7 ай бұрын

Im a 2d artist and i think it looks really awesome! I also really like the repainting effwct when the camera moves, but ofcourse that would make it unusable for most games But it looks reeaaly nice!

@BalCleric 7 ай бұрын

Anybody remember an obscure old proc gen MMORPG called Love from way back in 2010? That had a nice painterly shader, I always wanted to see more games take it up.

@bread8700 8 ай бұрын

this guy is awesome

@thomasnthlc4180 7 ай бұрын

Sujet intéressant et magnifiquement traité Genial

@levifzephyr 7 ай бұрын

Owww that's quite frustrating that you weren't able to go further, but that's right I totally understand your time limit. IMHO, you shall keep digging this rabbit hole a bit, it was very uncommon to watch and think about. And take care of yourself, don't over load 😉

@LandmineSoldier 8 ай бұрын

It's very impressive. I think someone can make new cartoon artstyle shader with this idea

@silverspuppet 8 ай бұрын

I really shouldn’t watch these videos. They are so cool, I try to recreate this and spend like a month on it instead of actually making any progress on games

@uselessgamedev 8 ай бұрын

That's precisely why they're here though, so you can sort of live the experiment through me without having to actually get your hands dirty

@Celestinal-lc5fy 7 ай бұрын

trop joli trop beau

@shooterdefronvrps2 7 ай бұрын

The way I know to make a painterly look is the kuwahara filter Its more blurring the image in a way that resembles a brushstroke than placing a stroke but achieve a similar effect

@Aerobiker1 8 ай бұрын

For the last problem, couldn't you use the motion field of the pixels to shift everything and not need to redraw that much ?

@uselessgamedev 8 ай бұрын

Yes, that's what 11-11 does, from what they say in the GDC talk, apparently Unity's motion vectors is enough for what they need. However it does require keeping track of every stroke and their parameters to shift them, which I don't do

@hardtofinduniquename 7 ай бұрын

Maybe another way to improve the effect as the camera moves is to use world space noise instead of screen space. If your shader was the material for each object in the world, then the calculations you do would be pegged to its position and be preserved regardless of the camera angle. Not certain it would work. Regardless, nice video!

@danielrachbauer2159 7 ай бұрын

It also kinda reminds me of impressionism - which got me the idea that you could try slightly randomizing the hue of each stroke to give it even more live and counter some of the sterile nature that cg renders often have

@uselessgamedev 7 ай бұрын

I do this not with the hue but the brightness. There's about a 10% random range on brightness otherwise the brown table in the fruit basket scene would be completely uniform. Hue is nice suggestion though maybe I should have done both

@danielrachbauer2159 7 ай бұрын

@@uselessgamedev awesome!

@Keimoj 7 ай бұрын

I wonder if you could do a rendertexture "skymap" that keeps traks of surroundings :D

@lephtovermeet 7 ай бұрын

1911 has a similar art style. I like te concept a lot but I wish things would remain static when the camera or actors in view arent moving. I think it would be much more beautiful if only motion caused the painting to kind of twinkle like that, then once stationary again, the image is static again.

@no-lifenoah7861 7 ай бұрын

Id love to see a followup to this in 6 years when its done

@jamessoth1479 7 ай бұрын

Super interesting, would a Kuwahara filter be more appropriate for the affect you're going for? Despite the performance hit.

@uselessgamedev 7 ай бұрын

Not sure. It's just a different effect

@felixinfinita3777 8 ай бұрын

Maybe the easy way to make the strokes persist is to do decals/particles. Like the camera initially shoots raycasts and you place particle with the appropriate color of what you hit at the world location (or maybe attached to it so the base object could be moved together with decals). That would be a completely different implementation without doing post-processing of the base image, but something to consider :)

@uselessgamedev 8 ай бұрын

Yes either way storing the list of strokes/particles would help moving them in the next frame instead of reprocessing everything from scratch!

@qfurgie 8 ай бұрын

Awesome video! You should team up with youtuber Acerola for a video! I think it would be a super cool collaboration.

@TheQxY 7 ай бұрын

If you can find a way to turn each R G B into a vector flow field which flows from high intensity to low intensity, then you can add and average these flow fields together to get the final ve toe flow field and use that to determine the brush angle.

@uselessgamedev 7 ай бұрын

Yes that would be ideal but I have no idea how to do this. Maybe something with gradient?

@raulzilla Ай бұрын

What if you use multiple cameras, each rendering a desired layer with this effect? It might be the solution for moving and every stroke getting blended and blurred together.

@minedantaken1684 7 ай бұрын

I have one question/idea. Making an outline shader that looks more pencil drawn. A bit jagged, not entirely even, maybe animated? Every single outline shader/toon shader I've seen does them perfectly even and smooth, to the point where I don't know if it's possible to do it differently

@uselessgamedev 7 ай бұрын

Yeah if only there was a video about jagged outlines on this very channel :O Although something I don't show in the video is the "not entirely even" / damaged feature which I did add but didn't look so good

@trolleymouse 8 ай бұрын

Orientation of strokes could take into account normal screenspace. And if you can figure out a way to create these screenspace textures on your own, you could use make some kind of "focus texture" for objects that's lighter in important areas that it can read off.

@uselessgamedev 8 ай бұрын

Interesting idea, thanks!

@OnWeek 7 ай бұрын

And what it's look if instead of pos processing the effect, you use this to bake the textures, mave doing a second mesh with the brushs in front of the normal mesh

@NeoShameMan 7 ай бұрын

Here is another version, just sample around a pixel, using a sampling pattern randomly or base on thebpixel values, and pick the best pixel according to some rules, for examples take the highest luminosity and apply to current pixel. The end. Sequel, do it but with rejection rules aka do nothing, and per mip map before blending down. No need for complex shader.

@notTryio 7 ай бұрын

how useless could a gamedev be if his avatar IS A GODDAMN KAPPA :D I LOVE KAPPAS

@ZooHair 8 ай бұрын

Art

@benverschoor6521 8 ай бұрын

I dont think you would need to make the 'brushstrokes' consistent much, if at all. It lends credibility to the illusion of the moving image being a series of paintings, which would have different strokes. You can see this in the movie Loving Vincent, which was actually made by oil painting thousands of Van Gogh-esque animation frames. Maybe the needs of a game would require something more stable, maybe for twitch reflex hit box stuff, but as an aesthetic I actually like it more than something silky-smooth.

@bluesillybeard 8 ай бұрын

Here's an idea: create a screenspace normal map by deriving it from the depth buffer, then use that to get the direction of the stroke.

@uselessgamedev 8 ай бұрын

That's a good idea yeah, with a 3D rotation maybe even. You don't even need to derive it from depth, unity does provide the world normal map

@ecruz2317 5 ай бұрын

kewl

@Psilocervine 6 ай бұрын

I think you might be able to get a more cohesive look (not temporally, but just in general) if you were to have some sort of directionality affect the way the brush strokes are drawn. You can probably modify sobel edge detection for this and use R and G as direction components depending on where the edges are detected. There'd be some overhead to be sure, but you'd get some more defined edges which could help the composition a whole bunch

@uselessgamedev 6 ай бұрын

Hmm I'm not sure I see what you mean because what you're describing is exactly what I do for brush stroke orientation

@Psilocervine 6 ай бұрын

@@uselessgamedev There is a VERY distinct possibility I missed something in the video. I was swapping between devices and youtube's playback resume can be a bit flaky. Sorry about that! edit: I rewatched the video from start to finish and you sure did! Now I'm even extra sorry, I left one of THOSE types of comments

@uselessgamedev 6 ай бұрын

Ha no worries!

@mcpepa 7 ай бұрын

Hi, may I ask why not use Fresnel to get the exact angle of the object to paint?

@uselessgamedev 7 ай бұрын

This is a screen space shader I don't have access to this information. I probably could put the info (either fresnel itself or just 3D stuff) in a texture and then use it in the screen space shader but as is I didn't have it

@kalpolproductions8558 7 ай бұрын

bro knows how to make me click on a video.

@SA_learner 8 ай бұрын

how to learn this computational texturing ? and will it work in unreal

@uselessgamedev 7 ай бұрын

The feature surely exists in Unreal however I don't know Unreal so I can't point you to resources to learn

@atomictraveller 6 ай бұрын

dreams ps4 used a fleck graphics system developed by sony in the early 2000s, assigning alpha "stroke" quads to define the objects (and a secondary 2d "paint" object). really lovely, works to excellent effect, but the entire dreams experience was dominated by Mm staff being all witchy and hating folks not fitting their agenda.

@N00BleSouP24 8 ай бұрын

Is the same possible in UE5? 🤨

@uselessgamedev 8 ай бұрын

Sure! I don't see why not

@matt92hun 7 ай бұрын

Instructions unclear, got a stroke.

@KiriRose 8 ай бұрын

this would be cool if you didin't have a hotbar or anything, and you just follow where the painting has higher detail to go to reach your goal, like the painting details are your compass. and then you could have a level where everything is the same detail as in you've reached your goal, now what. or some philosophical shit like that

@uselessgamedev 8 ай бұрын

yes, that or just depth of field blur/focus to emphasize where to go. There's a lot of possibilities to explore

@reguret2976 7 ай бұрын

boids!

@linuxrant 6 ай бұрын

I think the techniques you used with the hand drawn video could prove to be useful here to draw some details, your approach I think was a little better there I think because you did combine different techniques, the edge detection, crosshatching, gradients together. I saw a very interesting videos of guys trying to do this in blender, with much better results. they also tried couple of different techniques. One that caught my attention, as a smart, ingenious, slightly cheaty way of simulating paint was to distort the normal map(they smudged the normal map in photoshop). So we do not REDRAW the 3D scene. We draw in the 3D objects and on top of these objects. kzfaq.info/get/bejne/qZ5-Y5Ojz8_Qj2g.html This looked cool on still frames if you ignored the sharp edges of objects. To fix that we could generate billboard sprites on just the edges of objects with paint splats. Properly angled and curved if possible. Lastly we could fix somewhat the temporal problems by playing around with animating the painted normal maps, when the camera moves and every 2 or 3 frames. I would gladly work together with somebody on this, but I am a shader illiterate. I have no idea how shaders work, visual or text, But I will learn them eventually.