(Referring to DF Indirect) - "possibly more laughs" I'm glad you left room for the chance of there being no laughs

I also judge the quality of a hole and then use one of two techniques to fill it.

Fun fact, the reason why we use a checkerboard pattern is because you can derive a much better pixel in situations where you're unable to reuse a pixel from the previous frame (around 90% of pixels are reusable from the previous frame, but you will run into situations where you can't reuse a pixel as the pixel you want simply isn't available). Since you have all 4 surrounding pixels actually rendered, you can simply average those 4 pixels to get something fairly believable out of it. And of course in the next frame you will actually render that missing pixel as you alternate the pixels being rendered from frame to frame. So every 2 frames you render to every pixel on the screen, so usually any issues are resolved in the next frame.

Finally, 8K resolution on consoles... upscaled from 720p

I love how we evolved from analog interlaced video to progressive video, and now we're regressing to what's essentially digital interlaced video...

Dont know what alex did to his microphone but the annoying noise is finally gone

The different size holes idea sounds a lot like foveated rendering for VR, where the edges of vision would get increasingly lower resolution. Especially fitting if you remember that it was London studio that made Blood and Truth for PSVR

Talking about the quality of holes with a straight face, kudos

I feel there was a missed opportunity to call this show DF notionally-direct

Bring back the hardware checker boarding. The 4k quality on ps4 pro was better than the 4k on most current gen games.

I get why youre putting this behind a supporter programm, it does make sense. But it sucks to have more of my favourite type of content paywalled when I really dont need more monthly expenses now

A lot of hole filling in this episode!

Cone on guys, that'a a patent from October 2021. It's OLD. not new. The date is in your image at 12:26.

Even the patents that are eventually implemented have a lot of smoke and mirrors to confuse the competition and also they are as vague as possible so in actuality it's very difficult to make out what's the patent about other than the title. I have filled up patents for a company and it always surprises me what the patent attorney makes with the draft that is provided to them 😂

"Holes" of different sizes are generated when performing frame generation - 3D objects obscure things behind them, and what they obscure changes as they move or the camera moves. If you want to generate a new frame where part of the background isn't visible in the input frame/s then you need to "fill holes". So I think this is likely to do with frame gen rather than checkerboarding.

Lotsa hole filling discussion in this video

My theory: they describe a multi pass render. They render at f.e. at quarter resolution and then render additional pixels based on local contrast analysis.

Can't wait to see more on this.

I haven’t seen the patent, sorry, but is there a chance this is for their TVs?

The description make it sound a bit like focused rendering (like on VR eye tracking) for normal flat images. In example, the "holes" would be selective areas of lesser interest or detail, and scaling would be applied mainly there.

hole filling with various hole sizes: can be used as a sync option to always hit e.g. 60fps. * render the frame at (somewhat) randomized pixel locations * as long as you still got time, render more * once the frame needs to ship, fill those variously sized holes (eg like this patent describes) Never drop fps ever again. And the more power you got, the better the picture (more natively rendered pixels). I am assuming a lot. And ofc, how the pixels are chosen certainly also can be done beter than at random. eg more towards center / based on prev image / where current already rendered pixels suggest a high local contrast ..

I don’t know why they didn’t bring Super sampling from PS4 Pro over it was so good

Dont know if i have a subscription in my budget right now, but i really liked this deep dive style and am certainly contemplating joining for it.

Idea makes sense. Checkerboarding is a pretty rigid idea of what you don't render. Great way to save pixels, but you don't have a lot of options for reconstruction methods. If you increase the amount of "missing" pixels, into small groups, you now can have enough information inside each "hole" to maybe do something with. If the missing pixels are "simple" enough, then you can use some basic (cheap) inferencing algorithms on how to fill in the gaps or "reconstruct" the holes. For the more challenging gaps, you can use AI much like with DLSS. But rather than training the network on taking "complete" low res images and upscaling them to a higher res, you instead train it on "incomplete" high res images with holes, and getting the network to fill in the holes with (hopefully) accurate data. As long as the network is equally good (If not better) than the DLSS style complete image reconstruction, then this method has benefits as you can save on the amount of AI work (and required hardware) by only using it on the "hard" holes, which means you don't have to always AI reconstruct 50% of the pixels every time for example. I'd imagine it wont' be better than Nvidia style DLSS, as Nvidia is smart and has a lot of people working on this and so they would have likely tested this (and a variety of other) alternative methods and the one they ultimately chose for DLSS is probably the best one they found. But if it works good enough for a console, and more importantly if this software patent can "protect" their method as being a unique selling point in the console space, then they will be happy.

If DLSS and Checker-boarding had a love child:

Is bimonthly twice a month or once every two months?

My hope is that this is a new way for texture streaming to work, somewhat like nanite or micropoly. You mention that it sort of looks like an image up scaling algorithm, but maybe it’s a dynamic texture resolution scaling algorithm?

what i liked about cb is that though it didnt look as good as dlss the frame time usage was very minimal in comparison even with the tensor cores dlss is very heavy

Dope pics indeed!

So it's available in the here and now?

AWESOME content

The part about the holes, looks like its checking open parts in a modeo without modeling a hole.

Too bad I'm broken financially, because I'd love to be able to support DF, I would ask so many questions lol. That said, thanks a lot to all the supporters who make DF viable.

I think it will be inspired by edge finding and treatment from AA technologies. Initially the rendered pixels are most edge or near edge areas a long with a pattern of colour and material samples from surfaces with normal near or within the camera frustum. This the method could use a lot less than 50% percent of the final amount of pixels to create the full image. The different holes would be filled out as described and I think the word “nature” pertains to material, normal and motion vector.

Wait, how old is this podcast? It's an old patent describing PS VR2 reconstruction technique, not a DLSS analog.

This seems like it could be used with frame gen/reprojection because that a reprojected image would have holes that need to be filled.

Frame generation and res upscaling techniques r clearly the future not just on consoles but on PC/Mac as well because everybody wants quality high res images running at least 60 fps. Nvidia will be definitely investing tons of money in that with the RT so their GPU cores logic can run them even more efficiently and faster.

Is this for PS5 Pro or Next gen or is it something as a steppingstone for next-gen.

Doesn't RDNA3 have ML/AI cores or are these ML/AI cores drastically different to Tensor/XMX cores and not capable of upscaling in the same way?

4k checkerboard still uses more pixels than native 1440p. Or 4k with FSR set to Quality. So it'll use more pixels, be more demanding and give you less FPS than a FSR2 Quality at 4k. This better be one hell of a good looking image to be worth the performance cost vs FSR2_Q

I’ve always wanted you guys to take a look at the current supersampling on PS5 because several unpatched PS4 games come out looking absolutely terrible on a 4K screen on the PS5, even if I force 1080p. They look worse than they ever did on my PS4 Pro using the same monitor/TV. Specific PS4 games like Mad Max, Arkham Knight, Watch Dogs, Alien: Isolation, and a few others I forget become a jaggy blurry mess. I wanted you guys to cover it because I played the same games on Series X on the same 4K TV and they didn’t have the same issue. Mad Max, Alien: Isolation, and Watch Dog all look much better on my Series X than on my PS5, again using the same exact TV. I know those games have FPS Boost on Series consoles but AFAIK that doesn’t effect resolution/supersampling. So I just thought it might be an interesting video to compare how the Series X handles supersampling vs the PS5, specifically for unpatched games from last gen. I never tested Arkham Knight on my Series X since it would just use the 900p resolution from the base Xbox One but I just tried Arkham Knight again on my PS5 hoping it would look better after the recent updates (Battinsom suit patch) and it still looks awful. The image is so jaggy it’s insane. The low anisotropic filtering probably doesn’t help either.

Hiw did you guys say "hole filling" so many times without laughing 😂

I hope it will work well. Can't be worse than FSR. But I'm staying optimistically skeptical..

the size of the holes would change depending on resolution, smaller resolutions produce bigger holes

Alex LITERALLY almost looks liek he`s about to cry :D Hilarious

If ps5 pro can.t deliver 4k 60 fps it will be usless from start.

Oliemack is given me The Baron from Dune Vibes 😳

Rather than being checkerboard, maybe it be more like Variable Rate Shading, but it uses the previous frame to determine where it can skip rendering entirely and leaves holes of 1x1 or 2x2 or maybe also 1x2 or 2x1 pixels. So maybe the game engine has to render a pre-pass of depth and velocity, passes it into their proprietary system at which point it will determine which pixels to completely skip during rendering because they will reconstruct it. Not sure if that is feasible though...

AMD is able to make dedicated AI accelerator now I suppose? As their laptop APUs and some new desktop ones have Ryzen AI, it would be just a matter of decision making to have them come to RDNA cards? (and some software engineering to the driver)

Never thought I'll listen to you guys talking about the quality and size of a hole...

PSSR gonna be a nice new feature that shines on the PS5 Pro at the end of the year 👍

The hole filling sounds a bit like what some stochastic raytracers (and splat renderers) do. When you have too few rays per frame to do enough rays per pixel, you spread them across (the most important) parts of the space and fill the holes to get a coherent shape. There are older papers out there about this from the last two decades. That said, I don't know if this really applies here, as I haven't read Sony's patent. It would make sense to investigate it for limited RTRT machines like the PS5 and anything beyond. It's a bit like a denoiser, but not quite. Stochastic rendering can produce irregular multi-pixel holds, which is very different from checkerboarding. Mixed resolution / foveated rendering also comes to mind, that might result in variable size holes, if combined with checkerboarding, as a speculative idea. Actually, if they wanted to reconstruct a stereoscopic image pair for VR from a single image with depth, then they might use this to fill occluded holes with this. It sure seems like the most plausible explanation to me right now.

So… the area where the player is focused looks the best and the rest is scarified but good enough.

Not sure why people are laughing and doubting Sony’s solution. Mark Cenry did great with PS4 Pro checker boarding techniques. I’m sure he’s come up with a great solution for PS5 Pro.

Interesting - judging from the PS4 Pro and X1X comparisons, checkerboarding looked softer compared to native 4k. DLSS looks close to 4k and in some cases better so there must be something missing. The rumor is that they are adding 120 AI cores to PS5 Pro. Not sure why they would take the checkerboarding route because they can just upscale with AI acceleration but perhaps it's better to use previous data at higher res with alternating pixels to generate a more complete native image since 720p is lacking a lot of data whereas 1080p interlaced with another 1080p interlaced image offset by a pixel is almost a complete 1080p image and then you can upscale to 6k, not just 4k with the same performance hit. One thing is clear - FSR isn't cutting it largely due to developers abusing it. AI acceleration is no longer an option for AMD or Intel - they all need to get into AI to survive financially unless they want valuations of 5% compared to Nvidia which will make them acquisition targets. Not only that but a 7900XTX cannot be using FSR next to a 4080 with DLSS and selling for $800-$1,000. It's unacceptable. I do wonder what Microsoft is up to. I suspect they will go for higher hardware specs as opposed to image recontstruction.

Can't wait to see what other crazy upscaling tricks Sony has up their sleeve.

This feels to me like a mixture of checkerboard rendering with something like Nvidias multi- res shading. The size of the holes being lower resolution or bigger holes on the outer edge of the screen with higher resolution smaller holes in the center of the screen

Looking at tech patents is headache inducing, I pity everyone involved.

if they are doing 8K upscaling with PSSR (PS5 Pro’s AI upscaler) then it would be from 1440p -> 8K at best.

Here's a cynical theory about the patent. What if Sony is just going through all of the work left behind after they shut down the London office and laid off a bunch of staff, hoping to find things that they can patent before those workers get hired by a competitor? For all we know, it's possible they don't understand this piece of technology any more than we do, but they're still calling dibs just in case.

Could they do something similar to what Apple does in terms of dedicated machine learning hardware?

Sony has amazing upscaling in TVs. They could do that in PS5 also.

14:45 "Checkerboarding forces developers to use resolution on a fixed grid" Which is the BEST feature of Checkerboarding because we have all seen over and over by now that "variable resolution" is a pandora's box that leads to every game running at 720p.

@Oliemacks lighting makes him look like a 3D render

⚠️ STRICTLY FOR PROMOTIONAL PURPOSES ONLY

DF Bespoke!

Could be checkerboard 2.0 for ps5 professional

Here's the reality. There are two methods to upscaling. The first is to make a higher resolution of the original image. They can fill in pixels and it will just look like a bigger image. This is similar to resizing a Vector image. The second is to increase the resolution and add detail where they wasn't. That requires advanced AI to intelligently create a texture. This will be much more resource intensive and would be more like what Adobe does in Photoshop.

My eyes are ready.

Digital Foundry always delivers the goods.

Doesn't Sony already have 4k Checkerboarding from the PS4 Pro?

Probably not what was meant, but a mental model to understand what the base idea of this could have come from, especially since it looks like it came from their now closed VR studio: Imagine what combining foveated rendering with checkerboard rendering would look like in the raw, unprocessed buffer. Let's say the highest res part is native res (4k p), surrounding that a checkerboard of native res pixels (4k "interlaced"), surrounding that a quarter res checkerboard (1080 "interlaced") or a 9th res checkerboard (720 "interlaced"). That would explain the variable size "holes", just as a result of combing different checkerboarded resolutions into one image. Now separate the res choice from the players gaze as that data isn't available outside PSVR2, you could still have different importance of resolution for different parts of the image, for example per material, per contrast, motion vectors etc. Then you have this jumbled mess of different res parts with different sized "holes" all over the place - which could and probably _will_ also change from frame to frame as things move, new materials enter the image while others exit. You can't really do that without a final scaler step that takes in all that mess and constructs a coherent image from it. You also can't expect one of the now common scaling solutions to *just* *work* with that kind of input. Can imagine this breaking down horribly in some cases and cost a lot more than what you save in the lower res parts. But if it works as intended, it could give enough breathing room for a HW to punch well above its weight. TLDR Variable rate shading round 2 Sony Special Sauce edition.

we are gonna see a lot of flickering/upscaled next gen games until the technology is properly implemented

I wish DF InDirect was about Indie games. The name punch is just too good.

There is actually one 8k game on the PS5 its the Tourist. So hey at least we have one to boast about. Its great for those 4 people with 8k TVs.

this is quite odd to me that they would have to use two methods because of hardware limitations, considering we have rumours of even a switch 2 with DLSS (although i still dont buy that, it would be kinda uncharacteristic for nintendo to be ahead of the console market in implementing something like this, i mean were talking about the company that ships most of their games with 0 AA) surely AMD as the provider for the APUs would be happy to collaborate in the chip design to accommodate a solution that can fully upscale/fill all the holes, so really this only makes sense as a solution for hardware that is no longer in development, aka current gen

Put a link to it somewhere how I sign up? # yamam day

This could be a broadcasting technology patent. Sony files a lot of those

Does bimonthly mean biweekly?

Book mark this video so when the ps5 pro get shown we can compare

I have a question. What tea does Richard drink and which milk, how many sugars.

Hopefully the patent is more than Fig 2. Fig 2 sounds overly broad, like in the era of "done on a computer" patent trolling

If it is using AI inferencing, can it even run on a PS5?

So will gta6 be 4k60 or not

PC elitists tend to criticize Sony's checkerboard solution but pop their behinds for the PC driver software workarounds... Biggest problem is that despite the Teraflop performance of consoles and PCs there is actually very LITTLE in the way of talented programmers who can actually utilize the full potential of eight threads to sixteen threads of a CPU and much less the GPUs with 20 to 35 plus Compute Units... Instead people sip the kool-aid and assume that heavily marketed 3d engines are somehow the next big thing when it never even performs right plus most of these critics don't even sit down and play on a PS4 Pro and 4K HDR tv at 50 inches or larger to be able to whine like they do. That's for the critics... also before Checkerboarding, Sony's software devs made the MLAA solution by using the CellBE CPU... stuff like that actually uses the processor hardware and doesn't just glide over it. Whatever this new patient is, it's important to Sony and given that technology patients need to be protected, there is probably not much to go on until we finally see it used in action. Also "8K"... seriously? I wonder if people on here realize the minimum tv size needed to actually "appreciate" 4K HDR... 55 inches seems big, until you play on a 65 inch... so that means that to enjoy 8K you will need a minimum of 75 to 85 inch screen plus some advanced Mini LED Full Array Local Dimming LED technology that won't develop screen burn in like OLED... That kills tons of people on here who believe their 27 inch PC monitors are somehow enough to enjoy and critique games on.

Hope Sony wont implement a simple evolution of Farnebäck optical flow

Sony is master of upscaling. On TVs it is very good... even on the PS4 Pro it is very good. I think ingame will look great as well with Sony´s experience.

Hearing Alex say "Holes" this many times made me uncomfortable.

Sounds to me like it's an 8k upscaler for the next generation of Sony TV. Why make your console do all the work when the $3500 TV in the living room can handle upscaling workload as well.

So are you guys totally done with the Digital Foundry "Directs" that are just on KZfaq?

This does not sound like an resolution upscaling solution but more an frame generation solution. Usually with frame generation, especially in VR (think Oculus Application SpaceWarp) you get big holes in parallax movement that need to be filled in.

upscaling is ruining games. Devs used to create entire game engines to do specifically what they needed it to do or even come up with cool ways to cheat effects but now they just create a game over budget and use upscaling and call it a day and say thats good enough.

What, is Alex gonna cry again? 😢

DLSS is the goat and unfortunately at this point it seems that upscaling solutions not specific to AI/machine learning proprietary hardware just can't compete. Sorry but FSR is just garbage.

We have upscaling at home

I feel like Microsoft is going Nvidia on their next box with dlss. I think Sony is trying to figure out their own tech instead. Microsoft said something about things need to go in a different direction next gen and they have all been on amd forever. Plus they are talking huge jumps in quality on the next console and dlss could really help with that.

It’s probably ambiguous by intention.

16 minutes about holes and hole filling, nice

I have couple ways to feel the hole as well... :D

I’ve got a good technique for filling holes