One thing that's great about this aproach on learning this....is that, after you understand it on this "higher" level it's much more easy to get into the algorithm implementation part and the mahematical part. You get into it with a purpose, with a visual idea of your goal....and spend much more time in the abstract world without losing focus (pun intended).

Had an assignment last year to implement a Bayer filter on an FPGA which I failed because my professor couldn't explain what the filter was doing in terms I could understand - he insisted on trying to explain everything in terms of matrices: You've just explained it with perfect clarity in about 5 minutes. [Tears out hair] University sucks, Computerphile & KZfaq FTW.

Almost lost my sleep over how a sensors captures images. I couldn't find any information on how the sensor compensates for the extra green or what happens in the in betweens. This video helped me better understand all of this and even more. Thank you so much!

It might be a good idea to explain the antialiasing filter that tries to prevent demosaicing artifacts by blurring the lines between the color pixels.

Excellent video!! Please bring Mike back for more, He is a very articulate presenter and I would love to see more of him.

I've been looking for detailed information like this everywhere. Bryce Bayer bless this video. You guys are the best.

this is great. as a photographer and an technician I am loving this series

I want to know more about this. Thank you

Very nice graphics in this video. The content of the discussion is very interesting too!

I'm studying now electrical and computer engineering and all that technics are part of the courses that I have taken. Very nice presentation Congrats!

So many videos just to picture (eheh) the sensores and the filter... I got to Computerphile and understood it in the first minute... I've seen you for my Informatic Security exam, now for my optics exam.... You got it all. Thx a lot

this video is amazing I'm so hyped this could unlock so many new types of projects for me

I knew about the bayer filter but it is very interesting to hear about it more in depth. It's almost completely useless for camera operators like me to know, but my god it's so well explained and more knowledge never hurts.

Amazing description. Thank you. Used this to explain my computer vision class topic

More videos about graphics, 3D and CGI please!!!! :D :D Loving it so far!!!

I love the whiteboard marker you used, it's so smooth and soothing...I wish all my teachers used it :)

I'm just getting into photography and astrophotography so this is of extra interest to me now (on top of the generally interesting topics that you tend to cover). Thanks!

You know, when I was doing research into how they made color movies before color film was developed I found out that they would split the light beam into it's component colors and they would have 2 or 3 rolls of film all going through the camera at once, taking black and whit film. They would then tint the film with it's respective color and then put them together. Some times gluing them, some times using a processes to make a kind of stamp. It's pretty interesting. I'm sure if you wanted to get a more accurate picture you could do something similar and have a beam slitter with 3 separate sensors.

Most intriguing. More, please.

You should include more simplified diagrams & visuals for better learning. Its a huge topic to cover in a 6 minute video but I think it could've been clearer. Thanks for such informative videos & keep up the good work. (y)

Great video, also love the fan-fold paper :)

The Foveon X3 image sensors are different. I'd love to see a video on their tech.

U saved me 3 hours before my exam! Thanks

And the Lowered Color Resolution is why Space Probes use a Black and White (Luminance) Camera and then just Physically put a Big Filter infront of the whole camera one Color at a time, so you get the full resolution for each color.

love all of you videos! thanks

Nice overview, thanks for that.

Great explanation, thank you!

That was really interesting!

this answered so many things for me. great explanations.

Masterfully said.

is that the reason why sometimes when taking a picture of a fine mesh or looking at it on a screen can cause weird colour artifacts ?

I'd like to hear more about this subject.

you guys should do a video on light field cameras

I love this guy man! So much genius

great video. Thanks!

will you guys please explain why some video cameras record moving bars of light and dark when filming computer screens?

Watching this made me try to think of how to do that process (well, in a way) on each captured pixel. Granted it would be more expensive, but could there be a mini prism that "stretches" the light like a prism and then use that data? I'm guessing no because of the size of the photon packet, but it sounded good in my head.

Very interesting insight.

How about RGBC sensors? (C = Clear, not sure if it's the same as White as in RGBW) They're rare, but I know some things use them, like for example some Motorola phones. Great video btw. Very interesting topic.

Just to nitpick #2:16, we don't distinguish luminance with much more intensity in the green channel, as lumen refers to how bright the human eye will perceive light; 100 lumen green, blue or red would look identically intense as is the purpose.

Fantastic! I won't look at my camera the same way as before...

Awesome!

I think you should have shown examples of the demosaicing process. This is a purely technical theoretical explanation. I knew and understood that explanation for years before I first saw an actual visual demonstration of it. That visual completely changed my understanding of it, because just the technical story doesn't really explain it all that well. Wikipedia has a very nice demonstrative visual.

Really helpful since i'm currently implementing demosaicking in cuda c ;)

In film making the debate between film v digital is still very much alive and well. My Super 8 camera can give professional cinematic images. - The way the film is made and processed has improve dramatically over years, however, to gain that high widescreen quality is expensive. The company Pro-8 is famous worldwide for it's Super 8 processing (and all other film formats). Film is still largely used by Hollywood. - Most directors are anti digital. TV networks often still use film. - PBS has a strong relationship with Pro-8.

Sounds like we need a camera that detects the exact color for every pixel instead of creating the color by mixing nearby rgb values. I wonder if that's possible. A sensor that detects the exact color frequency hitting its sensor. Not rellying on using rgb filters to determine the rgb values.

How filters are placed over each pixel in sensor

Some higher end manufacturers used to be a bit more honest about the actual interpolation breakdown in their technical spec documents like hasselblad and phase one but for some reason I can't find that data anymore - I've spent a lot of time googling to put some more data into my comment but have to go from memory so please excuse that. Most 35mm format digital cameras are in fact as many people ask/suggest offering about 1/3 of the reported resolution due to interpolation and then also make it worse by using AA filters to get rid of interpolation artifacts. So a 30 MP camera is closer to 10MP as far as the physical sensors capability is concerned regarding the colour with the least amount of pixels dedicated to it and you still have the AA edge issues which are a lot more noticeable than the marketing would make you believe - it's not just fine patterns that are affected, those are just the extreme examples - all edge quality is visibly affected if you don't downsample. If you downsample a 36MP nikon d800 shot 2-3x then you can correct the edge detail and counter the quality lost due to interpolation, especially with the excellent downsampling that you can do nowadays (choose correct technique depending on image content etc.) and arrive at a resolution of around 16MP or what was the standard for MF digital backs many years ago. The problem is that even the most expensive MF digital backs do not report interpolation figures as honestly as they used to and when judging progress of digital sensor technology we often look at the MF market, at the expensive solutions! I have no idea how many physical "pixels" are on the sensor of say an PhaseOne IQ260 which talks of 60MP resolution, by old high quality standards for interpolated MF backs 60MP reported would suggest there are more than 60MP "pixels" in total on the sensor, Maybe 30MP for the colour with the least amount of pixels but definitely not 1/3 but lack of data might suggest that it is in fact the same breakdown as in small format full frame stuff from Canon and Nikon. It is the standard that small frame digital bodies are interpolated and the resolution is over reported and almost all have AA filters but that is not what we expect of medium format backs. Most of these campaigns against film photography are based on figures and opinions stemming from use of the medium format stuff not small frame. So not to start a debate between medium format and small frame - what I am trying to say is that we are all getting shafted nowadays with this ridiculous marketing, especially against film and the data to tell how much shafted we actually are is no longer made available. These interpolation techniques which are not an issue with say 3k dollars for a nikon body are an issue when we are talking about a 50k dollar medium format back, which is at the end of the day just a big sensor you add to a system you already own and paid for. For those of you who never used MF, my feelings translate like this - it's as if you would have to buy a 3k dollar sensor that you would put in your analog Nikon F and it would offer maybe 2x time the resolution of an iphone camera interpolated up 3 times - you would be very unhappy, especially given that film is being destroyed by this kind of marketing. You can take amazing digital photos with just a pinhole attached to an old digital camera - the point is that the marketing used to be a bit more honest about the resolution. The old expensive MF backs offered maybe 4-12 MP but that was without interpolation or AA and what is left nowadays is still thankfully lack of AA but the hidden interpolation details and lack of reporting in regards to how the sensor is physically divided between the colours is all about hiding the true progress of technology. Unlike many seemingly commenting on the web about too many MP, I do actually feel there is not enough MP - I want more. I want about 120MP on a full frame MF sensor if it has to be interpolated (that is at least 56mmx56m to cover the film area in my system) and I don't want less than 80MP per colour. This kind of digital back is maybe 15 years in the future. An interpolated file without the AA filter where you get say 16MP at least for the colour chosen to be least important resolution wise is of-course going to surpass a non interpolated file where you had 16MP for each colour because the interpolation techniques and filtering is very good nowadays - it's just that we are not in the high MP numbers as we think we are because through the years the reporting and measuring changes. This might be irrelevant to most but to those of us who are attacked all the time about our film being outdated have to deal with all kinds of crazy arguments. One person told me my slow speed MF film is at most 50MP, well I disagree but due to interpolation even if my film was 50MP he won't be able to find a MF back that actually approaches that save for maybe the newest 80MP offering from Phase One. The same problem is with scanners for film and general reproduction setups - they also use interpolation and are even more ridiculous in their marketing. Arguing that interpolating something to large sizes is as good as natively reproducing it at those sizes requires tinfoil hats - it's like saying on the other side of that argument, that a slide enlarged to cover a whole building magically increased in resolution - we might perceive more resolution and fantastic detail, the colours might be vibrant with that super expensive projector and so on and so forth but it's still not a ground to argue against a competing technology.

Huh. Always thought cameras captured the whole of the image in front of them, but it seems it's a game of knowing how human eyes work and how that can be applied to make image capturing easier. Does that mean that other species might see pictures far different from how we do?

2:13 I never got this explanation, here or elsewhere. What does our eyes capacity to capture green light have to do with how the sensor has to do it. The final compounded output of a 4x4 pixel array will be a algorithmic calculation (demosaicing ) anyway. I would assume, I can adjust this calculation/ demosaicing in a way, so that the final picture looks natural to our eyes then anyway. I would understand an explanation like, green being the most prevalent color in the environment, thus, it would make sense to focus on capturing that. Eyes capture -> Brain translates. Sensor captures -> Chip translates -> Monitor Output. 2 separte systems of capturing and translation, therefore no need to mimik one part of the "other chain". What am I missing?

Interesting. But what happens if for example a yellow color photon hits one of these filters? Will it get through or get discarded? Is there a range of frequency these R,G & B filters let through and if so does these ranges of frequencies cover the whole spectrum of visible light? So many questions, so little time. :)

Cool. This is a lot like digital audio interpolation.

Is there any software that analyzes an image and uses different demosaicing methods for different parts of the image? Also, is noise one pixel misreading it's signal? Can noise be reduced in the demosaicing phase? By excluding information by individual pixels that don't fit their surroundings?

Wow, that density plots instantly triggered a migraine with me. My brain can't handle it better than a camera...

whats up with fuji x-trans sensors?

Why does my Canon PowerShot SX400 IS give me green and purple fringing between bright and dark areas?

kudos to the animator!!

What's the difference between CCD and CMOS?

The problem of figuring out what goes in the gap of the color sensors seems like something that a neural network would work well to solve. Train a neural network to recognize patterns like edges, curves, and grain, by using incomplete color information. And maybe try to shape it's network structure so that it corroborates it's guess for one color of sensor with its given data from the other available sensor colors, rather than just a single type. I bet you could get an astounding quality image using this processing technique. (Come to think of it, I think that "RAW" image data may actually be the data from the individual color sensors, before the camera tries to squash them into normal pixels. So the neural network could be used well after taking the picture, if your camera saves the raw data.)

I came here to enjoy the experience

they answer what exactly i was looking for.

Verify your electrical circuits on the go! Look up: 'Circuit Solver' by Phasor Systems on Google Play.

Hmm.. i'd like to hear also about how IR cameras works.

Wouldn't the colour filtering cause 2/3 or so of the photons to go uncounted? I know they could keep detecting longer but can't that be more efficient? Couldn't they also use tiny optics like prisms to split the spectrum and get a larger amount of the source red, green, and blue light to the intended detectors? I saw that some cameras use a beam splitter to split the total light but even they filter out 2/3 or so of the photons before detection instead of sending most red, green, and blue to the appropriate detectors. Chromatic dispersion seems to be more efficient if they could just make the millions of tiny prisms over the detector.

Well, this says nothing about the sensor size in relation to the lens, which is also important to actually see anything, or even to know the optimum amount of sensors.

Telescope ccd?

Humm, I always though the actual sensors raw data are the frequency and intensity of light that hit the sensor. Seems I was wrong.

thanx

Wow

8640000 bits per color in a 1 mega pixel image, tell me if i am wrong but 8 bits per pixel one mega pixel is 1200 x 900, so 8x(1200x900) should be 8640000

I wonder if anyone has made a mosaic pattern that's not a grid of squares. If you did triangles you could get equal proportions of red green and blue.

Why are raw images so huge in file size? It seems like there is less data than in a demosaiced image because each and every pixel has three values, not just the one in the raw.

***** Nice to know how all of this works, thanks for the post. I have a question about these filters: how possible would it be to have a sensor with a specific pattern of R, G, and B filters that shift 3 times during the taking of a picture and each time capturing the specific amount of R, G, B for each separate pixel in the sensor. I understand this would have to be a very precise movement of the filters relative to the sensor, and that this would either work for still images or for very high shutter speeds; but would it work at all? I thought about this because I was thinking about HDR pictures and how, when using this setting, cameras take multiple pictures in different brightness settings and make a composite one based on what the software thinks is the best HDR picture. So thinking along this same line, with te shifting filters you record separate R,G, and B value based on three different pictures and make a composite RGB value.

I always thought each of these r/g/b sensors where grouped to 1 pixel which is independent from the other pixels and the brightness get's averaged out from these grouped sensors...

Rough idea: Build a camera with a semi-transparent mirror which transmits 67% of the light into a plain B&W sensor. The other 33% gets reflected into another sensor with a checkboard-like filter of just blue and red pixels. A chip could guess the green channel and you got the best of two worlds: great looking edges without texture aberrations and just two cheap sensors (the blue/red filter and the B&W sensor are more simple than the bayer R/G/B filter) not three sensors. Because it would be a YCbCr camera, the YCbCr 4:2:2 mp4 compression process would be easier as well.

anyone know who the interviewee is? I'm wondering if he has a KZfaq channel or not

Can you make a video about error detection?

The Foveon sensor bypasses all this.

Why can't we put RGB filters in front of every CMOS sensor so we don't have to do any demosaicing?

Aren't there 3 subpixels for each pixel? This guy says there are actually 3 megapixels in a 10 megapixels sensor, but it seems more plausible to me that there are actually 30 megasubpixels and therefore 10 megapixels, can someone clarify?

Subtitles, please!!

But can the Bayer Filter capture why kids love Cinnamon Toast Crunch?

Now I understand what the `Demosaic[]` function in Mathematica does 😁👍

Lightfields!

Please do an episode on Y2K!!!!!!!!!!!

2:11 I thought it was the other way round? He says our eyes are MORE sensitive to green than blue & red, isn't it less?

1. Why not CMYK? Would fit the 4 pixels perfectly, and CMYK is used for colour kind work no? 2. Insensitive to colour? Human eye will still see if you used rec.2020, if not, why is 2020 needed to begin with?

The way he writes the letter B is… interesting.

interesting, but it doesn't explain how i can get a gopro for free

So our eyes don't see color that well... Does this mean that pictures taken with our cameras would look quite strange to animals that see color better?

are they onle RGB? are there cameras with CMYK?

So are we interpolating pixels just for the sake of cramming more of them in? Couldn't we just used less pixels and stores only the ones whose values we truly know?

No, a camera with x megapixels isn't x/3 for red, x/3 for blue, x/3 for green... Every pixel is composed of an amount of g/r/b... So he's kind of contradicting himself since when the interviewer asks him he says yes, but then explains it differently.

Fuck this! I am going Sigma Merrill ! :D

the sun is green

Dd

welp i found my next project

2020 anyone?

A bit short...

What are you talking about I don't understand anything