I'm curious to see how the "Quad Bayer" mosaic works out. Other manufacturers have tried novel filter patterns before, but nothing so far has really been able to compete.
Essentially, almost all digital cameras today use a planar CMOS sensor with alternating RGB-sensitive pixels, arrayed like so:
RGRGRGRG
GBGBGBGB
RGRGRGRG
GBGBGBGB
This pattern is not perfect, but is highly effective. Luma (color-independent) resolution is essentially equivalent to the actual number of pixels, while chroma (color-dependent) resolution is only slightly less - we essentially get one "real" point of color information at each intersection of four color pixels, because at each of those intersections we have one red, one blue, and two green pixels.
In other words, "the luma information we gather for a given color on a pixel of that color is immediately relevant to the effective pixel composed by it and its adjacent neighbors at each of its four corners". In this 8x8 Bayer pattern pixel grid with 64 real pixels, we get 49 effective chroma pixels; one for each intersection of 4 physical pixels.
In comparison, here's the pattern for "Quad Bayer":
RRGGRRGG
RRGGRRGG
GGBBGGBB
GGBBGGBB
I'm concerned that chroma resolution and overall color accuracy will be much lower with this pattern. Essentially, with the original Bayer demosaicing, you only need to sample from the four color pixels adjacent to each corner in order to get a bit of the three channels, and the pattern gives equal weight to both red and blue, while providing extra accuracy in the green channel that human vision is most sensitive to.
In comparison, as far as I can tell, a single "effective pixel" (one with information on all three channels) using the Quad Bayer pattern has to be made up of data from nine individual pixels. Additionally, when an effective pixel is centered on an actual pixel with either red or blue filters, that color is relatively dominant in the pixels considered - it'll be equally weighted with the green channel, and the opposing color will only make up 1/9 of the total signal composing that effective pixel. Effective pixels centered on green pixels will give equal weight to red and blue, with slightly over 5/9 of the weight given to the green channel.
Granted, the sensor should still be able to produce a full 48MP of luma resolution, but chroma detail will be much more "smeared" because of the wide area that has to be considered to get a full color pixel, and the more substantial overlap of that full color pixel with other full color pixels. Color accuracy will likely also be lower, because in effective pixels centered on red and blue pixels, only a single pixel of the opposing color will be used, which means that any noise in that channel will have an outsized impact on the overall color.
What this boils down to is that, when used as a 48MP sensor, this sensor will have entirely different imaging characteristics than a traditional Bayer imager, and that those characteristics will be highly dependent on how the output of this sensor is processed - which will be interesting in a world full of software highly optimized to demosaic Bayer-pattern images.
What's slightly more interesting is the high-sensitivity 12MP mode. Essentially, it's an attempt to reduce the impact of random noise in the image by adding together four pixels of each channel to produce a "superpixel" less impacted by noise overall. These superpixels can then be processed in a standard Bayer pattern as a 12MP effective image.
Thinking about it overall, though, I become more and more confused. In both of these modes, this pattern doesn't give us anything, really, that we can't already do using a Bayer filter.
Let B represent a sensor using a standard Bayer filter pattern, and let Q represent a sensor using this "Quad Bayer" pattern, where each of these patterns have a red pixel in the top-left corner.
Let any given effective pixel be represented by a 3-tuple of the form (R, G, B), where R, G, and B are the number of physical pixels sensitive to each of the red, green, and blue channels which compose that effective pixel.
Let f(p, w, h, s, i) be a function returning a two-dimensional matrix of all the effective pixels produced by a matrix of physical RGB pixels, laid out in pattern p, with actual pixel width and height w and h, where an effective pixel measures s actual pixels horizontally and vertically, and where an offset of i actual pixels in either vertical or horizontal directions produces the "next" pixel in that direction.
Thus, our standard Bayer pattern produces the following:
Note that there are fewer effective pixels for the same total number of pixels - that's okay, though, because the number of effective pixels approaches the number of total pixels as the sensor scales in the X and Y dimensions - this very small hypothetical sensor doesn't benefit from that scale yet.
You can also see that each effective pixel is composed of a larger number of physical pixels - there's a tradeoff there, in that this means that overall, noise should have a smaller impact on the value of a given pixel, but there's a loss of resolution because those pixels are spread over a wider area.
This raises the question, "what if we do a 9-pixel effective pixel on a standard Bayer pattern?" Well, we get this:
Interestingly, while the exact arrangements of the different color channels within the effective pixels are different, the total number of pixels of each channel remains completely identical, meaning that any given effective pixel should have identical noise characteristics to the Quad Bayer pattern. In fact, it's arguable that the Bayer pattern is better, because the color physical pixels are more evenly distributed around the effective pixel.
What if we do the high-sensitivity superpixel sampling? For the Quad Bayer pattern, it looks like this:
Again, sampling in a similar pattern gives the same overall result. So, all else being equal, I'm not sure it makes sense.
Of course, there is the possibility that all else is not equal. Having multiple adjacent pixels of the same color could enable consolidating the signals of those pixels together earlier on in the image processing pipeline into an actual lower-resolution standard-Bayer signal. That could actually have real benefits if that early-stage signal combination results in a greater signal amplitude that drowns out noise.
Basically, this has all been kind of stream-of-consciousness and much longer than I originally planned, but here's the Cliff Notes from what I can tell.
In comparison to a Bayer sensor of the same pixel resolution...
Pros:
- (If implemented to take advantage, possibly) Ability to act as unified large pixels, increasing SNR at lower resolution settings
Cons:
- Less-fine maximum chroma resolution when all pixels are active
Essentially, almost all digital cameras today use a planar CMOS sensor with alternating RGB-sensitive pixels, arrayed like so:
This pattern is not perfect, but is highly effective. Luma (color-independent) resolution is essentially equivalent to the actual number of pixels, while chroma (color-dependent) resolution is only slightly less - we essentially get one "real" point of color information at each intersection of four color pixels, because at each of those intersections we have one red, one blue, and two green pixels.In other words, "the luma information we gather for a given color on a pixel of that color is immediately relevant to the effective pixel composed by it and its adjacent neighbors at each of its four corners". In this 8x8 Bayer pattern pixel grid with 64 real pixels, we get 49 effective chroma pixels; one for each intersection of 4 physical pixels.
In comparison, here's the pattern for "Quad Bayer":
I'm concerned that chroma resolution and overall color accuracy will be much lower with this pattern. Essentially, with the original Bayer demosaicing, you only need to sample from the four color pixels adjacent to each corner in order to get a bit of the three channels, and the pattern gives equal weight to both red and blue, while providing extra accuracy in the green channel that human vision is most sensitive to.In comparison, as far as I can tell, a single "effective pixel" (one with information on all three channels) using the Quad Bayer pattern has to be made up of data from nine individual pixels. Additionally, when an effective pixel is centered on an actual pixel with either red or blue filters, that color is relatively dominant in the pixels considered - it'll be equally weighted with the green channel, and the opposing color will only make up 1/9 of the total signal composing that effective pixel. Effective pixels centered on green pixels will give equal weight to red and blue, with slightly over 5/9 of the weight given to the green channel.
Granted, the sensor should still be able to produce a full 48MP of luma resolution, but chroma detail will be much more "smeared" because of the wide area that has to be considered to get a full color pixel, and the more substantial overlap of that full color pixel with other full color pixels. Color accuracy will likely also be lower, because in effective pixels centered on red and blue pixels, only a single pixel of the opposing color will be used, which means that any noise in that channel will have an outsized impact on the overall color.
What this boils down to is that, when used as a 48MP sensor, this sensor will have entirely different imaging characteristics than a traditional Bayer imager, and that those characteristics will be highly dependent on how the output of this sensor is processed - which will be interesting in a world full of software highly optimized to demosaic Bayer-pattern images.
What's slightly more interesting is the high-sensitivity 12MP mode. Essentially, it's an attempt to reduce the impact of random noise in the image by adding together four pixels of each channel to produce a "superpixel" less impacted by noise overall. These superpixels can then be processed in a standard Bayer pattern as a 12MP effective image.
Thinking about it overall, though, I become more and more confused. In both of these modes, this pattern doesn't give us anything, really, that we can't already do using a Bayer filter.
Let B represent a sensor using a standard Bayer filter pattern, and let Q represent a sensor using this "Quad Bayer" pattern, where each of these patterns have a red pixel in the top-left corner.
Let any given effective pixel be represented by a 3-tuple of the form (R, G, B), where R, G, and B are the number of physical pixels sensitive to each of the red, green, and blue channels which compose that effective pixel.
Let f(p, w, h, s, i) be a function returning a two-dimensional matrix of all the effective pixels produced by a matrix of physical RGB pixels, laid out in pattern p, with actual pixel width and height w and h, where an effective pixel measures s actual pixels horizontally and vertically, and where an offset of i actual pixels in either vertical or horizontal directions produces the "next" pixel in that direction.
Thus, our standard Bayer pattern produces the following:
The 9-pixel-effective-pixel Quad Bayer pattern produces this: Note that there are fewer effective pixels for the same total number of pixels - that's okay, though, because the number of effective pixels approaches the number of total pixels as the sensor scales in the X and Y dimensions - this very small hypothetical sensor doesn't benefit from that scale yet.You can also see that each effective pixel is composed of a larger number of physical pixels - there's a tradeoff there, in that this means that overall, noise should have a smaller impact on the value of a given pixel, but there's a loss of resolution because those pixels are spread over a wider area.
This raises the question, "what if we do a 9-pixel effective pixel on a standard Bayer pattern?" Well, we get this:
Interestingly, while the exact arrangements of the different color channels within the effective pixels are different, the total number of pixels of each channel remains completely identical, meaning that any given effective pixel should have identical noise characteristics to the Quad Bayer pattern. In fact, it's arguable that the Bayer pattern is better, because the color physical pixels are more evenly distributed around the effective pixel.What if we do the high-sensitivity superpixel sampling? For the Quad Bayer pattern, it looks like this:
And for the standard Bayer, like this: Again, sampling in a similar pattern gives the same overall result. So, all else being equal, I'm not sure it makes sense.Of course, there is the possibility that all else is not equal. Having multiple adjacent pixels of the same color could enable consolidating the signals of those pixels together earlier on in the image processing pipeline into an actual lower-resolution standard-Bayer signal. That could actually have real benefits if that early-stage signal combination results in a greater signal amplitude that drowns out noise.
Basically, this has all been kind of stream-of-consciousness and much longer than I originally planned, but here's the Cliff Notes from what I can tell.
In comparison to a Bayer sensor of the same pixel resolution...
Pros:
- (If implemented to take advantage, possibly) Ability to act as unified large pixels, increasing SNR at lower resolution settings
Cons:
- Less-fine maximum chroma resolution when all pixels are active