# Color Correction of Noisy Data

(Difference between revisions)
 Revision as of 13:32, 9 August 2015 (edit) (Created page with "Manu had been analyzing the consequences of noisy data for color balancing algorithms. Specifically we are looking at how the noise is amplified (or not) in the color corrected s...")← Older edit Revision as of 13:58, 9 August 2015 (edit) (undo)Newer edit → Line 1: Line 1: - Manu had been analyzing the consequences of noisy data for color balancing algorithms. Specifically we are looking at how the noise is amplified (or not) in the color corrected space. Over the years we have recognized that a color transform can be very good for getting the means right, but the noise shows up badly. This should be clear from the properties of the color transform; these were analyzed in one project by Manu and in another by UB. + Manu had been analyzing the consequences of noisy data for color balancing algorithms. Specifically we are looking at how the noise is amplified (or not) in the color corrected space. Over the years we have recognized that a color transform can be very good for getting the means right, but the noise shows up badly. This should be clear from the properties of the color transform; these were analyzed in one project by Manu and in another by UB. - Manu has discovered that properly accounting for the measurement noise covariance has a very positive effect on the estimations. We want to return to this problem and write a longer paper on this. + Manu has discovered that properly accounting for the measurement noise covariance has a very positive effect on the estimations. We want to return to this problem and write a longer paper on this. - '''This is interesting again because of the interest in using color channels with non-traditional sensors (not just RGB). This can be very problematic in terms of the amplification of measurement noise.''' + '''This is interesting again because of the interest in using color channels with non-traditional sensors (not just RGB). This can be very problematic in terms of the amplification of measurement noise.''' - Much of this was summarized in the June 2009 report to Samsung. We are editing for new ideas and a paper here. + Much of this was summarized in the June 2009 report to Samsung. We are editing for new ideas and a paper here. - = Background = + = Background = - A key issue that arises while considering color filters with wideband sensitivity functions for increasing imaging device sensitivity is the design of an appropriate color correction method.  This is especially important if color filters admit NIR energy.  If the color transformation is not carefully designed, any improvement in sensitivity gained by admitting a larger number of photons can be negated by the amplification of noise in the color transformation stage.  In this report we discuss the accuracy and robustness of conventional color transforms (that are implemented as matrix multiplies) applied to various types of color filter sets with NIR sensitivity. We show the effect of increasing levels of measurement noise (decreasing light levels) on the noise levels in the color-transformed output. Also, we show the tradeoffs involved in color accuracy and noise amplification for color filter sets with different amounts of sensitivity in the NIR range. We found that: + - #A color filter set with all four color sensors sensitive in the visible and NIR range can give acceptable results when the measured signal has high SNR.  But as input noise level increases, the noise-levels in the color-transformed signals become unacceptably high. + - #A color filter set with three color sensors sensitive in the visible range (RGB) and a fourth wideband sensor sensitive in the NIR range performs well in terms of accuracy and robustness to measurement noise. + - #A color transform derived as the minimum mean squared estimate of CIEXYZ values from color measurements can be used to improve the performance of the color filter set in (1) by using the statistics of surface spectra and noise. + + A key issue that arises while considering color filters with wideband sensitivity functions for increasing imaging device sensitivity is the design of an appropriate color correction method. This is especially important if color filters admit NIR energy. If the color transformation is not carefully designed, any improvement in sensitivity gained by admitting a larger number of photons can be negated by the amplification of noise in the color transformation stage. In this report we discuss the accuracy and robustness of conventional color transforms (that are implemented as matrix multiplies) applied to various types of color filter sets with NIR sensitivity. We show the effect of increasing levels of measurement noise (decreasing light levels) on the noise levels in the color-transformed output. Also, we show the tradeoffs involved in color accuracy and noise amplification for color filter sets with different amounts of sensitivity in the NIR range. We found that: - == Introduction == + #A color filter set with all four color sensors sensitive in the visible and NIR range can give acceptable results when the measured signal has high SNR. But as input noise level increases, the noise-levels in the color-transformed signals become unacceptably high. + #A color filter set with three color sensors sensitive in the visible range (RGB) and a fourth wideband sensor sensitive in the NIR range performs well in terms of accuracy and robustness to measurement noise. + #A color transform derived as the minimum mean squared estimate of CIEXYZ values from color measurements can be used to improve the performance of the color filter set in (1) by using the statistics of surface spectra and noise. +
+ + == Introduction  == Image sensors can not naturally discriminate between signals with energy in different wavelength regions. The sensor simply produces a signal proportional to all incident energy in its sensitivity range. Typical color cameras use color filters with different wavelength-dependent transmittances to distinguish between three or more separate wavelength bands (color channels). Camera measurements of a stimulus represent energy captured by the color channels. To display images, camera measurements must be converted to a form that can be related to intensities evoked by the same stimulus in the human visual system. Color conversion from camera space to CIEXYZ space helps with this transformation. Image sensors can not naturally discriminate between signals with energy in different wavelength regions. The sensor simply produces a signal proportional to all incident energy in its sensitivity range. Typical color cameras use color filters with different wavelength-dependent transmittances to distinguish between three or more separate wavelength bands (color channels). Camera measurements of a stimulus represent energy captured by the color channels. To display images, camera measurements must be converted to a form that can be related to intensities evoked by the same stimulus in the human visual system. Color conversion from camera space to CIEXYZ space helps with this transformation. - == Color transformation == + == Color transformation == - To confine the discussion to the properties of the color transformation, in this report we will assume that the illuminant under which signals are acquired is known. The experiments and methods described here can be generalized by considering an accurate illuminant estimation scheme in the framework. Here, we assume that all spectra are acquired under an illuminant that has equal energy along the wavelength range and the task is to convert camera measurements to CIEXYZ space. + - In matrix-vector notation, we write + To confine the discussion to the properties of the color transformation, in this report we will assume that the illuminant under which signals are acquired is known. The experiments and methods described here can be generalized by considering an accurate illuminant estimation scheme in the framework. Here, we assume that all spectra are acquired under an illuminant that has equal energy along the wavelength range and the task is to convert camera measurements to CIEXYZ space. In matrix-vector notation, we write $y = S^T c\,,$ and $y = S^T c\,,$ and Line 27: Line 29: $x = A^T c\,,$ $x = A^T c\,,$ - where the columns of the matrices $S\,$ and $A\,$ are the spectral sensitivities of the camera color filters and the CIEXYZ color matching functions respectively. One way to find the mapping from camera-colorspace to CIEXYZ space is to find a linear transformation (represented by the matrix $M$ ) so that the estimate $\hat{x}$ of the CIEXYZ values of a spectrum can be obtained from its camera measurements as + where the columns of the matrices $S\,$ and $A\,$ are the spectral sensitivities of the camera color filters and the CIEXYZ color matching functions respectively. One way to find the mapping from camera-colorspace to CIEXYZ space is to find a linear transformation (represented by the matrix $M$ ) so that the estimate $\hat{x}$ of the CIEXYZ values of a spectrum can be obtained from its camera measurements as - $\hat{x} = M y\,$. (1) + $\hat{x} = M y\,$. (1) - For a set of training spectra arranged in a matrix $C\,$, and the associated camera measurements and CIEXYZ values arranged in the matrices $Y\,$ and $X\,$, $M\,$ can be obtained by minimizing the least squares estimation error + For a set of training spectra arranged in a matrix $C\,$, and the associated camera measurements and CIEXYZ values arranged in the matrices $Y\,$ and $X\,$, $M\,$ can be obtained by minimizing the least squares estimation error - $||MY - X||^2_2$ (2) + $||MY - X||^2_2$ (2) - with respect to $M\,$. + with respect to $M\,$. - == Improving device sensitivity by adding IR to existing channels == + == Improving device sensitivity by adding IR to existing channels == - The wavelength-dependent sensitivity of a color channel depends on the transmittance function of the lens, the transmittance function of the color filter, and the sensitivity function of the photodiode. Ideally, color channels are designed to have sensitivity functions similar to the CIEXYZ color matching functions. For instance, Fig. 1 (a) shows the spectral sensitivity functions of the RGB color channels of a Nikon D100 DSLR camera. Color channels similar to CIEXYZ functions help with improving the accuracy and robustness of the transformation from the color space of the camera to CIEXYZ space. Since such color channels must be relatively narrow-band (over the wavelength range), this restricts the number of photons gathered at each color pixel and reduces the sensitivity of color-sensing pixels. + The wavelength-dependent sensitivity of a color channel depends on the transmittance function of the lens, the transmittance function of the color filter, and the sensitivity function of the photodiode. Ideally, color channels are designed to have sensitivity functions similar to the CIEXYZ color matching functions. For instance, Fig. 1 (a) shows the spectral sensitivity functions of the RGB color channels of a Nikon D100 DSLR camera. Color channels similar to CIEXYZ functions help with improving the accuracy and robustness of the transformation from the color space of the camera to CIEXYZ space. Since such color channels must be relatively narrow-band (over the wavelength range), this restricts the number of photons gathered at each color pixel and reduces the sensitivity of color-sensing pixels. One way to increase the number of photons gathered by a color-sensing pixel is to allow a wider wavelength-range of filter transmittance. The image sensor is naturally sensitive in the near infrared range (NIR) and the color filters can be allowed to transmit photons with energy beyond the visible range of wavelengths. Figure 1(b) shows the relative spectral sensitivities of the RGB color channels of a Nikon D200 DSLR camera which has been modified by removing the IR-blocking filter. One way to increase the number of photons gathered by a color-sensing pixel is to allow a wider wavelength-range of filter transmittance. The image sensor is naturally sensitive in the near infrared range (NIR) and the color filters can be allowed to transmit photons with energy beyond the visible range of wavelengths. Figure 1(b) shows the relative spectral sensitivities of the RGB color channels of a Nikon D200 DSLR camera which has been modified by removing the IR-blocking filter. - [[Image:Cfa rgb nikon.png|thumb|left|320px|'''Figure 1 (a)''' Relative efficiencies of the RGB color channels of a Nikon D100 DSLR camera (RGB).]] + [[Image:Cfa rgb nikon.png|thumb|left|320px]] [[Image:Cfa rgbir.png|thumb|none|320px]] - [[Image:Cfa rgbir.png|thumb|none|320px|'''Figure 1 (b)''' Relative efficiencies of the RGB color channels of a Nikon D200 DSLR without the IR-blocking filter in place (RGBir).]] + - === Accuracy and robustness of color transformation === + === Accuracy and robustness of color transformation === - [[Image:Chromaticity training data.png|thumb|right|320px|'''Figure X.''' Chromaticity coordinates of the 400 surface reflectance spectra in the training set.]] Wide-band color matching functions that deviate very far from the CIEXYZ CMFs have a deleterious effect, both on the quality of color correction, and its robustness to noisy measurements.  Here, we illustrate these effects. We collected a number of surface reflectance functions of common objects and color test charts to form a training set. These spectra were measured with a Photoresearch PR-715 spectrophotometer that allows measurements in the wavelength interval [380, 1168] nm and the spectra in the training set include a good sampling of surfaces of interest (as seen in the chromaticity coordinates of the training spectra in Figure 2). We derived the color transformation matrix ($M$) for the color channels in Fig. 1(a) (RGB) and Fig. 1(b) (RGBir) according to Eq. (2). + + [[Image:Chromaticity training data.png|thumb|right|320px]] Wide-band color matching functions that deviate very far from the CIEXYZ CMFs have a deleterious effect, both on the quality of color correction, and its robustness to noisy measurements. Here, we illustrate these effects. We collected a number of surface reflectance functions of common objects and color test charts to form a training set. These spectra were measured with a Photoresearch PR-715 spectrophotometer that allows measurements in the wavelength interval [380, 1168] nm and the spectra in the training set include a good sampling of surfaces of interest (as seen in the chromaticity coordinates of the training spectra in Figure 2). We derived the color transformation matrix ($M$) for the color channels in Fig. 1(a) (RGB) and Fig. 1(b) (RGBir) according to Eq. (2). - One of the most important spectra in terms of the need for accurate/pleasing color reproduction is that of skin. We illustrate the poor performance of sensors that have RGBir color channels (shown in Fig 1(b)) by comparing the color accuracy of reproductions of the ‘skin’ colored patch (patch 1) in the Macbeth Color Checker (MCC). We simulated 1600 measurements of the ‘skin’ patch at two different noise levels, 40 dB and 20 dB, for sensors with RGB and RGBir color channels. We transformed the camera measurements to CIEXYZ space as in Eq. (1). The chromaticity coordinates of the resulting color-transformed measurements are shown in Figure 3. The top row shows the chromaticity coordinates obtained for measurements made with sensors that have the RGB color channels (illustrated in Figure 1a) for noise levels of 40 dB (left panel) and 20 dB (right panel). The bottom row shows results of similar measurements made with sensors that have the RGBir color channels (illustrated in Figure 1b). As an objective measure of color transformation accuracy, in each plot, we also show the mean and variance of CIELAB ΔE values of the transformed measurements. The sensor with the RGB color channels performs well in terms of both reproduction error and the variance of the error (which is related to the visibility of noise). The sensor with the RGBir color channels fails to provide good estimates even when noise levels are low (ΔE = 8.39 at 40 dB measurement noise). Although better reproduction results can be achieved by appropriately weighting reproduction errors related to skin spectra in the training set, the other serious issue is that of amplification of noise. Note the spread of the values seen in the chromaticity coordinates obtained for measurements with the sensor with RGBir color channels at the noise-level of 20 dB (bottom-right panel). +
One of the most important spectra in terms of the need for accurate/pleasing color reproduction is that of skin. We illustrate the poor performance of sensors that have RGBir color channels (shown in Fig 1(b)) by comparing the color accuracy of reproductions of the ‘skin’ colored patch (patch 1) in the Macbeth Color Checker (MCC). We simulated 1600 measurements of the ‘skin’ patch at two different noise levels, 40 dB and 20 dB, for sensors with RGB and RGBir color channels. We transformed the camera measurements to CIEXYZ space as in Eq. (1). The chromaticity coordinates of the resulting color-transformed measurements are shown in Figure 3. The top row shows the chromaticity coordinates obtained for measurements made with sensors that have the RGB color channels (illustrated in Figure 1a) for noise levels of 40 dB (left panel) and 20 dB (right panel). The bottom row shows results of similar measurements made with sensors that have the RGBir color channels (illustrated in Figure 1b). As an objective measure of color transformation accuracy, in each plot, we also show the mean and variance of CIELAB ΔE values of the transformed measurements. The sensor with the RGB color channels performs well in terms of both reproduction error and the variance of the error (which is related to the visibility of noise). The sensor with the RGBir color channels fails to provide good estimates even when noise levels are low (ΔE = 8.39 at 40 dB measurement noise). Although better reproduction results can be achieved by appropriately weighting reproduction errors related to skin spectra in the training set, the other serious issue is that of amplification of noise. Note the spread of the values seen in the chromaticity coordinates obtained for measurements with the sensor with RGBir color channels at the noise-level of 20 dB (bottom-right panel). - {| border="1" cellpadding="4" cellspacing="0" style="margin: 1em 1em 1em 0; background: #ffffff; border: 1px #aaa solid; border-collapse: collapse; font-size: 100%;" + {| border="1" cellpadding="4" cellspacing="0" style="margin: 1em 1em 1em 0; background: #ffffff; border: 1px #aaa solid; border-collapse: collapse; font-size: 100%;" |- |- - |[[Image:xy rgb pi 40dB.png|thumb|left|320px|(a)RGB, 40 dB]] + | [[Image:Xy rgb pi 40dB.png|thumb|left|320px]] - | + | - [[Image:xy rgb pi 20dB.png|thumb|none|320px|(b) RGB, 20dB]] + [[Image:Xy rgb pi 20dB.png|thumb|none|320px]] + |- |- - |[[Image:xy rgbir pi 40dB.png|thumb|left|320px|(c) RGBir, 40 dB]] + | [[Image:Xy rgbir pi 40dB.png|thumb|left|320px]] - | + | - [[Image:xy rgbir pi 20dB.png|thumb|none|320px|(d) RGBir, 20 dB]] + [[Image:Xy rgbir pi 20dB.png|thumb|none|320px]] + |- |- - |colspan="3" align="left"| '''Figure 3'''. Chromaticity coordinates of color-transformed measurements of the ‘skin’ patch (patch 1) of the Macbeth Color Checker test chart. Results are shown for measurements simulated with: (a) RGB color channels (shown in Fig. 1(a)) at 40 dB, (b) RGB color channels at 20 dB, (c) RGBir color channels (shown in Fig. 1(b)) at 40 dB, and (d) RGBir color channels at 20 dB. The mean and variance of ΔE values are shown in each plot. The red dot shows the true chromaticity coordinates of the ‘skin’ patch of the MCC. + | colspan="3" align="left" | '''Figure 3'''. Chromaticity coordinates of color-transformed measurements of the ‘skin’ patch (patch 1) of the Macbeth Color Checker test chart. Results are shown for measurements simulated with: (a) RGB color channels (shown in Fig. 1(a)) at 40 dB, (b) RGB color channels at 20 dB, (c) RGBir color channels (shown in Fig. 1(b)) at 40 dB, and (d) RGBir color channels at 20 dB. The mean and variance of ΔE values are shown in each plot. The red dot shows the true chromaticity coordinates of the ‘skin’ patch of the MCC. |} |} +
+ [[Image:Snr in vs out rgb rgbir pi.png|thumb|right|360px]] The amplification of noise by color transformation for sensors with the RGBir color channels is shown for simulated measurements of all patches of the MCC in Figure 4. We simulated 1600 measurements of each of the 24 MCC patches with sensors with RGB and RGBir color channels. Input SNR was calculated as the SNR in the green channel of 50% gray patch. We assume that the visibility of noise is quantified by the SNR in the luminance channel (we call this the output SNR) of the same patch. At all noise levels, the luminance channel of the color-transformed output of sensors with RGBir color has an SNR that is approximately 7 dB smaller than the output SNR of sensors with RGB color channels. - [[Image:Snr in vs out rgb rgbir pi.png|thumb|right|360px|'''Figure 4.'''  SNR in the luminance channel of the MCC 50% gray patch as a function of the SNR in the green channel of the same patch.]] The amplification of noise by color transformation for sensors with the RGBir color channels is shown for simulated measurements of all patches of the MCC in Figure 4. We simulated 1600 measurements of each of the 24 MCC patches with sensors with RGB and RGBir color channels. Input SNR was calculated as the SNR in the green channel of 50% gray patch. We assume that the visibility of noise is quantified by the SNR in the luminance channel (we call this the output SNR) of the same patch.  At all noise levels, the luminance channel of the color-transformed output of sensors with RGBir color has an SNR that is approximately 7 dB smaller than the output SNR of sensors with RGB color channels. +
From the results of analyses shown in Figs. 3 and 4 we conclude that including NIR energy in the RGB color channels has an adverse impact on both color reproduction accuracy and the robustness to noisy measurements. Next, we consider the more interesting case in the context of interleaved imaging – the addition of a fourth wideband channel which is achromatic. + === Adding a wideband channel to the RGB color filter sets – the RGBW case  === - From the results of analyses shown in Figs. 3 and 4 we conclude that including NIR energy in the RGB color channels has an adverse impact on both color reproduction accuracy and the robustness to noisy measurements. Next, we consider the more interesting case in the context of interleaved imaging – the addition of a fourth wideband channel which is achromatic. + We have previously described a sensor architecture called interleaved imaging that has four types of color channels; a fourth color sensor that has a wideband transmittance function (W channel) is included in addition to the RGB color channels. NIR energy can be used in this context in two different ways: - ===Adding a wideband channel to the RGB color filter sets – the RGBW case === + #All color channels transmit NIR energy (the wideband channel is transparent and its sensitivity is decided by the sensitivity of the photodetector). We call this the RGBWir color sensor (Fig. 5(a)). - + - We have previously described a sensor architecture called interleaved imaging that has four types of color channels; a fourth color sensor that has a wideband transmittance function (W channel) is included in addition to the RGB color channels. NIR energy can be used in this context in two different ways: + - #All color channels transmit NIR energy (the wideband channel is transparent and its sensitivity is decided by the sensitivity of the photodetector). We call this the RGBWir color sensor (Fig. 5(a)). + #The RGB channels have an IR blocking filter and the wideband channel is transparent. We call this the RGB-Wir color sensor (Fig. 5(b)). #The RGB channels have an IR blocking filter and the wideband channel is transparent. We call this the RGB-Wir color sensor (Fig. 5(b)). - [[Image:Cfa rgbw ir interleaved.png|thumb|left|320px|'''Figure 5 (a)''' Relative spectral sensitivities of sensors with the RGBWir color channels.]] + [[Image:Cfa_rgbw_ir_interleaved.png|thumb|left|320px}]] - [[Image:Cfa rgbsi interleaved.png|thumb|none|320px|'''Figure 1 (b)''' Relative spectral sensitivities of sensors with the RGBir color channels.]] + [[Image:Cfa rgbsi interleaved.png|thumb|none|320px]] - The color transformation for the RGBWir and RGB-Wir color channels are implemented as a multiplication with a 3 x 4 matrix. We found the color transformation matrices as described in Section 2 and applied them to a large number of simulated measurements of the MCC patches. To relate SNR values to visible noise, we converted the CIEXYZ values to sRGB space and arranged the color-transformed values in a grid similar to the arrangement of the MCC. This allows us to present transformed values in smooth regions of reasonable size where the variance in multiple color-transformed values of a single spectrum can be related to the visibility of noise. The results of color transformation at three different noise levels, 40 dB (good light), 30 dB (typical conditions), and 20 dB (low light) are shown in Figure 6. The top row shows results for the RGBWir color channels and the bottom row shows results for the RGB-Wir color channels. Table 1 shows SNR values for each case. Input SNR was calculated in the G channel of the measurements of the 50% gray patch of the MCC and the output SNR was calculated in the luminance channel of color transformed values of the same patch. Note that there was no spatial processing and each pixel was transformed independently. + The color transformation for the RGBWir and RGB-Wir color channels are implemented as a multiplication with a 3 x 4 matrix. We found the color transformation matrices as described in Section 2 and applied them to a large number of simulated measurements of the MCC patches. To relate SNR values to visible noise, we converted the CIEXYZ values to sRGB space and arranged the color-transformed values in a grid similar to the arrangement of the MCC. This allows us to present transformed values in smooth regions of reasonable size where the variance in multiple color-transformed values of a single spectrum can be related to the visibility of noise. The results of color transformation at three different noise levels, 40 dB (good light), 30 dB (typical conditions), and 20 dB (low light) are shown in Figure 6. The top row shows results for the RGBWir color channels and the bottom row shows results for the RGB-Wir color channels. Table 1 shows SNR values for each case. Input SNR was calculated in the G channel of the measurements of the 50% gray patch of the MCC and the output SNR was calculated in the luminance channel of color transformed values of the same patch. Note that there was no spatial processing and each pixel was transformed independently. +
+ {| border="1" cellpadding="2" cellspacing="0" style="margin: 1em 1em 1em 0; background: #ffffff; border: 1px #aaa solid; border-collapse: collapse; font-size: 100%;" + |- + | [[Image:Mcc rgbwir pi 40dB.png|thumb|left|220px]] + | + [[Image:Mcc rgbwir pi 30dB.png|thumb|left|220px]] + + | + [[Image:Mcc rgbwir pi 20dB.png|thumb|left|220px]] - {| border="1" cellpadding="2" cellspacing="0" style="margin: 1em 1em 1em 0; background: #ffffff; border: 1px #aaa solid; border-collapse: collapse; font-size: 100%;" |- |- - |[[Image:Mcc rgbwir pi 40dB.png|thumb|left|220px|(a) RGBWir, 40 dB]] + | [[Image:Mcc rgb wir pi 40dB.png|thumb|left|220px]] - | + | [[Image:Mcc rgb wir pi 30dB.png|thumb|left|220px]] - [[Image:Mcc rgbwir pi 30dB.png|thumb|left|220px|(b) RGBWir, 30 dB]] + | [[Image:Mcc rgb wir pi 20dB.png|thumb|left|220px]] - | + - [[Image:Mcc rgbwir pi 20dB.png|thumb|left|220px|(c) RGBWir, 20 dB]] + |- |- - |[[Image:Mcc rgb wir pi 40dB.png|thumb|left|220px|(d) RGB-Wir, 40 dB]] + | colspan="3" align="left" | '''Figure 6.''' Color-transformed values of camera measurements simulated for sensors with RGBWir (top row) and RGB-Wir (bottom row) color channels at three different noise levels – 40 dB (left column), 30 dB (middle column) and 20 dB (right column). Input SNR was calculated in the G channel of the 50% gray patch. - |[[Image:Mcc rgb wir pi 30dB.png|thumb|left|220px|(e) RGB-Wir, 30 dB]] + - |[[Image:Mcc rgb wir pi 20dB.png|thumb|left|220px|(f) RGB-Wir, 20 dB]] + - |- + - |colspan="3" align="left"| '''Figure 6.''' Color-transformed values of camera measurements simulated for sensors with RGBWir (top row) and RGB-Wir (bottom row) color channels at three different noise levels – 40 dB (left column), 30 dB (middle column) and 20 dB (right column). Input SNR was calculated in the G channel of the 50% gray patch. + |} |} +
- {| border="1" cellpadding="4" cellspacing="0" align="center" + {| border="1" cellpadding="4" cellspacing="0" align="center" - |- align=center + |- align="center" - |colspan="3" align="left"| '''Table 1.''' SNR values for the MCC measurements shown in Fig. 6 + | colspan="3" align="left" | '''Table 1.''' SNR values for the MCC measurements shown in Fig. 6 - |- align=center + |- align="center" - |style="background:#ffdead;"|Input SNR (dB) + | style="background:#ffdead;" | Input SNR (dB) - |style="background:#ffdead;" colspan="2"| Output SNR (dB) + | style="background:#ffdead;" colspan="2" | Output SNR (dB) - |- + |- align="center" - |- align=center + |   - |  + | style="background:#efefef;" | RGBWir - |style="background:#efefef;"|RGBWir + | style="background:#efefef;" | RGB-Wir - |style="background:#efefef;"|RGB-Wir + |- align="center" - |- align=center + | 40 - |40 + | 29.43 - |29.43 + | 37.23 - |37.23 + |- align="center" - |- align=center + | 30 - |30 + | 19.94 - |19.94 + | 28.95 - |28.95 + |- align="center" - |- align=center + | 20 - |20 + | 10.13 - |10.13 + | 18.29 - |18.29 + |} |} - +
The color transformation for sensors with the RGB-Wir color channels is more robust to noise. Although the results obtained for sensors with the RGBWir color channels are acceptable when SNR in the camera measurements is high, the results become unacceptable even for typical lighting conditions (Fig. 6(b)). In conditions expected in low-light (SNR ~ 20dB), the sensor with the RGBWir color channels amplifies the noise to a large extent (Fig. 6(c)). The trends in SNR and color accuracy can be observed more clearly in Figures 7 and 8. - The color transformation for sensors with the RGB-Wir color channels is more robust to noise. Although the results obtained for sensors with the RGBWir color channels are acceptable when SNR in the camera measurements is high, the results become unacceptable even for typical lighting conditions (Fig. 6(b)). In conditions expected in low-light (SNR ~ 20dB), the sensor with the RGBWir color channels amplifies the noise to a large extent (Fig. 6(c)). The trends in SNR and color accuracy can be observed more clearly in Figures 7 and 8. + - + - Figure 7 shows the variation of CIELAB ΔE values as the input noise level increases (SNR decreases) for the sensors with RGBWir (Fig. 7(a)) and RGB-Wir (Fig. 7(b)) color channels. The ΔE values shown in the figure are the average ΔE values of the color-transformed values of multiple simulated measurements of the 24 MCC patches at different levels of input SNR. The difference in ΔE for sensors with the RGBWir and RGB-Wir color channels is not very significant.  Figure 8 shows plots of input SNR vs. output SNR for the same measurements. The sensor with RGB-Wir (Fig. 8(b)) color channels performs reasonably well; there is very little amplification of input noise. For instance, note that for an input noise level of 25 dB, the output noise level is 25 dB. But the sensor with the RGBWir color channels significantly amplifies input noise (Fig. 8(a)). Note that for an input noise level of 25 dB, the output noise level is only 15 dB. As before, input SNR was calculated in the G channel of the 50% gray patch and output SNR was calculated in the luminance channel of the same patch. + + Figure 7 shows the variation of CIELAB ΔE values as the input noise level increases (SNR decreases) for the sensors with RGBWir (Fig. 7(a)) and RGB-Wir (Fig. 7(b)) color channels. The ΔE values shown in the figure are the average ΔE values of the color-transformed values of multiple simulated measurements of the 24 MCC patches at different levels of input SNR. The difference in ΔE for sensors with the RGBWir and RGB-Wir color channels is not very significant. Figure 8 shows plots of input SNR vs. output SNR for the same measurements. The sensor with RGB-Wir (Fig. 8(b)) color channels performs reasonably well; there is very little amplification of input noise. For instance, note that for an input noise level of 25 dB, the output noise level is 25 dB. But the sensor with the RGBWir color channels significantly amplifies input noise (Fig. 8(a)). Note that for an input noise level of 25 dB, the output noise level is only 15 dB. As before, input SNR was calculated in the G channel of the 50% gray patch and output SNR was calculated in the luminance channel of the same patch. +
+
- + {| border="1" cellpadding="2" cellspacing="0" style="margin: 1em 1em 1em 0; background: #ffffff; border: 1px #aaa solid; border-collapse: collapse; font-size: 100%;" - {| border="1" cellpadding="2" cellspacing="0" style="margin: 1em 1em 1em 0; background: #ffffff; border: 1px #aaa solid; border-collapse: collapse; font-size: 100%;" + |- |- - |[[Image:Snr vs delE rgbwir pi.png||thumb|left|320px|(a)]] + | [[Image:Snr vs delE rgbwir pi.png - |[[Image:Snr vs delE rgb wir pi.png|thumb|left|320px|(b)]] + | left|320px|(a)]] + | [[Image:Snr vs delE rgb wir pi.png|thumb|left|320px]] |- |- - |colspan="3" align="left"| '''Figure 7.''' Average CIELAB ΔE values of measurements of the 24 MCC patches shown as a function of the SNR in the green channel of the same patch for (a) the sensor with RGBWir color channels, and (b) the sensor with RGB-Wir color channels. + | colspan="3" align="left" | '''Figure 7.''' Average CIELAB ΔE values of measurements of the 24 MCC patches shown as a function of the SNR in the green channel of the same patch for (a) the sensor with RGBWir color channels, and (b) the sensor with RGB-Wir color channels. |} |} - {| border="1" cellpadding="2" cellspacing="0" style="margin: 1em 1em 1em 0; background: #ffffff; border: 1px #aaa solid; border-collapse: collapse; font-size: 100%;" + {| border="1" cellpadding="2" cellspacing="0" style="margin: 1em 1em 1em 0; background: #ffffff; border: 1px #aaa solid; border-collapse: collapse; font-size: 100%;" |- |- - |[[Image:Snr in vs out pi rgbwir.png||thumb|left|320px|(a)]] + | [[Image:Snr in vs out pi rgbwir.png - |[[Image:Snr in vs out pi rgb wir.png|thumb|left|320px|(b)]] + | left|320px|(a)]] + | [[Image:Snr in vs out pi rgb wir.png|thumb|left|320px]] |- |- - |colspan="3" align="left"| '''Figure 8.''' SNR in the luminance channel of the MCC 50% gray patch as a function of the SNR in the green channel of the same patch for (a) the sensor with the RGBWir color channels, and (b) the sensor with the RGB-Wir color channels. + | colspan="3" align="left" | '''Figure 8.''' SNR in the luminance channel of the MCC 50% gray patch as a function of the SNR in the green channel of the same patch for (a) the sensor with the RGBWir color channels, and (b) the sensor with the RGB-Wir color channels. |} |} - It is clear that the color transformation stage used for filter sets with RGB filters that admit NIR energy has noise amplifying properties. Consequently, there exists a trade-off between increasing color sensor sensitivity by widening the filter transmittance functions and maintaining color accuracy. In spite of the sensor with RGB-Wir color filters being superior in terms of color correction performance, the sensor with the RGBWir filter set may be preferred for the following reasons: + It is clear that the color transformation stage used for filter sets with RGB filters that admit NIR energy has noise amplifying properties. Consequently, there exists a trade-off between increasing color sensor sensitivity by widening the filter transmittance functions and maintaining color accuracy. In spite of the sensor with RGB-Wir color filters being superior in terms of color correction performance, the sensor with the RGBWir filter set may be preferred for the following reasons: - #A sensor with RGBWir color channels can be easily realized (by removing the IR blocking filter) + + #A sensor with RGBWir color channels can be easily realized (by removing the IR blocking filter) #In a sensor with the RGBWir color channels, the photon gathering capacity of all four color channels is increased. This has implications both in achieving good performance in low light conditions and in matching the saturation exposure-time of the different color channels. #In a sensor with the RGBWir color channels, the photon gathering capacity of all four color channels is increased. This has implications both in achieving good performance in low light conditions and in matching the saturation exposure-time of the different color channels. - ==RGBWir color channels with different IR cutoff wavelengths== + == RGBWir color channels with different IR cutoff wavelengths == + It is possible to use RGBWir sensor if a limited range of wavelengths of the NIR energy is used instead of the entire sensitivity range of the photodetector. With this approach one can gain some improvements in sensitivity at an acceptable cost to color correction accuracy and robustness. We present results of simulations of measurements of MCC patches with the RGBWir filter set done by varying the IR cutoff wavelength in Figure 9. The left panel shows the variation in average ΔE values of the color-transformed measurements for three different noise levels, 20 dB, 30 dB, and 40 dB. The right channel shows the variation in output SNR in the 50% gray patch for these simulations. It is possible to use RGBWir sensor if a limited range of wavelengths of the NIR energy is used instead of the entire sensitivity range of the photodetector. With this approach one can gain some improvements in sensitivity at an acceptable cost to color correction accuracy and robustness. We present results of simulations of measurements of MCC patches with the RGBWir filter set done by varying the IR cutoff wavelength in Figure 9. The left panel shows the variation in average ΔE values of the color-transformed measurements for three different noise levels, 20 dB, 30 dB, and 40 dB. The right channel shows the variation in output SNR in the 50% gray patch for these simulations. +
- {| border="1" cellpadding="2" cellspacing="0" style="margin: 1em 1em 1em 0; background: #ffffff; border: 1px #aaa solid; border-collapse: collapse; font-size: 100%;" + {| border="1" cellpadding="2" cellspacing="0" style="margin: 1em 1em 1em 0; background: #ffffff; border: 1px #aaa solid; border-collapse: collapse; font-size: 100%;" |- |- - |[[Image:Ircutoff vs dE rgbwir 20 30 40.png||thumb|left|320px|(a)]] + | [[Image:Ircutoff vs dE rgbwir 20 30 40.png - |[[Image:Ircutoff vs snr rgbwir 20 30 40.png|thumb|left|320px|(b)]] + | left|320px|(a)]] + | [[Image:Ircutoff vs snr rgbwir 20 30 40.png|thumb|left|320px]] |- |- - |colspan="3" align="left"| '''Figure 9.''' The effect of changing the cutoff wavelength of the IR blocking filter. (a) The change in ΔE values as a function of IR cutoff wavelength for three different noise levels. (b) The change in output SNR in the MCC 50% gray patch as a function of IR cutoff wavelength for three different noise levels. + | colspan="3" align="left" | '''Figure 9.''' The effect of changing the cutoff wavelength of the IR blocking filter. (a) The change in ΔE values as a function of IR cutoff wavelength for three different noise levels. (b) The change in output SNR in the MCC 50% gray patch as a function of IR cutoff wavelength for three different noise levels. |} |} +
Figure 9 shows that a reasonable compromise between gain in sensitivity and loss in color correction accuracy may be reached by setting the IR cutoff wavelength at 800 nm. Note that this conclusion applies specifically to the color filter transmittance functions shown in Fig. 5(a). It is conceivable that an optimal set of 4 color filter transmittances may be found for better results. This is an area of future research. - Figure 9 shows that a reasonable compromise between gain in sensitivity and loss in color correction accuracy may be reached by setting the IR cutoff wavelength at 800 nm. Note that this conclusion applies specifically to the color filter transmittance functions shown in Fig. 5(a). It is conceivable that an optimal set of 4 color filter transmittances may be found for better results. This is an area of future research. + When input SNR is low (20 dB), we see that the RGBWir filter set will not provide acceptable color accuracy (ΔE > 15 for IR cutoffs > 800 nm). Conventional color transformation methods cannot be reliably used to correct noisy RGBWir measurements and we need an improved method to convert RGBWir measurements to CIEXYZ values. - When input SNR is low (20 dB), we see that the RGBWir filter set will not provide acceptable color accuracy (ΔE > 15 for IR cutoffs > 800 nm). Conventional color transformation methods cannot be reliably used to correct noisy RGBWir measurements and we need an improved method to convert RGBWir measurements to CIEXYZ values. + == Color transformation using signal and noise statistics == - == Color transformation using signal and noise statistics == + There is a need to understand how correlations between the energy in visible and NIR range can be used to improve color correction performance. The color correction method described in Section 2 implicitly uses NIR spectral statistics since the transform is derived from a training set that has information in both the visible and NIR ranges. But, a non-deterministic approach to color correction is called for in this particular application. The advantage of such an approach is that it will allow us to explicitly account for the presence of noise in the measurements. - There is a need to understand how correlations between the energy in visible and NIR range can be used to improve color correction performance. The color correction method described in Section 2 implicitly uses NIR spectral statistics since the transform is derived from a training set that has information in both the visible and NIR ranges. But, a non-deterministic approach to color correction is called for in this particular application. The advantage of such an approach is that it will allow us to explicitly account for the presence of noise in the measurements. + [[Image:Spectrum covariance matrix.png|thumb|right|384px]]The amplification of noise during color correction can be a serious problem (mp will put up some figures showing this..). If some information about the signal and noise is known (the first and second order moments), one can solve the problem of color-correction in the presence of noise is to use an affine minimum mean squared estimator (also commonly called the linear MMSE). In this method, we find the optimal estimate $\hat{x}$ of $x\,$ that can be expressed as - [[Image:Spectrum covariance matrix.png|thumb|right|384px|'''Figure X.'''  The covariance matrix found from a set of 400 surface reflectance spectra of common objects and test targets measured with a PR-715 spectraphotometer. The green dashed square shows the approximate extent of the visible range of the spectrum.]]The amplification of noise during color correction can be a serious problem (mp will put up some figures showing this..). If some information about the signal and noise is known (the first and second order moments), one can solve the problem of color-correction in the presence of noise is to use an affine minimum mean squared estimator (also commonly called the linear MMSE). In this method, we find the optimal estimate $\hat{x}$ of $x\,$ that can be expressed as + $\hat{x} = W y + b$ - $\hat{x} = W y + b$ + The optimal estimate is found by minimizing the expected estimation error - The optimal estimate is found by minimizing the expected estimation error + $E \{|| \hat{x} - {x} ||^2 \}$ - + - $E \{|| \hat{x} - {x} ||^2 \}$ + The optimal estimate is obtained for The optimal estimate is obtained for $[itex] - W = A^TR_{xx} S (S^T R_{xx} S + R_{nn})^{-1}\,$ and + W = A^TR_{xx} S (S^T R_{xx} S + R_{nn})^{-1}\, [/itex] and $[itex] b = E\{x\} - W(E\{y\} - E\{n\})\, , b = E\{x\} - W(E\{y\} - E\{n\})\, , -$ + [/itex] - where $R_{xx}\,$ and $R_{nn}\,$ are the signal and noise autocorrelation matrices. The signal autocorrelation matrix is found from the spectra in the training set. Figure 10 shows a visualization of $R_{xx}\,$ found from the training set described in Section 3 (Fig. 2). + where $R_{xx}\,$ and $R_{nn}\,$ are the signal and noise autocorrelation matrices. The signal autocorrelation matrix is found from the spectra in the training set. Figure 10 shows a visualization of $R_{xx}\,$ found from the training set described in Section 3 (Fig. 2). - We repeated our simulations for the sensor with RGBWir color filters with the no IR cutoff filter. Simulated measurements of the MCC patches were transformed with the conventional pseudoinverse-based color transformation method and the MMSE-based estimate. A comparison of the two methods is shown in Figure 11. The input SNR was calculated in the green channel of the MCC 50% gray patch and the output SNR was calculated in the luminance channel of the same patch. The MMSE-based method significantly outperforms the conventional method. + We repeated our simulations for the sensor with RGBWir color filters with the no IR cutoff filter. Simulated measurements of the MCC patches were transformed with the conventional pseudoinverse-based color transformation method and the MMSE-based estimate. A comparison of the two methods is shown in Figure 11. The input SNR was calculated in the green channel of the MCC 50% gray patch and the output SNR was calculated in the luminance channel of the same patch. The MMSE-based method significantly outperforms the conventional method. +
- {| border="1" cellpadding="2" cellspacing="0" style="margin: 1em 1em 1em 0; background: #ffffff; border: 1px #aaa solid; border-collapse: collapse; font-size: 100%;" + {| border="1" cellpadding="2" cellspacing="0" style="margin: 1em 1em 1em 0; background: #ffffff; border: 1px #aaa solid; border-collapse: collapse; font-size: 100%;" |- |- - |[[Image:Snr vs delE pi am rgbir.png||thumb|left|320px|(a)]] + | [[Image:Snr vs delE pi am rgbir.png - |[[Image:Snr in vs out rgbwir pi am.png|thumb|left|320px|(b)]] + | left|320px|(a)]] + | [[Image:Snr in vs out rgbwir pi am.png|thumb|left|320px]] |- |- - |colspan="3" align="left"| '''Figure 11.''' Comparisons of conventional color transformation (pseudoinverse) with MMSE-based color transformation. (a) Average ΔE over the MCC patches shown as a function of input SNR. (b) Output SNR shown as a function of input SNR. + | colspan="3" align="left" | '''Figure 11.''' Comparisons of conventional color transformation (pseudoinverse) with MMSE-based color transformation. (a) Average ΔE over the MCC patches shown as a function of input SNR. (b) Output SNR shown as a function of input SNR. |} |} +
- {| border="1" cellpadding="4" cellspacing="0" align="center" + {| border="1" cellpadding="4" cellspacing="0" align="center" - |- align=center + |- align="center" - |colspan="3" align="left"| '''Table 2.''' Objective metric values for the simulation results shown in Fig. 12 + | colspan="3" align="left" | '''Table 2.''' Objective metric values for the simulation results shown in Fig. 12 - |- align=center + |- align="center" - |colspan="3" style="background:#ffdead;"| Input SNR = 25 dB + | colspan="3" style="background:#ffdead;" | Input SNR = 25 dB - |- align=center + |- align="center" - |  + |   - |style="background:#bbbbbb;"|Output SNR + | style="background:#bbbbbb;" | Output SNR - |style="background:#bbbbbb;"|CIELAB $\Delta$E + | style="background:#bbbbbb;" | CIELAB $\Delta$E - |- align=center + |- align="center" - |Pseudoinverse + | Pseudoinverse - |14.88 dB + | 14.88 dB - |13.89 + | 13.89 - |- align=center + |- align="center" - |MMSE + | MMSE - |22.31 dB + | 22.31 dB - |7.38 + | 7.38 |} |} +
- {| border="1" cellpadding="2" cellspacing="0" style="margin: 1em 1em 1em 0; background: #ffffff; border: 1px #aaa solid; border-collapse: collapse; font-size: 100%;" + {| border="1" cellpadding="2" cellspacing="0" style="margin: 1em 1em 1em 0; background: #ffffff; border: 1px #aaa solid; border-collapse: collapse; font-size: 100%;" |- |- - |[[Image:Mcc rgbwir pi 25dB.png||thumb|left|320px|(a)]] + | [[Image:Mcc rgbwir pi 25dB.png - |[[Image:Mcc rgbwir am 25dB.png|thumb|left|320px|(b)]] + | left|320px|(a)]] + | [[Image:Mcc rgbwir am 25dB.png|thumb|left|320px]] |- |- - |colspan="3" align="left"| '''Figure 11.''' Comparisons of conventional color transformation (pseudoinverse) with MMSE-based color transformation. (a) Average ΔE over the MCC patches shown as a function of input SNR. (b) Output SNR shown as a function of input SNR. + | colspan="3" align="left" | '''Figure 11.''' Comparisons of conventional color transformation (pseudoinverse) with MMSE-based color transformation. (a) Average ΔE over the MCC patches shown as a function of input SNR. (b) Output SNR shown as a function of input SNR. |} |} - Note that the MMSE-based method requires knowledge of the signal and noise correlation matrices. Although the signal correlation matrix can be obtained from the training set, the noise correlation matrix needs to be estimated. In our experiments we assumed knowledge of noise statistics. In practice, noise statistics will not be available and will need to be estimated from the measurements. This is an area of future research. + Note that the MMSE-based method requires knowledge of the signal and noise correlation matrices. Although the signal correlation matrix can be obtained from the training set, the noise correlation matrix needs to be estimated. In our experiments we assumed knowledge of noise statistics. In practice, noise statistics will not be available and will need to be estimated from the measurements. This is an area of future research. - ==Conclusions and future work== + == Conclusions and future work == - It is very important to design appropriate color transformation methods for sensors with color channels that admit NIR energy. In such filter sets conventional methods of color transformation do not perform well in terms of color correction accuracy. Moreover, conversion of measurements that contain NIR energy to a display space leads to amplification of any noise present in the measurements. + It is very important to design appropriate color transformation methods for sensors with color channels that admit NIR energy. In such filter sets conventional methods of color transformation do not perform well in terms of color correction accuracy. Moreover, conversion of measurements that contain NIR energy to a display space leads to amplification of any noise present in the measurements. - In this report we detailed the following issues for filter sets that admit NIR energy: + In this report we detailed the following issues for filter sets that admit NIR energy: - #The effect of increasing levels of measurement noise (decreasing light levels) on the noise levels in the conventional color-transformed output. + + #The effect of increasing levels of measurement noise (decreasing light levels) on the noise levels in the conventional color-transformed output. #The tradeoffs involved in color accuracy and noise amplification for color filter sets with different amounts of sensitivity in the NIR range. #The tradeoffs involved in color accuracy and noise amplification for color filter sets with different amounts of sensitivity in the NIR range. #A method for color conversion that accounts for noise and signal statistics for improved performance for NIR-sensitive color filter sets. #A method for color conversion that accounts for noise and signal statistics for improved performance for NIR-sensitive color filter sets. - Our main conclusions are: + Our main conclusions are: - #A sensor that has all four color channels sensitive in the visible and NIR range can give acceptable results when the measured signal has high SNR. But as input noise level increases, the noise-levels in the color-transformed signals become unacceptably high. + - #A sensor that has three color channels sensitive in the visible range (RGB) and a fourth wideband channel sensitive in the NIR range performs well in terms of accuracy and robustness to measurement noise. + #A sensor that has all four color channels sensitive in the visible and NIR range can give acceptable results when the measured signal has high SNR. But as input noise level increases, the noise-levels in the color-transformed signals become unacceptably high. + #A sensor that has three color channels sensitive in the visible range (RGB) and a fourth wideband channel sensitive in the NIR range performs well in terms of accuracy and robustness to measurement noise. #A color transform derived as the minimum mean squared estimate of CIEXYZ values from color measurements can be used to improve the performance of the sensor with spectral sensitivities described in (1) by using the statistics of surface spectra and noise. #A color transform derived as the minimum mean squared estimate of CIEXYZ values from color measurements can be used to improve the performance of the sensor with spectral sensitivities described in (1) by using the statistics of surface spectra and noise. - +
+ + In the next stages of the project we propose to study the following problems: - In the next stages of the project we propose to study the following problems: + #Methods to analyze color correction in the context of interleaved imaging. Specifically, the wideband channel will be used to denoise the color channels before color correction. - #Methods to analyze color correction in the context of interleaved imaging. Specifically, the wideband channel will be used to denoise the color channels before color correction. + #Improving the color transformation of sensors with NIR-admitting color channels by using estimated noise statistics. - #Improving the color transformation of sensors with NIR-admitting color channels by using estimated noise statistics. + #The effect of NIR and noise on color saturation. #The effect of NIR and noise on color saturation.

## Revision as of 13:58, 9 August 2015

Manu had been analyzing the consequences of noisy data for color balancing algorithms. Specifically we are looking at how the noise is amplified (or not) in the color corrected space. Over the years we have recognized that a color transform can be very good for getting the means right, but the noise shows up badly. This should be clear from the properties of the color transform; these were analyzed in one project by Manu and in another by UB.

Manu has discovered that properly accounting for the measurement noise covariance has a very positive effect on the estimations. We want to return to this problem and write a longer paper on this.

This is interesting again because of the interest in using color channels with non-traditional sensors (not just RGB). This can be very problematic in terms of the amplification of measurement noise.

Much of this was summarized in the June 2009 report to Samsung. We are editing for new ideas and a paper here.

# Background

A key issue that arises while considering color filters with wideband sensitivity functions for increasing imaging device sensitivity is the design of an appropriate color correction method. This is especially important if color filters admit NIR energy. If the color transformation is not carefully designed, any improvement in sensitivity gained by admitting a larger number of photons can be negated by the amplification of noise in the color transformation stage. In this report we discuss the accuracy and robustness of conventional color transforms (that are implemented as matrix multiplies) applied to various types of color filter sets with NIR sensitivity. We show the effect of increasing levels of measurement noise (decreasing light levels) on the noise levels in the color-transformed output. Also, we show the tradeoffs involved in color accuracy and noise amplification for color filter sets with different amounts of sensitivity in the NIR range. We found that:

1. A color filter set with all four color sensors sensitive in the visible and NIR range can give acceptable results when the measured signal has high SNR. But as input noise level increases, the noise-levels in the color-transformed signals become unacceptably high.
2. A color filter set with three color sensors sensitive in the visible range (RGB) and a fourth wideband sensor sensitive in the NIR range performs well in terms of accuracy and robustness to measurement noise.
3. A color transform derived as the minimum mean squared estimate of CIEXYZ values from color measurements can be used to improve the performance of the color filter set in (1) by using the statistics of surface spectra and noise.

## Introduction

Image sensors can not naturally discriminate between signals with energy in different wavelength regions. The sensor simply produces a signal proportional to all incident energy in its sensitivity range. Typical color cameras use color filters with different wavelength-dependent transmittances to distinguish between three or more separate wavelength bands (color channels). Camera measurements of a stimulus represent energy captured by the color channels. To display images, camera measurements must be converted to a form that can be related to intensities evoked by the same stimulus in the human visual system. Color conversion from camera space to CIEXYZ space helps with this transformation.

## Color transformation

To confine the discussion to the properties of the color transformation, in this report we will assume that the illuminant under which signals are acquired is known. The experiments and methods described here can be generalized by considering an accurate illuminant estimation scheme in the framework. Here, we assume that all spectra are acquired under an illuminant that has equal energy along the wavelength range and the task is to convert camera measurements to CIEXYZ space. In matrix-vector notation, we write

$y = S^T c\,,$ and

$x = A^T c\,,$

where the columns of the matrices $S\,$ and $A\,$ are the spectral sensitivities of the camera color filters and the CIEXYZ color matching functions respectively. One way to find the mapping from camera-colorspace to CIEXYZ space is to find a linear transformation (represented by the matrix $M$ ) so that the estimate $\hat{x}$ of the CIEXYZ values of a spectrum can be obtained from its camera measurements as

$\hat{x} = M y\,$. (1)

For a set of training spectra arranged in a matrix $C\,$, and the associated camera measurements and CIEXYZ values arranged in the matrices $Y\,$ and $X\,$, $M\,$ can be obtained by minimizing the least squares estimation error

$||MY - X||^2_2$ (2)

with respect to $M\,$.

## Improving device sensitivity by adding IR to existing channels

The wavelength-dependent sensitivity of a color channel depends on the transmittance function of the lens, the transmittance function of the color filter, and the sensitivity function of the photodiode. Ideally, color channels are designed to have sensitivity functions similar to the CIEXYZ color matching functions. For instance, Fig. 1 (a) shows the spectral sensitivity functions of the RGB color channels of a Nikon D100 DSLR camera. Color channels similar to CIEXYZ functions help with improving the accuracy and robustness of the transformation from the color space of the camera to CIEXYZ space. Since such color channels must be relatively narrow-band (over the wavelength range), this restricts the number of photons gathered at each color pixel and reduces the sensitivity of color-sensing pixels.

One way to increase the number of photons gathered by a color-sensing pixel is to allow a wider wavelength-range of filter transmittance. The image sensor is naturally sensitive in the near infrared range (NIR) and the color filters can be allowed to transmit photons with energy beyond the visible range of wavelengths. Figure 1(b) shows the relative spectral sensitivities of the RGB color channels of a Nikon D200 DSLR camera which has been modified by removing the IR-blocking filter.

### Accuracy and robustness of color transformation

Wide-band color matching functions that deviate very far from the CIEXYZ CMFs have a deleterious effect, both on the quality of color correction, and its robustness to noisy measurements. Here, we illustrate these effects. We collected a number of surface reflectance functions of common objects and color test charts to form a training set. These spectra were measured with a Photoresearch PR-715 spectrophotometer that allows measurements in the wavelength interval [380, 1168] nm and the spectra in the training set include a good sampling of surfaces of interest (as seen in the chromaticity coordinates of the training spectra in Figure 2). We derived the color transformation matrix ($M$) for the color channels in Fig. 1(a) (RGB) and Fig. 1(b) (RGBir) according to Eq. (2).

One of the most important spectra in terms of the need for accurate/pleasing color reproduction is that of skin. We illustrate the poor performance of sensors that have RGBir color channels (shown in Fig 1(b)) by comparing the color accuracy of reproductions of the ‘skin’ colored patch (patch 1) in the Macbeth Color Checker (MCC). We simulated 1600 measurements of the ‘skin’ patch at two different noise levels, 40 dB and 20 dB, for sensors with RGB and RGBir color channels. We transformed the camera measurements to CIEXYZ space as in Eq. (1). The chromaticity coordinates of the resulting color-transformed measurements are shown in Figure 3. The top row shows the chromaticity coordinates obtained for measurements made with sensors that have the RGB color channels (illustrated in Figure 1a) for noise levels of 40 dB (left panel) and 20 dB (right panel). The bottom row shows results of similar measurements made with sensors that have the RGBir color channels (illustrated in Figure 1b). As an objective measure of color transformation accuracy, in each plot, we also show the mean and variance of CIELAB ΔE values of the transformed measurements. The sensor with the RGB color channels performs well in terms of both reproduction error and the variance of the error (which is related to the visibility of noise). The sensor with the RGBir color channels fails to provide good estimates even when noise levels are low (ΔE = 8.39 at 40 dB measurement noise). Although better reproduction results can be achieved by appropriately weighting reproduction errors related to skin spectra in the training set, the other serious issue is that of amplification of noise. Note the spread of the values seen in the chromaticity coordinates obtained for measurements with the sensor with RGBir color channels at the noise-level of 20 dB (bottom-right panel).

 Figure 3. Chromaticity coordinates of color-transformed measurements of the ‘skin’ patch (patch 1) of the Macbeth Color Checker test chart. Results are shown for measurements simulated with: (a) RGB color channels (shown in Fig. 1(a)) at 40 dB, (b) RGB color channels at 20 dB, (c) RGBir color channels (shown in Fig. 1(b)) at 40 dB, and (d) RGBir color channels at 20 dB. The mean and variance of ΔE values are shown in each plot. The red dot shows the true chromaticity coordinates of the ‘skin’ patch of the MCC.

The amplification of noise by color transformation for sensors with the RGBir color channels is shown for simulated measurements of all patches of the MCC in Figure 4. We simulated 1600 measurements of each of the 24 MCC patches with sensors with RGB and RGBir color channels. Input SNR was calculated as the SNR in the green channel of 50% gray patch. We assume that the visibility of noise is quantified by the SNR in the luminance channel (we call this the output SNR) of the same patch. At all noise levels, the luminance channel of the color-transformed output of sensors with RGBir color has an SNR that is approximately 7 dB smaller than the output SNR of sensors with RGB color channels.

From the results of analyses shown in Figs. 3 and 4 we conclude that including NIR energy in the RGB color channels has an adverse impact on both color reproduction accuracy and the robustness to noisy measurements. Next, we consider the more interesting case in the context of interleaved imaging – the addition of a fourth wideband channel which is achromatic.

### Adding a wideband channel to the RGB color filter sets – the RGBW case

We have previously described a sensor architecture called interleaved imaging that has four types of color channels; a fourth color sensor that has a wideband transmittance function (W channel) is included in addition to the RGB color channels. NIR energy can be used in this context in two different ways:

1. All color channels transmit NIR energy (the wideband channel is transparent and its sensitivity is decided by the sensitivity of the photodetector). We call this the RGBWir color sensor (Fig. 5(a)).
2. The RGB channels have an IR blocking filter and the wideband channel is transparent. We call this the RGB-Wir color sensor (Fig. 5(b)).
320px}

The color transformation for the RGBWir and RGB-Wir color channels are implemented as a multiplication with a 3 x 4 matrix. We found the color transformation matrices as described in Section 2 and applied them to a large number of simulated measurements of the MCC patches. To relate SNR values to visible noise, we converted the CIEXYZ values to sRGB space and arranged the color-transformed values in a grid similar to the arrangement of the MCC. This allows us to present transformed values in smooth regions of reasonable size where the variance in multiple color-transformed values of a single spectrum can be related to the visibility of noise. The results of color transformation at three different noise levels, 40 dB (good light), 30 dB (typical conditions), and 20 dB (low light) are shown in Figure 6. The top row shows results for the RGBWir color channels and the bottom row shows results for the RGB-Wir color channels. Table 1 shows SNR values for each case. Input SNR was calculated in the G channel of the measurements of the 50% gray patch of the MCC and the output SNR was calculated in the luminance channel of color transformed values of the same patch. Note that there was no spatial processing and each pixel was transformed independently.

 Figure 6. Color-transformed values of camera measurements simulated for sensors with RGBWir (top row) and RGB-Wir (bottom row) color channels at three different noise levels – 40 dB (left column), 30 dB (middle column) and 20 dB (right column). Input SNR was calculated in the G channel of the 50% gray patch.

 Table 1. SNR values for the MCC measurements shown in Fig. 6 Input SNR (dB) Output SNR (dB) RGBWir RGB-Wir 40 29.43 37.23 30 19.94 28.95 20 10.13 18.29

The color transformation for sensors with the RGB-Wir color channels is more robust to noise. Although the results obtained for sensors with the RGBWir color channels are acceptable when SNR in the camera measurements is high, the results become unacceptable even for typical lighting conditions (Fig. 6(b)). In conditions expected in low-light (SNR ~ 20dB), the sensor with the RGBWir color channels amplifies the noise to a large extent (Fig. 6(c)). The trends in SNR and color accuracy can be observed more clearly in Figures 7 and 8.

Figure 7 shows the variation of CIELAB ΔE values as the input noise level increases (SNR decreases) for the sensors with RGBWir (Fig. 7(a)) and RGB-Wir (Fig. 7(b)) color channels. The ΔE values shown in the figure are the average ΔE values of the color-transformed values of multiple simulated measurements of the 24 MCC patches at different levels of input SNR. The difference in ΔE for sensors with the RGBWir and RGB-Wir color channels is not very significant. Figure 8 shows plots of input SNR vs. output SNR for the same measurements. The sensor with RGB-Wir (Fig. 8(b)) color channels performs reasonably well; there is very little amplification of input noise. For instance, note that for an input noise level of 25 dB, the output noise level is 25 dB. But the sensor with the RGBWir color channels significantly amplifies input noise (Fig. 8(a)). Note that for an input noise level of 25 dB, the output noise level is only 15 dB. As before, input SNR was calculated in the G channel of the 50% gray patch and output SNR was calculated in the luminance channel of the same patch.

 [[Image:Snr vs delE rgbwir pi.png left|320px|(a)]] Figure 7. Average CIELAB ΔE values of measurements of the 24 MCC patches shown as a function of the SNR in the green channel of the same patch for (a) the sensor with RGBWir color channels, and (b) the sensor with RGB-Wir color channels.
 [[Image:Snr in vs out pi rgbwir.png left|320px|(a)]] Figure 8. SNR in the luminance channel of the MCC 50% gray patch as a function of the SNR in the green channel of the same patch for (a) the sensor with the RGBWir color channels, and (b) the sensor with the RGB-Wir color channels.

It is clear that the color transformation stage used for filter sets with RGB filters that admit NIR energy has noise amplifying properties. Consequently, there exists a trade-off between increasing color sensor sensitivity by widening the filter transmittance functions and maintaining color accuracy. In spite of the sensor with RGB-Wir color filters being superior in terms of color correction performance, the sensor with the RGBWir filter set may be preferred for the following reasons:

1. A sensor with RGBWir color channels can be easily realized (by removing the IR blocking filter)
2. In a sensor with the RGBWir color channels, the photon gathering capacity of all four color channels is increased. This has implications both in achieving good performance in low light conditions and in matching the saturation exposure-time of the different color channels.

## RGBWir color channels with different IR cutoff wavelengths

It is possible to use RGBWir sensor if a limited range of wavelengths of the NIR energy is used instead of the entire sensitivity range of the photodetector. With this approach one can gain some improvements in sensitivity at an acceptable cost to color correction accuracy and robustness. We present results of simulations of measurements of MCC patches with the RGBWir filter set done by varying the IR cutoff wavelength in Figure 9. The left panel shows the variation in average ΔE values of the color-transformed measurements for three different noise levels, 20 dB, 30 dB, and 40 dB. The right channel shows the variation in output SNR in the 50% gray patch for these simulations.

 [[Image:Ircutoff vs dE rgbwir 20 30 40.png left|320px|(a)]] Figure 9. The effect of changing the cutoff wavelength of the IR blocking filter. (a) The change in ΔE values as a function of IR cutoff wavelength for three different noise levels. (b) The change in output SNR in the MCC 50% gray patch as a function of IR cutoff wavelength for three different noise levels.

Figure 9 shows that a reasonable compromise between gain in sensitivity and loss in color correction accuracy may be reached by setting the IR cutoff wavelength at 800 nm. Note that this conclusion applies specifically to the color filter transmittance functions shown in Fig. 5(a). It is conceivable that an optimal set of 4 color filter transmittances may be found for better results. This is an area of future research.

When input SNR is low (20 dB), we see that the RGBWir filter set will not provide acceptable color accuracy (ΔE > 15 for IR cutoffs > 800 nm). Conventional color transformation methods cannot be reliably used to correct noisy RGBWir measurements and we need an improved method to convert RGBWir measurements to CIEXYZ values.

## Color transformation using signal and noise statistics

There is a need to understand how correlations between the energy in visible and NIR range can be used to improve color correction performance. The color correction method described in Section 2 implicitly uses NIR spectral statistics since the transform is derived from a training set that has information in both the visible and NIR ranges. But, a non-deterministic approach to color correction is called for in this particular application. The advantage of such an approach is that it will allow us to explicitly account for the presence of noise in the measurements.

The amplification of noise during color correction can be a serious problem (mp will put up some figures showing this..). If some information about the signal and noise is known (the first and second order moments), one can solve the problem of color-correction in the presence of noise is to use an affine minimum mean squared estimator (also commonly called the linear MMSE). In this method, we find the optimal estimate $\hat{x}$ of $x\,$ that can be expressed as

$\hat{x} = W y + b$

The optimal estimate is found by minimizing the expected estimation error

$E \{|| \hat{x} - {x} ||^2 \}$

The optimal estimate is obtained for

$W = A^TR_{xx} S (S^T R_{xx} S + R_{nn})^{-1}\,$ and

$b = E\{x\} - W(E\{y\} - E\{n\})\, ,$

where $R_{xx}\,$ and $R_{nn}\,$ are the signal and noise autocorrelation matrices. The signal autocorrelation matrix is found from the spectra in the training set. Figure 10 shows a visualization of $R_{xx}\,$ found from the training set described in Section 3 (Fig. 2).

We repeated our simulations for the sensor with RGBWir color filters with the no IR cutoff filter. Simulated measurements of the MCC patches were transformed with the conventional pseudoinverse-based color transformation method and the MMSE-based estimate. A comparison of the two methods is shown in Figure 11. The input SNR was calculated in the green channel of the MCC 50% gray patch and the output SNR was calculated in the luminance channel of the same patch. The MMSE-based method significantly outperforms the conventional method.

 [[Image:Snr vs delE pi am rgbir.png left|320px|(a)]] Figure 11. Comparisons of conventional color transformation (pseudoinverse) with MMSE-based color transformation. (a) Average ΔE over the MCC patches shown as a function of input SNR. (b) Output SNR shown as a function of input SNR.

 Table 2. Objective metric values for the simulation results shown in Fig. 12 Input SNR = 25 dB Output SNR CIELAB $\Delta$E Pseudoinverse 14.88 dB 13.89 MMSE 22.31 dB 7.38

 [[Image:Mcc rgbwir pi 25dB.png left|320px|(a)]] Figure 11. Comparisons of conventional color transformation (pseudoinverse) with MMSE-based color transformation. (a) Average ΔE over the MCC patches shown as a function of input SNR. (b) Output SNR shown as a function of input SNR.

Note that the MMSE-based method requires knowledge of the signal and noise correlation matrices. Although the signal correlation matrix can be obtained from the training set, the noise correlation matrix needs to be estimated. In our experiments we assumed knowledge of noise statistics. In practice, noise statistics will not be available and will need to be estimated from the measurements. This is an area of future research.

## Conclusions and future work

It is very important to design appropriate color transformation methods for sensors with color channels that admit NIR energy. In such filter sets conventional methods of color transformation do not perform well in terms of color correction accuracy. Moreover, conversion of measurements that contain NIR energy to a display space leads to amplification of any noise present in the measurements.

In this report we detailed the following issues for filter sets that admit NIR energy:

1. The effect of increasing levels of measurement noise (decreasing light levels) on the noise levels in the conventional color-transformed output.
2. The tradeoffs involved in color accuracy and noise amplification for color filter sets with different amounts of sensitivity in the NIR range.
3. A method for color conversion that accounts for noise and signal statistics for improved performance for NIR-sensitive color filter sets.

Our main conclusions are:

1. A sensor that has all four color channels sensitive in the visible and NIR range can give acceptable results when the measured signal has high SNR. But as input noise level increases, the noise-levels in the color-transformed signals become unacceptably high.
2. A sensor that has three color channels sensitive in the visible range (RGB) and a fourth wideband channel sensitive in the NIR range performs well in terms of accuracy and robustness to measurement noise.
3. A color transform derived as the minimum mean squared estimate of CIEXYZ values from color measurements can be used to improve the performance of the sensor with spectral sensitivities described in (1) by using the statistics of surface spectra and noise.

In the next stages of the project we propose to study the following problems:

1. Methods to analyze color correction in the context of interleaved imaging. Specifically, the wideband channel will be used to denoise the color channels before color correction.
2. Improving the color transformation of sensors with NIR-admitting color channels by using estimated noise statistics.
3. The effect of NIR and noise on color saturation.