Method and apparatus for processing RGB-infrared (RGB-IR) sensor data

Patent metadata

Field	Value
Publication number	US-12493925-B2
Application number	US-202318383576-A
Country	US
Kind code	B2
Filing date	Oct 25, 2023
Priority date	Dec 21, 2020
Publication date	Dec 9, 2025
Grant date	Dec 9, 2025

Abstract

A method for processing RGB-Infrared (RGB-IR) sensor data is provided that includes receiving a raw RGB-IR image, determining whether to process the raw RGB-IR image in day mode or night mode, generating, when day mode is determined, an infrared (IR) subtracted raw Bayer image from the raw RGB-IR image and processing the IR subtracted raw Bayer image in an image signal processor (ISP), and generating, when night mode is determined, an IR image from the raw RGB-IR image.

First claim

What is claimed is:

1 . A color filter array (CFA) interpolation component, comprising:
a storage component configured to store a first set of pixel values of an image captured by an RGB-IR (Red, Green, Blue, and Infrared) sensor; a filter component configured to interpolate the first set of pixel values to generate a first set of outputs representing pixel values corresponding to RGB pixels, and interpolate the first set of pixel values to generate a third set of outputs representing pixel values corresponding to IR pixels; an output select component configured to generate a second set of pixel values based on the first set of outputs; and a subtract component configured to generate a first image by subtracting the third set of outputs from the second set of pixel values, wherein the subtract component is configured to determine whether to perform the subtraction for a pixel value of the second set of pixel values based on comparison of the pixel value with a first threshold.

2 . The CFA interpolation component of claim 1 , wherein:
the first image is generated based on a determination to process the first set of pixel values in a first mode, the CFA interpolation component further comprises a gain component coupled between the storage component and the filter component, the gain component is configured to apply gain values to the first set of pixel values to generate a fourth set of outputs based on a determination to process the first set of pixel values in a second mode, and the filter component is configured to generate a second image based on the fourth set of outputs based on the determination to process the first set of pixel values in the second mode.

3 . The CFA interpolation component of claim 1 , wherein the filter component is configured to interpolate the first set of pixel values to generate a second set of outputs representing pixel values corresponding to the RGB pixels.

4 . The CFA interpolation component of claim 3 , wherein the output select component is configured to generate the second set of pixel values by combining the first and second sets of outputs.

5 . The CFA interpolation component of claim 4 , wherein to combine the first and second sets of outputs, the output select component is configured to combine the first and second sets of outputs based on an order associated with a Bayer array.

6 . The CFA interpolation component of claim 1 , wherein the subtract component is configured to skip performance of the subtraction on the pixel value and other pixel values in a 2×2 Bayer array comprising the pixel value, based on a determination that the pixel value exceeds the first threshold.

7 . The CFA interpolation component of claim 1 , wherein the subtract component is configured to:
determine that an output of the third set of outputs corresponding to a pixel value of the second set of pixel values exceeds a second threshold; and in response to the determination,
clip the output to the second threshold to generate a clipped output; and
perform the subtraction on the pixel value corresponding to the output using the clipped output.

8 . The CFA interpolation component of claim 7 , wherein the second threshold is determined as a percentage of the pixel value of the second set of pixel values.

9 . The CFA interpolation component of claim 1 , wherein the subtract component is configured to perform the subtraction based on weights associated with the second set of pixel values and the third set of outputs.

10 . The CFA interpolation component of claim 1 , wherein:
the filter component is configured to interpolate the first set of pixel values to generate the first set of outputs using a first filter and interpolate the first set of pixel values to generate the third set of outputs using a third filter, and the CFA interpolation component further comprises a coefficient table component configured to store coefficients for the first and third filters.

11 . A system, comprising:
a memory storing program instructions; a processor coupled to the memory and comprising a color filter array (CFA) interpolation component, wherein the processor is configured to execute the program instructions to:
receive a first set of pixel values of an image captured by an image sensor;
determine to process the first set of pixel values in a first mode; and
cause the CFA interpolation component to:
in response to the determination,
interpolate the first set of pixel values to generate a first set of outputs representing pixel values corresponding to RGB pixels;
interpolate the first set of pixel values to generate a third set of outputs representing pixel values corresponding to IR pixels;
generate a second set of pixel values based on the first set of outputs; and
generate a first image by subtracting the third set of outputs from the second set of pixel values, wherein the processor is configured to execute the program instructions to cause the CFA to determine whether to perform the subtraction for a pixel value of the second set of pixel values based on comparison of the pixel value with a first threshold.

12 . The system of claim 11 , wherein the processor is configured to execute the program instructions to cause the CFA interpolation component to:
interpolate the first set of pixel values to generate a second set of outputs representing pixel values corresponding to the RGB pixels.

13 . The system of claim 12 , wherein the processor is configured to execute the program instructions to cause the CFA interpolation component to:
generate the second set of pixel values by combining the first and second sets of outputs based on an order associated with a Bayer array.

14 . The system of claim 11 , wherein the processor is configured to execute the program instructions to cause the CFA interpolation component to:
determine that an output of the third set of outputs corresponding to a pixel value of the second set of pixel values exceeds a second threshold; and in response to the determination,
clip the output to the second threshold to generate a clipped output; and
perform the subtraction on the pixel value corresponding to the output using the clipped output.

15 . The system of claim 11 , wherein the processor is configured to execute the program instructions to cause the CFA interpolation component to:
perform the subtraction based on weights associated with the second set of pixel values and the third set of outputs.

16 . The system of claim 11 , wherein the processor is configured to execute the program instructions to:
determine to process the first set of pixel values in a second mode; and cause the CFA interpolation component to:
in response to the determination,
apply gain values to the first set of pixel values to generate a fourth set of outputs; and
generate a second image based on the fourth set of outputs.

17 . The system of claim 16 , wherein the processor is configured to execute the program instructions to:
determine a first histogram corresponding to a non-infrared color based on the first set of pixel values; determine a second histogram corresponding to infrared based on the first set of pixel values; sum bins of the first histogram to generate a first sum until a number of pixels of the first histogram included in the first sum reaches a first value; sum bins of the second histogram to generate a second sum until a number of pixels of the second histogram included in the second sum reaches a second value; and determine whether to process the first set of pixel v

Assignees

• Texas Instruments Inc

Inventors

• Dabral Shashank
• Hua Gang
• Mody Mihir Narendra

Classifications

• H04N23/11

  for generating image signals from visible and infrared light wavelengths · CPC title

• H10F39/8053 Primary

  Colour filters · CPC title

• H10F39/184

  Infrared image sensors · CPC title

• H04N25/131

  including elements passing infrared wavelengths · CPC title

• H04N25/134

  based on three different wavelength filter elements · CPC title