Method and Apparatus for Processing RGB-Infrared (RGB-IR) Sensor Data

Patent metadata

Field	Value
Publication number	US-2024054601-A1
Application number	US-202318383576-A
Country	US
Kind code	A1
Filing date	Oct 25, 2023
Priority date	Dec 21, 2020
Publication date	Feb 15, 2024
Grant date	—

Abstract

A method for processing RGB-Infrared (RGB-IR) sensor data is provided that includes receiving a raw RGB-IR image, determining whether to process the raw RGB-IR image in day mode or night mode, generating, when day mode is determined, an infrared (IR) subtracted raw Bayer image from the raw RGB-IR image and processing the IR subtracted raw Bayer image in an image signal processor (ISP), and generating, when night mode is determined, an IR image from the raw RGB-IR image.

First claim

What is claimed is:

1 . A color filter array (CFA) interpolation component, comprising:
a storage component configured to store a first set of pixel values of an image captured by an RGB-IR (Red, Green, Blue, and Infrared) sensor; a filter component configured to interpolate the first set of pixel values to generate first and second sets of outputs representing pixel values corresponding to RGB pixels, and interpolate the first set of pixel values to generate a third set of outputs representing pixel values corresponding to IR pixels; an output select component configured to combine the first and second sets of outputs to generate a second set of pixel values; and a subtract component configured to generate a first image based on the second set of pixel values and the third set of outputs.

2 . The CFA interpolation component of claim 1 , wherein:
the first image is generated based on a determination to process the first set of pixel values in a first mode, the CFA interpolation component further comprises a gain component coupled between the storage component and the filter component, the gain component is configured to apply gain values to the first set of pixel values to generate a fourth set of outputs based on a determination to process the first set of pixel values in a second mode, and the filter component is configured to generate a second image based on the fourth set of outputs based on the determination to process the first set of pixel values in the second mode.

3 . The CFA interpolation component of claim 1 , wherein to combine the first and second sets of outputs, the output select component is configured to combine the first and second sets of outputs based on an order associated with a Bayer array.

4 . The CFA interpolation component of claim 1 , wherein the subtract component is configured to generate the first image based on subtraction between the second set of pixel values and the third set of outputs.

5 . The CFA interpolation component of claim 4 , wherein the subtract component is configured to determine whether to perform the subtraction on a pixel value of the second set of pixel values based on comparison of the pixel value to a first threshold.

6 . The CFA interpolation component of claim 5 , wherein the subtract component is configured to skip performance of the subtraction on the pixel value and other pixel values in a 2×2 Bayer array comprising the pixel value, based on a determination that the pixel value exceeds the first threshold.

7 . The CFA interpolation component of claim 4 , wherein the subtract component is configured to:
determine that an output of the third set of outputs corresponding to a pixel value of the second set of pixel values exceeds a second threshold; and in response to the determination,
clip the output to the second threshold to generate a clipped output; and
perform the subtraction on the pixel value corresponding to the output using the clipped output.

8 . The CFA interpolation component of claim 7 , wherein the second threshold is determined as a percentage of the pixel value of the second set of pixel values.

9 . The CFA interpolation component of claim 4 , wherein the subtract component is configured to perform the subtraction based on weights associated with the second set of pixel values and the third set of outputs.

10 . The CFA interpolation component of claim 1 , wherein:
the filter component is configured to interpolate the first set of pixel values to generate the first and second sets of outputs using respectively a first filter and a second filter, and interpolate the first set of pixel values to generate the third set of outputs using a third filter, and the CFA interpolation component further comprises a coefficient table component configured to store coefficients for the first, second, and third filters.

11 . A system, comprising:
a memory storing program instructions; a processor coupled to the memory and comprising a color filter array (CFA) interpolation component, wherein the processor is configured to execute the program instructions to:
receive a first set of pixel values of an image captured by an image sensor;
determine to process the first set of pixel values in a first mode; and
cause the CFA interpolation component to:
in response to the determination,
interpolate the first set of pixel values to generate first and second sets of outputs representing pixel values corresponding to RGB pixels;
interpolate the first set of pixel values to generate a third set of outputs representing pixel values corresponding to IR pixels;
combine the first and second sets of outputs to generate a second set of pixel values; and
generate a first image based on the second set of pixel values and the third set of outputs.

12 . The system of claim 11 , wherein to generate the first image, the processor is configured to execute the program instructions to cause the CFA interpolation component to:
generate the first image based on subtraction between the second set of pixel values and the third set of outputs.

13 . The system of claim 12 , wherein the processor is configured to execute the program instructions to cause the CFA interpolation component to:
determine whether to perform the subtraction on a pixel value of the second set of pixel values based on comparison of the pixel value to a first threshold.

14 . The system of claim 12 , wherein the processor is configured to execute the program instructions to cause the CFA interpolation component to:
determine that an output of the third set of outputs corresponding to a pixel value of the second set of pixel values exceeds a second threshold; and in response to the determination,
clip the output to the second threshold to generate a clipped output; and
perform the subtraction on the pixel value corresponding to the output using the clipped output.

15 . The system of claim 12 , wherein the processor is configured to execute the program instructions to cause the CFA interpolation component to:
perform the subtraction based on weights associated with the second set of pixel values and the third set of outputs.

16 . The system of claim 11 , wherein the processor is configured to execute the program instructions to:
determine to process the first set of pixel values in a second mode; and cause the CFA interpolation component to:
in response to the determination,
apply gain values to the first set of pixel values to generate a fourth set of outputs; and
generate a second image based on the fourth set of outputs.

17 . The system of claim 16 , wherein the processor is configured to execute the program instructions to:
determine a first histogram corresponding to a non-infrared color based on the first set of pixel values; determine a second histogram corresponding to infrared based on the first set of pixel values; sum bins of the first histogram to generate a first sum until a number of pixels of the first histogram included in the first sum reaches a first value; sum bins of the second histogram to generate a second sum until a number of pixels of the second histogram included in the second sum reaches a second value; and determine whether to process the first set of pixel values in the first mode or the second based, based on a difference between a number of the bins included in the first sum and a number of bins included in the second sum.

18 . The system of claim 17 , wherein the first mode corresponds to daytime, and the second mode corresponds

Assignees

• Texas Instruments Inc

Inventors

• Dabral Shashank
• Hua Gang
• Mody Mihir Narendra

Classifications

• H04N23/11

  for generating image signals from visible and infrared light wavelengths · CPC title

• H10F39/8053 Primary

  Colour filters · CPC title

• H10F39/184

  Infrared image sensors · CPC title

• G06T3/4015 Primary

  Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns · CPC title

• H04N5/33

  Transforming infrared radiation (cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20) · CPC title