Nighttime sensing

Patent metadata

Field	Value
Publication number	US-11600075-B2
Application number	US-202117201316-A
Country	US
Kind code	B2
Filing date	Mar 15, 2021
Priority date	Sep 28, 2017
Publication date	Mar 7, 2023
Grant date	Mar 7, 2023

Abstract

Systems and methods for night vision combining sensor image types. Some implementations may include obtaining a long wave infrared image from a long wave infrared sensor; detecting an object in the long wave infrared image; identifying a region of interest associated with the object; adjusting a control parameter of a near infrared sensor based on data associated with the region of interest; obtaining a near infrared image captured using the adjusted control parameter of the near infrared sensor; and determining a classification of the object based on data of the near infrared image associated with the region of interest.

First claim

What is claimed is:

1. A system, comprising:
a long wave infrared sensor;
a near infrared sensor;
a visible spectrum sensor; and
one or more processors coupled to the long wave infrared sensor, the near infrared sensor, and the visible spectrum sensor, configured to:
obtain a long wave infrared image using the long wave infrared sensor,
identify a region of interest associated with an object in the long wave infrared image,
obtain a near infrared image captured using the near infrared sensor,
obtain a visible spectrum image captured using the visible spectrum sensor,
extract features for the region of interest using the near infrared image and the visible spectrum image,
fuse the features for the region of interest from the near infrared sensor and the visible spectrum sensor, and
determine a classification of the object using the fused features for the region of interest.

2. The system of claim 1 , wherein the one or more processors are configured to:
resample channels of data from the visible spectrum image and the near infrared image at a common resolution.

3. The system of claim 1 , wherein determining the classification of the object comprises applying the fused features for the region of interest to a convolutional neural network.

4. The system of claim 1 , comprising:
a near infrared illuminator;
wherein the one or more processors are configured to adjust a near infrared illuminator control parameter based on data associated with the region of interest; and
wherein the near infrared image is captured using the adjusted near infrared illuminator control parameter.

5. The system of claim 4 , wherein the near infrared illuminator control parameter is a brightness.

6. The system of claim 4 , wherein the near infrared illuminator control parameter is field of illumination.

7. The system of claim 1 , comprising:
a visible spectrum illuminator;
wherein the one or more processors are configured to adjust a visible spectrum illuminator control parameter based on data associated with the region of interest; and
wherein the visible spectrum image is captured using the adjusted visible spectrum illuminator control parameter.

8. The system of claim 1 , wherein the one or more processors are configured to:
adjust a computational control parameter based on data associated with the region of interest; and
wherein the classification of the object is determined using the computational control parameter.

9. A method comprising:
obtaining a long wave infrared image using a long wave infrared sensor;
identifying a region of interest associated with an object in the long wave infrared image;
obtaining a near infrared image captured using a near infrared sensor;
obtaining a visible spectrum image captured using a visible spectrum sensor;
extracting features for the region of interest using the near infrared image and the visible spectrum image;
fusing the features for the region of interest from the near infrared sensor and the visible spectrum sensor; and
determining a classification of the object using the fused features for the region of interest.

10. The method of claim 9 , comprising:
resampling channels of data from the visible spectrum image and the near infrared image at a common resolution.

11. The method of claim 9 , wherein determining the classification of the object comprises applying the fused features for the region of interest to a convolutional neural network.

12. The method of claim 9 , comprising:
adjusting a visible spectrum illuminator control parameter based on data associated with the region of interest; and
wherein the visible spectrum image is captured using the adjusted visible spectrum illuminator control parameter.

13. The method of claim 9 , comprising:
adjusting a near infrared illuminator control parameter based on data associated with the region of interest; and
wherein the near infrared image is captured using the adjusted near infrared illuminator control parameter.

14. The method of claim 9 , wherein the near infrared image has a higher resolution than the long wave infrared image.

15. A vehicle comprising:
a vehicle body;
actuators operable to cause motion of the vehicle body;
a long wave infrared sensor;
a near infrared sensor;
a visible spectrum sensor; and
an automated controller configured to:
obtain a long wave infrared image using the long wave infrared sensor,
identify a region of interest associated with an object in the long wave infrared image,
obtain a near infrared image captured using the near infrared sensor,
obtain a visible spectrum image captured using the visible spectrum sensor,
extract features for the region of interest using the near infrared image and the visible spectrum image,
fuse the features for the region of interest from the near infrared sensor and the visible spectrum sensor, and
determine a classification of the object using the fused features for the region of interest.

16. The vehicle of claim 15 , wherein the automated controller is configured to:
resample channels of data from the visible spectrum image and the near infrared image at a common resolution.

17. The vehicle of claim 15 , wherein determining the classification of the object comprises applying the fused features for the region of interest to a convolutional neural network.

18. The vehicle of claim 15 , comprising:
a visible spectrum illuminator;
wherein the automated controller is configured to adjust a visible spectrum illuminator control parameter based on data associated with the region of interest; and
wherein the visible spectrum image is captured using the adjusted visible spectrum illuminator control parameter.

19. The vehicle of claim 15 , comprising:
a near infrared illuminator;
wherein the automated controller is configured to adjust a near infrared illuminator control parameter based on data associated with the region of interest; and
wherein the near infrared image is captured using the adjusted near infrared illuminator control parameter.

20. The vehicle of claim 15 , wherein the near infrared image has a higher resolution than the long wave infrared image.

Assignees

• Apple Inc

Inventors

• Ion Lucian
• Potter Daniel E
• Cull Evan C
• Tang Xiaofeng

Classifications

• G06V10/141

  Control of illumination · CPC title

• H04N23/61

  based on recognised objects · CPC title

• H04N23/74

  by influencing the scene brightness using illuminating means · CPC title

• G06V20/56 Primary

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title

• G06V10/255

  Detecting or recognising potential candidate objects based on visual cues, e.g. shapes · CPC title