Nighttime sensing

Patent metadata

Field	Value
Publication number	US-10949679-B2
Application number	US-201816126509-A
Country	US
Kind code	B2
Filing date	Sep 10, 2018
Priority date	Sep 28, 2017
Publication date	Mar 16, 2021
Grant date	Mar 16, 2021

Abstract

Systems and methods for night vision combining sensor image types. Some implementations may include obtaining a long wave infrared image from a long wave infrared sensor; detecting an object in the long wave infrared image; identifying a region of interest associated with the object; adjusting a control parameter of a near infrared sensor based on data associated with the region of interest; obtaining a near infrared image captured using the adjusted control parameter of the near infrared sensor; and determining a classification of the object based on data of the near infrared image associated with the region of interest.

First claim

What is claimed is:

1. A system, comprising:
a long wave infrared sensor;
a near infrared sensor;
a visible spectrum sensor;
a data processing apparatus; and
a data storage device storing instructions executable by the data processing apparatus that upon execution by the data processing apparatus cause the data processing apparatus to perform operations comprising:
obtaining a long wave infrared image from the long wave infrared sensor,
detecting an object in the long wave infrared image,
identifying a region of interest associated with the object,
adjusting a control parameter of the near infrared sensor based on data associated with the region of interest,
adjusting a control parameter of the visible spectrum sensor based on data associated with the region of interest,
obtaining a near infrared image captured using the adjusted control parameter of the near infrared sensor,
obtaining a visible spectrum image from the visible spectrum sensor that is captured using the adjusted control parameter of the visible spectrum sensor, and
determining a classification of the object based on data of the near infrared image associated with the region of interest and based on data of the visible spectrum image associated with the region of interest.

2. The system of claim 1 , further comprising:
a near infrared illuminator;
wherein the operations comprise adjusting a near infrared illuminator control parameter based on data associated with the region of interest; and
wherein the near infrared image is captured using the adjusted near infrared illuminator control parameter.

3. The system of claim 2 , wherein the near infrared illuminator control parameter is a brightness.

4. The system of claim 2 , wherein the near infrared illuminator control parameter is field of illumination.

5. The system of claim 1 , wherein the control parameter of the near infrared sensor is an integration time.

6. The system of claim 1 , wherein the control parameter of the near infrared sensor is an aperture size.

7. The system of claim 1 , wherein the control parameter of the near infrared sensor is a filter selection.

8. The system of claim 1 , wherein the control parameter of the near infrared sensor is an amplification gain.

9. The system of claim 1 , further comprising:
a visible spectrum illuminator;
wherein the operations comprise adjusting a visible spectrum illuminator control parameter based on data associated with the region of interest; and
wherein the visible spectrum image is captured using the adjusted visible spectrum illuminator control parameter.

10. The system of claim 1 , wherein the operations further comprise:
adjusting a computational control parameter based on data associated with the region of interest; and
wherein the classification of the object is determined using the computational control parameter.

11. A method, comprising:
obtaining a long wave infrared image from a long wave infrared sensor;
detecting an object in the long wave infrared image;
identifying a region of interest associated with the object;
adjusting a control parameter of a near infrared sensor based on data associated with the region of interest;
adjusting a visible spectrum sensor control parameter based on data associated with the region of interest;
obtaining a near infrared image captured using the adjusted control parameter of the near infrared sensor;
obtaining a visible spectrum image from the visible spectrum sensor that is captured using the adjusted control parameter of the visible spectrum sensor; and
determining a classification of the object based on data of the near infrared image associated with the region of interest and based on data of the visible spectrum image associated with the region of interest.

12. The method of claim 11 , further comprising:
adjusting a visible spectrum illuminator control parameter based on data associated with the region of interest; and
wherein the visible spectrum image is captured using the adjusted visible spectrum illuminator control parameter.

13. The method of claim 11 , further comprising:
adjusting a near infrared illuminator control parameter based on data associated with the region of interest; and
wherein the near infrared image is captured using the adjusted near infrared illuminator control parameter.

14. The method of claim 11 , wherein the near infrared image has a higher resolution than the long wave infrared image.

15. The method of claim 11 , further comprising:
adjusting a computational control parameter based on data associated with the region of interest; and
wherein the classification of the object is determined using the computational control parameter.

16. A vehicle, comprising:
a vehicle body;
actuators operable to cause motion of the vehicle body;
a long wave infrared sensor;
a near infrared sensor;
a visible spectrum sensor; and
an automated controller configured to:
obtain a long wave infrared image from the long wave infrared sensor,
detect an object in the long wave infrared image,
identify a region of interest associated with the object,
adjust a control parameter of the near infrared sensor based on data associated with the region of interest,
adjust a control parameter of the visible spectrum sensor based on data associated with the region of interest,
obtain a near infrared image captured using the adjusted control parameter of the near infrared sensor,
obtain a visible spectrum image from the visible spectrum sensor that is captured using the adjusted control parameter of the visible spectrum sensor,
determine a classification of the object based on data of the near infrared image associated with the region of interest and based on data of the visible spectrum image associated with the region of interest,
determine a motion plan based on the classification of the object, and
output commands to the actuators to maneuver the vehicle.

17. The vehicle of claim 16 , further comprising:
a visible spectrum illuminator;
wherein the automated controller is configured to adjust a visible spectrum illuminator control parameter based on data associated with the region of interest; and
wherein the visible spectrum image is captured using the adjusted visible spectrum illuminator control parameter.

18. The vehicle of claim 16 , further comprising:
a near infrared illuminator;
wherein the automated controller is configured to adjust a near infrared illuminator control parameter based on data associated with the region of interest; and
wherein the near infrared image is captured using the adjusted near infrared illuminator control parameter.

19. The vehicle of claim 16 , wherein the near infrared image has a higher resolution than the long wave infrared image.

20. The vehicle of claim 16 , wherein the automated controller is further configured to:
adjust a computational control parameter based on data associated with the region of interest; and
wherein the classification of the object is determined using the computational control parameter.

Assignees

• Apple Inc

Inventors

• Ion Lucian
• Potter Daniel E
• Cull Evan C
• Tang Xiaofeng

Classifications

• G06V20/56

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title

• G06V10/143 Primary

  Sensing or illuminating at different wavelengths · CPC title

• H04N23/61

  based on recognised objects · CPC title

• H04N23/45

  for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images · CPC title

• H04N23/74

  by influencing the scene brightness using illuminating means · CPC title