Camera system, method and instructions using images captured by a first mage sensor and a second image sensor to generate a third image corresponding to a simulated lens having an intermediate focal length

Patent metadata

Field	Value
Publication number	US-11917281-B2
Application number	US-202117517882-A
Country	US
Kind code	B2
Filing date	Nov 3, 2021
Priority date	Dec 28, 2017
Publication date	Feb 27, 2024
Grant date	Feb 27, 2024

Abstract

Example embodiments relate to multiple operating modes to expand dynamic range. An example embodiment includes a camera system. The camera system may include a first image sensor having a first dynamic range corresponding to a first range of luminance levels in a scene. The system may also include a second image sensor having a second dynamic range corresponding to a second range of luminance levels in the scene. The camera system may further include a processor coupled to the first image sensor and the second image sensor. The processor may be configured to execute instructions to identify objects of a first type in a first image of the scene captured by the first image sensor and identify objects of a second object type in a second image of the scene captured by the second image sensor.

First claim

What is claimed is:

1. A camera system, comprising:
a first image sensor having a first associated focal length;
a second image sensor having a second associated focal length, wherein the second focal length is greater than the first focal length; and
a processor coupled to the first image sensor and the second image sensor, wherein the processor is configured to execute instructions to generate, using a first image of a scene captured by the first image sensor and a second image of the scene captured by the second image sensor, a third image corresponding to a simulated lens having an intermediate focal length, wherein the intermediate focal length is between the first focal length and the second focal length, and
wherein:
(i) the camera system further comprises a shared astigmatic lens configured to provide light from the scene to the first image sensor and the second image sensor at different focal lengths; or
(ii) the camera system further comprises:
a first lens configured to provide light from the scene to the first image sensor; and
a second lens configured to provide light from the scene to the second image sensor, wherein:
the first lens is rotatable relative to the first image sensor;
the second lens is rotatable relative to the second image sensor;
the first image sensor is configured to capture additional focal lengths based on the first lens being rotatable relative to the first image sensor; and
the second image sensor is configured to capture additional focal lengths based on the second lens being rotatable relative to the second image sensor; or

(iii) the first image sensor comprises a first aperture having a first size;
the second image sensor comprises a second aperture having a second size; and
the processor is further configured to execute instructions to:
adjust the first aperture to cause the first image sensor to capture different depths of field;
adjust the second aperture to cause the second image sensor to capture different depths of field; and
generate additional simulated images of the scene, each having a different respective depth of field; or

(iv) generating the third image using the first and second images comprises:
sharpening at least one region of the first image or at least one region of the second image; and
generating the third image using the sharpened first image or the sharpened second image; or

(v) generating the third image using the first and second images comprises:
blurring at least one region of the first image or at least one region of the second image; and
generating the third image using the blurred first image or the blurred second image; or

(vi) the first image sensor has a variable exposure that is adjusted by a controller executing instructions based on illuminance; and
the second image sensor has a fixed exposure; or
(vii) the second image sensor receives light from the scene via a neutral-density filter; or
(viii) the first image sensor is optically coupled to a first astigmatic lens configured to provide light from the scene to the first image sensor; and
the second image sensor is optically coupled to a second astigmatic lens configured to provide light from the scene to the second image sensor.

2. The camera system of claim 1 , further comprising:
the shared astigmatic lens configured to provide light from the scene to the first image sensor and the second image sensor at different focal lengths.

3. The camera system of claim 1 , further comprising:
the first lens configured to provide light from the scene to the first image sensor; and
the second lens configured to provide light from the scene to the second image sensor.

4. The camera system of claim 3 , wherein:
the first lens is rotatable relative to the first image sensor;
the second lens is rotatable relative to the second image sensor;
the first image sensor is configured to capture additional focal lengths based on the first lens being rotatable relative to the first image sensor; and
the second image sensor is configured to capture additional focal lengths based on the second lens being rotatable relative to the second image sensor.

5. The camera system of claim 1 , wherein:
the first image sensor has a first dynamic range corresponding to a first range of luminance levels in the scene;
the second image sensor has a second dynamic range corresponding to a second range of luminance levels in the scene; and
the second range of luminance levels includes luminance levels that are higher than the first range of luminance levels.

6. The camera system of claim 1 , wherein:
the first image sensor comprises the first aperture having the first size;
the second image sensor comprises the second aperture having the second size; and
the processor is further configured to execute instructions to:
adjust the first aperture to cause the first image sensor to capture different depths of field;
adjust the second aperture to cause the second image sensor to capture different depths of field; and
generate additional simulated images of the scene, each having a different respective depth of field.

7. The camera system of claim 1 , wherein generating the third image using the first and second images comprises:
sharpening at least one region of the first image or at least one region of the second image; and
generating the third image using the sharpened first image or the sharpened second image.

8. The camera system of claim 1 , wherein generating the third image using the first and second images comprises:
blurring at least one region of the first image or at least one region of the second image; and
generating the third image using the blurred first image or the blurred second image.

9. The camera system of claim 1 , wherein generating the third image using the first and second images comprises:
sharpening a first region of the first image;
blurring a second region of the first image; and
generating the third image using a first composite image based on:
an unsharpened and unblurred version of the first image;
the sharpened first region of the first image; and
the blurred first region of the first image.

10. The camera system of claim 1 , wherein generating the third image using the first and second images comprises:
sharpening a first region of the second image;
blurring a second region of the second image; and
generating the third image using a second composite image based on:
an unsharpened and unblurred version of the second image;
the sharpened first region of the second image; and
the blurred first region of the second image.

11. The camera system of claim 1 , wherein:
the first image sensor has the variable exposure that is adjusted by the controller executing instructions based on illuminance; and
the second image sensor has the fixed exposure.

12. The camera system of claim 1 , wherein the second image sensor receives light from the scene via the neutral-density filter.

13. The camera system of claim 1 , wherein the camera system is mounted on an autonomous vehicle and used for object detection and avoidance.

14. The camera system of claim 1 , wherein the processor is further configured to execute instructions to:
identify an object in the scene based on the third image.

15. A method comprising:
capturing, by a first image sensor having a first associated focal length, a first image of a scene;
capturing, by a second image sensor having a second associated focal length, a second image of the scene, wherein the second focal length is greater than the first focal length; and
using, by a process

Assignees

• Waymo Llc

Inventors

• Wendel Andreas
• Dittmer Jeremy
• Hermalyn Brendan
• Ingram Benjamin

Classifications

• H04N25/57

  Control of the dynamic range · CPC title

• H04N23/61 Primary

  based on recognised objects · CPC title

• H04N23/45 Primary

  for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images · CPC title

• H04N23/73

  by influencing the exposure time · CPC title

• H04N23/741

  by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors · CPC title