Fast foveation camera and controlling algorithms

Patent metadata

Field	Value
Publication number	US-11800205-B2
Application number	US-202016844597-A
Country	US
Kind code	B2
Filing date	Apr 9, 2020
Priority date	Apr 18, 2019
Publication date	Oct 24, 2023
Grant date	Oct 24, 2023

Abstract

Provided is a camera system to obtain high resolution imagery on multiple regions of interest even when the regions of interest are ate different focal depths and in different viewing directions. Embodiments include a high speed camera that operates by reflecting a beam of interest corresponding to a scene of interest into a high-speed passive sensor from a dynamic optical modulator. Embodiments described herein provide a foveating camera design that distributes resolution onto regions of interest by imaging reflections off a scanning micro-electromechanical system (MEMS) mirror. MEMS mirrors are used herein to modulate viewing direction. Embodiments include a camera capturing reflections off of a tiny, fast moving mirror.

First claim

The invention claimed is:

1. A camera comprising:
a passive sensor;
a dynamic optical modulator;
an optical modulator controller configured to control an orientation of the dynamic optical modulator; and
a processor configured to receive digital image data from the passive sensor and to control the optical modulator controller,
wherein the dynamic optical modulator is configured to reflect a beam of light from a scene of interest onto the passive sensor,
wherein the processor is configured to control the optical modulator controller in accordance with a control algorithm, and
wherein the control algorithm provides a mirror speed for movement of the dynamic optical modulator and a frame-rate speed for capturing digital image data by the passive sensor, wherein the mirror speed and the frame-rate speed are calibrated by:
(a) an image is captured of a calibration scene using the camera;
(b) analysis of the image is performed to determine if a blur is present in the image;
(c) (i) reduction of a frame rate of the camera and returning to step (a) in response to a determination that no blur is present in the image;
(ii) reduce a mirror speed of the dynamic optical modulator and returning to step (a) in response to a determination that a blur above a predetermined size is present in the image; and
(iii) determine that the control algorithm is calibrated in response to a determination that a blur below the predetermined size is present in the image.

2. The camera of claim 1 , wherein the passive sensor is a CCD or CMOS image sensor.

3. The camera of claim 1 , wherein the dynamic optical modulator is a scanning mirror or an array of scanning mirrors.

4. The camera of claim 1 , wherein the dynamic optical modulator is a MEMs mirror or an array of MEMs mirrors.

5. The camera of claim 4 , wherein each MEMs mirror comprises a plurality of microactuators.

6. The camera of claim 5 , wherein a central reflective mirror plate of each MEMs mirror is suspended by four microactuators.

7. The camera of claim 6 , wherein each of the microactuators are configured to be electrically controlled to set a height of a corresponding edge of the central reflective mirror plate, the microactuator being anchored to the corresponding edge.

8. The camera of claim 5 , wherein each of the microactuators is made of a folded thin-film beam.

9. The camera of claim 5 , wherein the optical modulator controller is configured to control each of the microactuators.

10. The camera of claim 1 , further comprising a cover glass positioned between the dynamic optical modulator and the passive sensor, wherein the passive sensor and the dynamic optical modulator are positioned at angles relative to the cover glass established to dampen reflections from the cover glass.

11. A composite camera comprising:
a secondary camera,
a primary camera, wherein the primary camera is the camera of claim 1 .

12. The composite camera of claim 11 , wherein the secondary camera is one of an optical camera, a thermal camera, or an ultra-violet camera.

13. The composite camera of claim 11 , wherein a field of view of the secondary camera and a field of view of the primary camera overlap at least in part.

14. The composite camera of claim 11 , wherein the secondary camera is configured to capture a first image of a scene and the primary camera is configured to capture one or more second images, each of the second images being of an area of interest within the scene.

15. The composite camera of claim 14 , wherein each second image has a higher resolution than the first image.

16. The composite camera of claim 14 , configured to associate with one or more second images with the first image and provide and/or store the first and second images.

17. The composite camera of claim 14 , wherein a field of view of the secondary camera and a field of view of the primary camera only partially overlap.

18. A method of imaging a scene with a composite camera, the composite camera being the composite camera of claim 11 , the method comprising:
capturing a first image of a scene with the secondary camera;
analyzing the first image with a processor to identify one or more areas of interest within the scene;
causing the optical modulator to control the dynamic optical modulator such that the primary camera captures one or more second images, each second image being an image of one of the one or more areas of interest within the scene;
associating the first and second images; and
storing and/or providing the first and second images.

19. The method of claim 18 , wherein each second image has a higher resolution than the associated first image.

20. The method of claim 18 , further comprising performing post-processing on at least one second image to remove imaging artifacts from the at least one second image.

21. A method for calibrating a control algorithm for a mirror speed relative to a frame rate, the method comprising:
(a) capturing an image of a calibration scene using a camera;
(b) analyzing the calibration scene to determine if a blur is present in the image;
(c) (i) responsive to determining that no blur is present in the image, reducing a frame rate of the camera and returning to step (a);
(ii) responsive to determining that a blur above a predetermined size is present in the image, reducing a mirror speed of a dynamic optical modulator and returning to step (a); and
(iii) responsive to determining that a blur below the predetermined size is present in the image, determining that the control algorithm is calibrated.

22. The method of claim 21 , wherein the calibration scene is a textured scene.

23. The method of claim 22 , wherein the calibration scene is a texture plane.

24. The method of claim 22 , wherein the textured scene is textured with at least one of a tactile texture or a visual texture.

Assignees

• Univ Florida

Inventors

• Koppal Sanjeev Jagannatha
• Tasneem Zaid
• Wang Dingkang
• Xie Huikai
• Tilmon Brevin Jeffery

Classifications

• H04N23/45 Primary

  for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images · CPC title

• G02B26/0841

  the reflecting element being moved or deformed by electrostatic means · CPC title

• G03B39/04

  using moving plate or film (G03B39/06 takes precedence) · CPC title

• G06T5/003

  Physics · mapped topic

• G06T7/80

  Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration · CPC title