Sensor misalignment compensation

Patent metadata

Field	Value
Publication number	US-11575874-B2
Application number	US-202117553287-A
Country	US
Kind code	B2
Filing date	Dec 16, 2021
Priority date	Dec 6, 2019
Publication date	Feb 7, 2023
Grant date	Feb 7, 2023

Abstract

Camera compensation methods and systems that compensate for misalignment of sensors/camera in stereoscopic camera systems. The compensation includes identifying a pitch angle offset between a first camera and a second camera, determining misalignment of the first and second cameras from the identified pitch angle offset, determining a relative compensation delay responsive to the determined misalignment, introducing the relative compensation delay to image streams produced by the cameras, and producing a stereoscopic image on a display from the first and second image streams with the introduced delay.

First claim

What is claimed is:

1. A camera compensation system comprising:
an eyewear device including:
a frame; and
a stereoscopic camera including a first camera and a second camera supported by the frame, the first camera producing a first image stream and the second camera producing a second image stream, and the first and second cameras having an overlapping field of view and a pitch angle offset, whereby feature points obtained by the first and second cameras in the first and second image streams have an exposure delay, the exposure delay counted in sensor lines whereby the feature points fall on different sensor lines at the same time;
an image display for presenting a stereoscopic image produced from the first and second image streams;
an image display driver coupled to the image display to control the image display to present the stereoscopic image responsive to the pitch angle offset;
a memory;
a processor coupled to the stereoscopic camera, the image display driver, and the memory; and
programming in the memory, wherein execution of the programming by the processor configures the camera compensation system to perform functions, including functions to:
identify the pitch angle offset;
determine misalignment of the first and second cameras from the identified pitch angle offset;
determine a relative compensation delay responsive to the determined misalignment;
introduce the relative compensation delay to at least one of the first and second image streams by adjusting the exposure delay of the at least one of the first and second cameras prior to capturing the first and second images; and
produce a stereoscopic image on the display from the first and second image streams with the introduced relative compensation delay to compensate for the pitch angle offset.

2. The system of claim 1 , wherein execution of the programming by the processor determines the relative compensation delay by performing functions to:
synchronize readouts of the first and second cameras;
determine an offset time corresponding to the determined misalignment; and
adjust the relative compensation delay by the determined offset time.

3. The system of claim 1 , wherein execution of the programming by the processor identifies the pitch angle offset by performing functions to:
determine a first pitch angle of the first camera;
determine a second pitch angle of the second camera; and
determine the pitch angle offset from a difference between the first pitch angle and the second pitch angle.

4. The system of claim 1 , wherein execution of the programming by the processor identifies the pitch angle offset by performing functions to:
determine flexure of the frame; and
determine the pitch angle offset from the flexure of the frame.

5. The system of claim 1 , wherein the pitch angle offset is stored in the memory and wherein execution of the programming by the processor identifies the pitch angle offset by performing functions to:
retrieve the pitch angle offset from the memory; and
determine the pitch angle offset from the retrieved pitch angle offset.

6. The system of claim 1 , wherein execution of the programming by the processor determines the misalignment of the first and second cameras by performing functions to:
determine a number of offset sensor lines between a first plurality of sensor lines and a second plurality of sensor lines corresponding to the pitch angle offset.

7. The system of claim 6 , wherein execution of the programming by the processor determines the relative compensation delay by performing functions to:
determine an offset time corresponding to the number of offset lines; and
adjust the relative compensation delay by the determined offset time.

8. The system of claim 7 , wherein execution of the programming by the processor determines the relative compensation delay by further performing functions to:
synchronize readouts of the first and second cameras.

9. The system of claim 1 , wherein execution of the programming by the processor determines the relative compensation delay by performing functions to:
introduce delay after capturing the first and second images by changing a vertical blanking interval of at least one of the first and second cameras in order to minimize compensation delay between feature points of the first and second image streams from the first and second cameras.

10. The system of claim 1 , wherein execution of the programming by the processor determines the relative compensation delay by performing functions to:
introduce delay prior to capturing the first and second images by configuring an exposure delay of at least one of the first and second cameras such that feature points are exposed at substantially the same time.

11. A camera compensation method comprising:
identifying a pitch angle offset between a first camera and a second camera of a stereoscopic camera system supported by a frame of an eyewear device, the first camera producing a first image stream and the second camera producing a second image stream, and the first and second cameras having an overlapping field of view and a pitch angle offset, whereby feature points obtained by the first and second cameras in the first and second image streams have an exposure delay, the exposure delay counted in sensor lines whereby the feature points fall on different sensor lines at the same time;
determining misalignment of the first and second cameras from the identified pitch angle offset;
determining a relative compensation delay responsive to the determined misalignment;
introducing the relative compensation delay to at least one of the first and second image streams by adjusting the exposure delay of the at least one of the first and second cameras prior to capturing first and second image streams; and
producing a stereoscopic image on a display from the first and second image streams with the introduced relative compensation delay to compensate for the pitch angle offset.

12. The method of claim 11 , wherein determining the relative compensation delay comprises:
determining an offset time corresponding to the determined misalignment; and
adjusting the relative compensation delay by the determined offset time.

13. The method of claim 12 , wherein determining the relative compensation delay further comprises:
synchronizing readouts of the first and second cameras.

14. The method of claim 11 , wherein determining the relative compensation delay comprises:
introducing delay after capturing the first and second images by changing a vertical blanking interval of at least one of the first and second cameras in order to minimize compensation delay between feature points of the first and second image streams from the first and second cameras.

15. The method of claim 11 , wherein determining the relative compensation delay comprises:
introducing delay prior to capturing the first and second images by configuring an exposure delay of at least one of the first and second cameras such that feature points are exposed at substantially the same time.

16. A non-transitory computer readable medium comprising instructions which, when executed by a processor, cause an electronic system to:
identify a pitch angle offset between a first camera and a second camera of a stereoscopic camera system supported by a frame of an eyewear device, the first camera producing a first image stream and the second camera producing a second image stream, and the first and second cameras having an overlapping field of view and a pitch angle offset, whereby feature points obtained by the first and second cameras i

Assignees

• Snap Inc

Inventors

• Ryuma Dmitry
• Ben Ezra David
• Hu Dunxu
• Simons Patrick Timothy Mcsweeney

Classifications

• H04N13/25

  using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor · CPC title

• G06T2207/10012

  Stereo images · CPC title

• H04N13/122

  Improving the three-dimensional [3D] impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues (H04N13/128 takes precedence) · CPC title

• H04N13/344

  with head-mounted left-right displays · CPC title

• H04N13/254

  in combination with electromagnetic radiation sources for illuminating objects · CPC title