Flexible eyewear device with dual cameras for generating stereoscopic images

Patent metadata

Field	Value
Publication number	US-12108019-B2
Application number	US-202318384273-A
Country	US
Kind code	B2
Filing date	Oct 26, 2023
Priority date	Dec 20, 2018
Publication date	Oct 1, 2024
Grant date	Oct 1, 2024

Abstract

Three-dimensional image calibration and presentation for eyewear including a pair of image capture devices is described. Calibration and presentation includes obtaining a calibration offset to accommodate flexure in the support structure for the eyewear, adjusting a three-dimensional rendering offset by the obtained calibration offset, and presenting the stereoscopic images using the three-dimension rendering offset.

First claim

What is claimed is:

1. Image capture eyewear, including:
a support structure;
dual cameras connected to the support structure to capture stereoscopic images of a scene;
a processor;
a memory accessible to the processor; and
programming in the memory, wherein execution of the programming by the processor configures the eyewear to perform functions, including functions to:
capture, using the dual cameras, stereoscopic images of the scene, the scene containing at least one object with known dimensions;
obtain a distance to the at least one object with known dimensions based on a size of the at least one object with known dimensions;
identify an actual offset between the stereoscopic images for the at least one object with known dimensions;
determine a plurality of calibration offsets as one of (1) a difference between the actual offset and a previously determined offset for the at least one object with known dimensions determined with other image capture eyewear not experiencing any flexure, or (2) an amount of flexure experienced by the image capture eyewear; and
store the calibration offsets in the memory along with flexure amounts corresponding to the calibration offsets.

2. The eyewear of claim 1 , wherein the support structure includes a frame and wherein execution of the programming by the processor further configures the eyewear to perform functions, including functions to:
determine an amount of flexure of the frame during use of the image capture eyewear; and
generate a stereoscopic image using a calibration offset corresponding to the amount of flexure of the frame.

3. The eyewear of claim 1 , wherein the execution of the programming by the processor further configures the eyewear to rectify the stereoscopic images to remove distortion.

4. The eyewear of claim 1 , wherein execution of the programming by the processor further configures the eyewear to obtain the distance to the at least one object with known dimensions by determining at least one of a height or a width of the at least one object with known dimensions from bounding rectangles detected by a deep neural network.

5. The eyewear of claim 4 , wherein the deep neural network is trained to directly estimate a distance to the at least one object with known dimensions.

6. The eyewear of claim 1 , further comprising a laser measuring device incorporated into the support structure that provides a distance measurement to the at least one object with known dimensions.

7. The eyewear of claim 1 , wherein execution of the programming by the processor further configures the eyewear to identify the actual offset between the stereoscopic images for the at least one object with known dimensions by determining a number of pixels between a position of a feature in a left image and a position of the feature in a right image in a horizontal direction as the actual offset.

8. The eyewear of claim 1 , further comprising a strain gauge in the support structure, wherein execution of the programming by the processor further configures the eyewear to determine the amount of flexure experienced by the image capture eyewear as an estimate based on predefined calibration offset values associated with predefined levels of strain measured by the strain gauge.

9. A calibration method for image capture eyewear, the method comprising the steps of:
capturing, using dual cameras on a support structure of the image capture eyewear, stereoscopic images of a scene containing at least one object with known dimensions;
obtaining, using a processor of the image capture eyewear, a distance to the at least one object with known dimensions based on a size of the at least one object with known dimensions;
identifying, using the processor, an actual offset between the stereoscopic images for the at least one object with known dimensions;
determining, using the processor, a plurality of calibration offsets as one of (1) a difference between the actual offset and a previously determined offset for the at least one object with known dimensions determined with other image capture eyewear not experiencing any flexure, or (2) an amount of flexure experienced by the image capture eyewear; and
storing the calibration offsets in a memory coupled to the processor along with flexure amounts corresponding to the calibration offsets.

10. The method of claim 9 , wherein the support structure includes a frame, further comprising:
determining an amount of flexure of the frame during use of the image capture eyewear; and
generating a stereoscopic image using a calibration offset corresponding to the amount of flexure of the frame.

11. The method of claim 9 , further comprising rectifying the stereoscopic images to remove distortion.

12. The method of claim 9 , wherein obtaining the distance to the at least one object with known dimensions includes determining at least one of a height or a width of the at least one object with known dimensions from bounding rectangles detected by a deep neural network.

13. The method of claim 12 , further comprising training the deep neural network to directly estimate a distance to the at least one object with known dimensions.

14. The method of claim 9 , further comprising providing a distance measurement to the at least one object with known dimensions using a laser measuring device incorporated into the image capture eyewear.

15. The method of claim 9 , wherein identifying the actual offset between the stereoscopic images for the at least one object with known dimensions comprises determining a number of pixels between a position of a feature in a left image and a position of the feature in a right image in a horizontal direction as the actual offset.

16. The method of claim 9 , further comprising determining the amount of flexure experienced by the image capture eyewear as an estimate based on predefined calibration offset values associated with predefined levels of strain measured by a strain gauge.

17. A non-transitory computer readable medium comprising instructions which, when executed by one or more processors of image capture eyewear, cause the one or more processors to calibrate the image capture eyewear by performing operations comprising:
capturing stereoscopic images of a scene containing at least one object with known dimensions;
obtaining a distance to the at least one object with known dimensions based on a size of the at least one object with known dimensions;
identifying an actual offset between the stereoscopic images for the at least one object with known dimensions;
determining a plurality of calibration offsets as one of (1) a difference between the actual offset and a previously determined offset for the at least one object with known dimensions determined with other image capture eyewear not experiencing any flexure, or (2) an amount of flexure experienced by the image capture eyewear; and
storing the calibration offsets in a memory along with flexure amounts corresponding to the calibration offsets.

18. The medium of claim 17 , wherein the instructions, when executed by the one or more processors, further cause the one or more processors to:
determine an amount of flexure of a frame of the image capture eyewear during use of the image capture eyewear; and
generate a stereoscopic image using a calibration offset corresponding to the amount of flexure of the frame.

19. The medium of claim 17 , wherein the instructions, when executed by the one or more processors, further cause the one or more processors to rectify the stereoscopic images to remov

Assignees

• Snap Inc

Inventors

• Ardisana Ii John Bernard
• Daube Nir
• Ben Ezra David
• Hintermann Mathias Andreas
• Katz Sagi
• Kligler Nati
• Ryner Michael
• Shukla Ashutosh

Classifications

• G02B27/0172

  characterised by optical features · CPC title

• H04N13/344 Primary

  with head-mounted left-right displays · CPC title

• H04N23/54

  Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils · CPC title

• H04N13/246

  Calibration of cameras · CPC title

• G02B2027/014

  comprising information/image processing systems · CPC title