Ultra low power camera pipeline for CV in AR systems

Patent metadata

Field	Value
Publication number	US-12267585-B2
Application number	US-202117485761-A
Country	US
Kind code	B2
Filing date	Sep 27, 2021
Priority date	Sep 30, 2020
Publication date	Apr 1, 2025
Grant date	Apr 1, 2025

Abstract

An eyewear device having an image processor operable in a camera pipeline for computer vision (CV) and in augmented reality (AR) systems. The image processor is configured to selectively control a plurality of cameras to provide images having a first resolution in the high power AR mode, and to provide the images having a second resolution in the low power CV mode. The first resolution is higher than the second resolution, and the plurality of cameras consume less power in the CV mode than the AR mode. The image processor controls the camera pipeline to process the first resolution high IQ images from the plurality of cameras to operate in the AR mode, and controls the camera pipeline to process the second resolution lower IQ images from the plurality of cameras to operate in the CV mode. Substantial power is saved by reducing the resolution of the images using downscaling in the cameras themselves in the CV mode.

First claim

What is claimed is:

1. An eyewear, comprising:
a frame;
a display supported by the frame;
a plurality of cameras coupled to the frame and configured to generate images, wherein the plurality of cameras is configured to provide images having a first resolution in a high power augmented reality (AR) mode, and to provide images having a second resolution in a low power computer vision (CV) mode;
a camera pipeline;
an ambient light sensor (ALS);
a processor configured to:
selectively control the plurality of cameras to provide the images, wherein each of the plurality of cameras is configured to provide the images having the first resolution in the AR mode;
selectively control the plurality of cameras such that only one of the plurality of cameras is configured to provide the images having a second resolution in the CV mode, wherein the first resolution is higher than the second resolution, and wherein the plurality of cameras consume less power in the CV mode than the AR mode;
control the camera pipeline to perform an auto white balance (AWB) process on the first resolution images from each of the plurality of cameras, without using data from the ALS; and
control the camera pipeline to perform an AWB process on the second resolution images from the one of the plurality of cameras, using data from the ALS.

2. The eyewear of claim 1 , wherein the processor is an image processor.

3. The eyewear of claim 1 , wherein the processor controls the one of the plurality of cameras to capture images in the first resolution and downscale the images captured in the first resolution to the second resolution.

4. The eyewear of claim 1 , wherein the processor is configured to selectively turn off a first camera of the plurality of cameras in the CV mode, and wherein the processor is configured to selectively turn the first camera back on and synchronize the plurality of cameras in the AR mode.

5. The eyewear of claim 1 , wherein the processor is configured to generate stitched, synchronized images.

6. The eyewear of claim 5 , wherein the processor is configured to match images from the plurality of cameras and put the matched images in a buffer.

7. A method of operating eyewear comprising a frame, a display supported by the frame, a plurality of cameras coupled to the frame and configured to generate images, wherein the plurality of cameras is configured to provide images having a first resolution in a high power augmented reality (AR) mode, and to provide images in a low power computer vision (CV) mode, a camera pipeline, an ambient light sensor (ALS), and a processor, the method comprising:
selectively controlling the plurality of cameras to provide the images, wherein each of the plurality of cameras is configured to provide the images having the first resolution in the AR mode;
selectively controlling the plurality of cameras such that only one of the plurality of cameras provides the images having a second resolution in the CV mode, wherein the first resolution is higher than the second resolution, and wherein the plurality of cameras consume less power in the CV mode than the AR mode;
in the AR mode, controlling the camera pipeline to perform an auto white balance (AWB) process on the first resolution images from each of the plurality of cameras, without using data from the ALS; and
in the CV mode, controlling the camera pipeline to perform an AWB process on the second resolution images from the one of the plurality of cameras, using data from the ALS.

8. The method of claim 7 , wherein the processor is an image processor.

9. The method of claim 7 , wherein the processor controls the one of the plurality of cameras to capture images in the first resolution and downscale the images captured in the first resolution to the second resolution.

10. The method of claim 7 , wherein the processor is configured to turn off a first camera of the plurality of cameras in the CV mode, and wherein the processor is configured to selectively turn the first camera back on and synchronize the plurality of cameras in the AR mode.

11. The method of claim 7 , wherein the processor generates stitched, synchronized images.

12. The method of claim 11 , wherein the processor matches images from the plurality of cameras and puts the matched images in a buffer.

13. A non-transitory computer-readable medium storing program code which, when executed by a processor of eyewear having a frame, a display supported by the frame, a plurality of cameras coupled to the frame and configured to generate images, wherein the plurality of cameras is configured to provide images having the first resolution in a high power augmented reality mode (AR) mode, and to provide images having a second resolution in a low power computer vision (CV) mode, an ambient light sensor (ALS), and a camera pipeline, is operative to cause the processor to perform the steps of:
selectively controlling the plurality of cameras to provide the images, wherein each of the plurality of cameras is configured to provide the images having the first resolution in the AR mode;
selectively controlling the plurality of cameras such that only one of plurality of cameras provides the images having the second resolution in the CV mode, wherein the first resolution is higher than the second resolution, and wherein the plurality of cameras consume less power in the CV mode than the AR mode;
in the AR mode, controlling the camera pipeline to perform an auto white balance (AWB) process on the first resolution images from each of the plurality of cameras, without using data from the ALS; and
in the CV mode, controlling the camera pipeline to perform an AWB process on the second resolution images from the one of the plurality of cameras, using data from the ALS.

14. The non-transitory computer-readable medium as specified in claim 13 , wherein the program code, when executed, is operative to cause the processor to control the one of the plurality of cameras to capture images in the first resolution, and downscale the images captured in the first resolution to the second resolution.

15. The non-transitory computer-readable medium as specified in claim 13 , wherein the processor is configured to turn off a first camera of the plurality of cameras in the CV mode, and wherein the processor is configured to selectively turn the first camera back on and synchronizes the plurality of cameras in the AR mode.

16. The non-transitory computer-readable medium as specified in claim 13 , wherein the program code, when executed, is operative to cause the processor to stitch and synchronize images.

Assignees

• Snap Inc

Inventors

• Ding Bo
• Doshi Chintan
• Kane Alexander
• Kondratenko Anton
• Nilles Gerald
• Robertson John James
• Ryuma Dmitry
• Kalkgruber Matthias

Classifications

• H04N23/80

  Camera processing pipelines; Components thereof · CPC title

• H04N23/90

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title

• H04N23/957

  Light-field or plenoptic cameras or camera modules · CPC title

• H04N23/951

  by using two or more images to influence resolution, frame rate or aspect ratio · CPC title

• H04N23/651 Primary

  for reducing power consumption by affecting camera operations, e.g. sleep mode, hibernation mode or power off of selective parts of the camera · CPC title