Method and apparatus for image processing in augmented reality systems

Patent metadata

Field	Value
Publication number	US-9626803-B2
Application number	US-201414569549-A
Country	US
Kind code	B2
Filing date	Dec 12, 2014
Priority date	Dec 12, 2014
Publication date	Apr 18, 2017
Grant date	Apr 18, 2017

Abstract

Disclosed are a system, apparatus, and method for depth and color camera image synchronization. Depth and color camera input images are received or otherwise obtained unsynchronized and without associated creation timestamps. An image of one type is compared with an image of a different type to determine a match for synchronization. Matches may be determined according to edge detection or depth coordinate detection. When a match is determined a synchronized pair is formed for processing within an augmented reality output. Optionally the synchronized pair may be transformed to improve the match between the image pair.

First claim

What is claimed is:

1. A method for depth and color camera image synchronization, the method comprising:
obtaining a plurality of input images comprising a first image type and a second image type different from the first image type, wherein an image type is either a depth camera image type or color camera image type, wherein each image type is unsynchronized with respect to images of other types, and wherein each of the plurality of input images of the first image type and of the second image type are without identification of an input image's actual creation time;
selecting, from input images of the first image type, a target image;
selecting, from input images of the second image type, a comparison image;
determining whether the comparison image is compatible for synchronization with the target image;
in response to a determination that the comparison image is compatible for synchronization with the target image, identifying the target image and the comparison image as a synchronized image pair; and
processing the synchronized image pair comprising the target image of the first image type and the comparison image of the second image type together for use in an augmented reality output.

2. The method of claim 1 , wherein the target image is a color camera image type, and the comparison image is a depth camera image type, and wherein determining whether the comparison image is compatible for synchronization with the target image further comprises:
tracking three-dimensional (3D) coordinates from the plurality of input images of the color camera image type, wherein the 3D coordinates include depth values;
determining, within the comparison image, depth values for each point having a geometrically equivalent point to the coordinates tracked in the target image;
comparing the depth values within the comparison image to depth values within the target image; and
determining a compatibility score according to a number of depth value matches between the target image and the comparison image.

3. The method of claim 1 , wherein determining whether the comparison image is compatible for synchronization with the target image further comprises:
performing edge detection on the target image and the comparison image; and
determining a compatibility score according to a number of same edge matches between the target image and the comparison image.

4. The method of claim 1 , further comprising:
determining a time of receipt for each of the plurality of input images;
creating a comparison queue comprising a subset of images from the plurality of input images, wherein the subset of images in the comparison queue comprises images of the second image type received within a threshold time from receipt of the target image, and wherein the comparison image is selected from the comparison queue.

5. The method of claim 4 , wherein the comparison queue comprises a threshold number of images of the second image type received before and after receipt of the target image.

6. The method of claim 1 , further comprising:
determining a timing offset between receipt time of the comparison image and the receipt time of the target image; and
referencing the timing offset to determine a next synchronized pair.

7. The method of claim 6 , further comprising:
tracking one or both of an amount of time passed or a number of images processed;
determining one or both of threshold amount of time passed or number of images processed is met;
determining compatibility between a next target image and a next comparison image; and
assigning a next synchronized image pair of together for use in the augmented reality output.

8. The method of claim 1 , further comprising:
transforming one or both of the target image or the comparison image according to parameters extracted from one or both of the target image or the comparison image.

9. A device for depth and color camera image synchronization comprising:
memory; and
a processor coupled to the memory and configured to:
obtain, from the memory, a plurality of input images comprising a first image type and a second image type different from the first image type, wherein an image type is either a depth camera image type or color camera image type, wherein each image type is unsynchronized with respect to images of other types, and wherein each of the plurality of input images of the first image type and of the second image type are without identification of an input image's actual creation time;
select, from input images of the first image type, a target image;
select, from input images of the second image type, a comparison image;
determine whether the comparison image is compatible for synchronization with the target image;
in response to a determination that the comparison image is compatible for synchronization with the target image, identify the target image and the comparison image as a synchronized image pair; and
process the synchronized image pair comprising the target image of the first image type and the comparison image of the second image type together for use in an augmented reality output.

10. The device of claim 9 , wherein the target image is a color camera image type, and the comparison image is a depth camera image type, and wherein to determine whether the comparison image is compatible for synchronization with the target image the processor is further configured to:
track three-dimensional (3D) coordinates from the plurality of input images of the target image, wherein the 3D coordinates include depth values;
determine, within the comparison image, depth values for each point having a geometrically equivalent point to the coordinates tracked in the target image;
compare the depth values within the comparison image to depth values within the target image; and
determine a compatibility score according to a number of depth value matches between the target image and the comparison image.

11. The device of claim 9 , wherein to determine whether the comparison image is compatible for synchronization with the target image the processor the processor is further configured to:
perform edge detection on the target image and the comparison image; and
determine a compatibility score according to a number of same edge matches between the target image and the comparison image.

12. The device of claim 9 , wherein the processor is further configured to:
determine a time of receipt for each of the plurality of input images;
create a comparison queue comprising a subset of images from the plurality of input images, wherein the subset of images in the comparison queue comprises images of the second image type received within a threshold time from receipt of the target image, and wherein the comparison image is selected from the comparison queue.

13. The device of claim 12 , wherein the comparison queue comprises a threshold number of images of the second image type received before and after receipt of the target image.

14. The device of claim 9 , wherein the processor is further configured to:
determine a timing offset between receipt time of the comparison image and the receipt time of the target image; and
reference the timing offset to determine a next synchronized pair.

15. The device of claim 14 , wherein the processor is further configured to:
track one or both of an amount of time passed or a number of images processed;
determine one or both of threshold amount of time passed or number of images processed is met;
determine compatibility between a next target image and a next comparison image; and
assign a next synchronized image pair of together fo

Assignees

• Qualcomm Inc

Inventors

• Park Youngmin
• Mendez Mendez Erick
• Reitmayr Gerhard
• Wagner Daniel
• Diaz Spindola Serafin

Classifications

• H04N13/0037

  Electricity · mapped topic

• G06T2207/10016

  Video; Image sequence · CPC title

• H04N2213/003

  Aspects relating to the "2D+depth" image format · CPC title

• G06T2200/04

  involving 3D image data · CPC title

• G06T7/13

  Edge detection · CPC title