Augmented reality occlusion

Patent metadata

Field	Value
Publication number	US-10600247-B2
Application number	US-201715623690-A
Country	US
Kind code	B2
Filing date	Jun 15, 2017
Priority date	Jun 17, 2016
Publication date	Mar 24, 2020
Grant date	Mar 24, 2020

Abstract

A method for generating an augmented reality image from first and second images, wherein at least a portion of at least one of the first and the second image is captured from a real scene, the method comprising: identifying a confidence region in which a confident determination as to which of the first and second image to render in that region of the augmented reality image can be made; identifying an uncertainty region in which it is uncertain as to which of the first and second image to render in that region of the augmented reality image; determining at least one blending factor value in the uncertainty region based upon a similarity between a first colour value in the uncertainty region and a second colour value in the confidence region; and generating an augmented reality image by combining, in the uncertainty region, the first and second images using the at least one blending factor value.

First claim

The invention claimed is:

1. A method for generating an augmented reality image from first and second images, wherein at least a portion of at least one of the first and the second image is captured from a real scene, the method comprising:
identifying a confidence region in which a confident determination as to which of the first and second image to render in that region of the augmented reality image can be made;
identifying an uncertainty region in which it is uncertain as to which of the first and second image to render in that region of the augmented reality image;
determining at least one blending factor value in the uncertainty region based upon a similarity between a first colour value in the uncertainty region and at least one second colour value in the confidence region; and
generating an augmented reality image by combining, in the uncertainty region, the first and second images using the at least one blending factor value.

2. The method according to claim 1 , wherein the first image and the second image each have associated therewith a plurality of colour values and a corresponding plurality of depth values.

3. The method according to claim 2 , wherein the method further comprises making the confident determination as to which of the first image and the second image to render based upon a depth value of the first image and the corresponding depth value of the second image in the confidence region.

4. The method according to claim 2 , wherein the uncertainty region is identified based upon at least one depth value associated with at least one of the first and the second image, the at least one depth value being derived from a depth value captured from a real scene.

5. The method according to claim 4 , wherein the at least one depth value is derived from an unreliable or incomplete depth value captured from the real scene.

6. The method according to claim 1 , further comprising generating at least one initial blending factor value in a confidence region based upon the confident determination and wherein generating the augmented reality image further comprises combining a corresponding colour value of the first image and a corresponding colour value of the second image in the confidence region using the at least one initial blending factor value.

7. The method according to claim 1 , wherein identifying the confidence region further comprises categorising portions of the confidence region as first confidence regions or second confidence regions, wherein:
first confidence regions are confidence regions in which a colour value of the first image is to be rendered in the corresponding region of the augmented reality image; and
second confidence regions are confidence regions in which a colour value of the second image is to be rendered in the corresponding region of the augmented reality image.

8. The method according to claim 1 , wherein determining the at least one blending factor value is further based upon the distance between the position of the first colour value and the position of the at least one second colour value.

9. The method according to claim 1 , wherein the uncertainty region comprises a plurality of sample points and determining the at least one blending factor value further comprises processing, for each of a plurality of sample points in the uncertainty region, that sample point based upon colour values at a plurality of sample points located in a confidence region within a predetermined distance of that sample point.

10. The method according to claim 1 , wherein determining at least one blending factor value in the uncertainty region is based upon a similarity between a colour value in the uncertainty region and at least one corresponding colour value of each of the first image and the second image.

11. The method according to claim 1 , further comprising performing an erosion operation on the confidence region, wherein the erosion operation is configured to re-categorise at least one portion of the confidence region as forming a part of an uncertainty region.

12. The method according to claim 1 , wherein the first image is a captured image of a real scene and the second image is an image of a virtual object.

13. An augmented reality processing system for generating for generating an augmented reality image from first and second images, wherein at least a portion of at least one of the first and the second image is captured from a real scene, the augmented reality processing system comprising:
a confidence identification module arranged to identify a confidence region in which a confident determination as to which of the first and second image to render in that region of the augmented reality image can be made;
an uncertainty identification module arranged to identify an uncertainty region in which it is uncertain as to which of the first and second image to render in that region of the augmented reality image;
a blend module arranged to determine at least one blending factor value in the uncertainty region based upon a similarity between a first colour value in the uncertainty region and at least one second colour value in the confidence region; and
an image generation module arranged to generate an augmented reality image by combining, in the uncertainty region, the first and second images using the at least one blending factor value.

14. An augmented reality processing system according to claim 13 , wherein the first image and the second image each have associated therewith a plurality of colour values and a corresponding plurality of depth values.

15. The augmented reality processing system according to claim 13 , wherein the confidence identification module is arranged to categorise portions of the confidence region as first confidence regions or second confidence regions, wherein:
first confidence regions are confidence regions in which a colour value of the first image is to be rendered in the corresponding region of the augmented reality image; and
second confidence regions are confidence regions in which a colour value of the second image is to be rendered in the corresponding region of the augmented reality image.

16. The augmented reality processing system according to claim 13 , wherein the blend module is arranged to determine the at least one blending factor value further based upon the distance between the position of the first colour value and the position of the at least one second colour value.

17. The augmented reality processing system according to claim 13 , wherein the uncertainty region comprises a plurality of sample points and the blend module is arranged to determine the at least one blending factor value based upon processing, for each of a plurality of sample points in the uncertainty region, that sample point based upon colour values at a plurality of sample points located in a confidence region within a predetermined distance of that sample point.

18. The augmented reality processing system according to claim 13 , wherein the blend module is arranged to determine at least one blending factor value in the uncertainty region based upon a similarity between a colour value in the uncertainty region and at least one corresponding colour value of each of the first image and the second image.

19. The augmented reality processing system according to claim 13 , wherein the augmented reality processing system is embodied in hardware on an integrated circuit.

20. A non-transitory computer readable storage medium having stored thereon a computer readable dataset description

Assignees

• Imagination Tech Ltd

Inventors

• Walton David

Classifications

• G06T15/503

  Blending, e.g. for anti-aliasing · CPC title

• G06T11/00

  Two-dimensional [2D] image generation · CPC title

• G06T7/30

  Determination of transform parameters for the alignment of images, i.e. image registration · CPC title

• G06T3/4038

  Image mosaicing, e.g. composing plane images from plane sub-images · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title