Overlaying 3D augmented reality content on real-world objects using image segmentation

Patent metadata

Field	Value
Publication number	US-11869161-B2
Application number	US-202318105436-A
Country	US
Kind code	B2
Filing date	Feb 3, 2023
Priority date	Jan 21, 2019
Publication date	Jan 9, 2024
Grant date	Jan 9, 2024

Abstract

Various embodiments are generally directed to techniques of overlaying a virtual object on a physical object in augmented reality (AR). A computing device may receive one or more images of the physical object, perform analysis on the images (such as image segmentation) to generate a digital outline, and determine a position and a scale of the physical object based at least in part on the digital outline. The computing device may configure (e.g., rotate, scale) a 3D model of the physical object to match the determined position and scale of the physical object. The computing device may place or overlay a 3D virtual object on the physical object in AR based on a predefined location relation between the 3D virtual object and the 3D model of the physical object, and further, generate a composite view of the placement or overlay.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
identifying, by a processor, a symmetry-based mismatch between a three-dimensional (3D) model of a physical object and a digital outline of the physical object at a viewpoint;
causing, by the processor, the 3D model of the physical object to match a position and a scale of the physical object at the viewpoint based on the symmetry-based mismatch;
overlaying, by the processor, a 3D virtual object at a first location on an exterior portion of the physical object in augmented reality based on a predetermined locational relation between the 3D virtual object and the 3D model of the physical object; and
generating, by the processor, a composite view of the 3D virtual object overlaid on the exterior portion of the physical object.

2. The computer-implemented method of claim 1 , further comprising:
determining a type of the physical object based on a classification of an image of the physical object at the viewpoint; and
accessing the 3D model of the physical object based on the determined type of the physical object.

3. The computer-implemented method of claim 2 , wherein the image of the physical object is captured by a camera of a device comprising the processor.

4. The computer-implemented method of claim 2 , wherein the physical object is one of a vehicle or a house.

5. The computer-implemented method of claim 1 , the symmetry-based mismatch based at least in part on symmetrical characteristics of the physical object.

6. The computer-implemented method of claim 1 , further comprising prior to identifying the symmetry-based mismatch:
accessing the position and the scale of the physical object, wherein the position and the scale of the physical object based on an image, wherein the image is from the viewpoint.

7. The computer-implemented method of claim 6 , wherein the position of the physical object is based at least in part on one or more feature points on a floor in the image and a location of the 3D model of the physical object relative to a virtual floor corresponding to the floor, wherein the floor is based on a plane in the image.

8. The computer-implemented method of claim 1 , wherein the 3D virtual object is partially obstructed from view by the physical object at the viewpoint.

9. The computer-implemented method of claim 1 , wherein the 3D virtual object is an exterior object and a portion of the 3D virtual object is omitted from the composite view based on the portion of the 3D virtual object being partially obstructed by the physical object at the viewpoint.

10. The computer-implemented method of claim 1 , further comprising:
applying, by the processor, a machine learning (ML) model to the digital outline, wherein the ML model is trained on a two-dimensional (2D) outline of a three-dimensional (3D) model of the physical object at a plurality of rotations;
extracting, by the ML model, the 2D outline of the 3D model at each of the plurality of rotations of the physical object;
comparing, by the ML model, the 2D outline of the 3D model to the digital outline of the physical object at each of the plurality of rotations of the physical object; and
determining, by the ML model, a match between the 2D outline of the 3D model and the digital outline at a first rotation of the plurality of rotations of the physical object.

11. The computer-implemented method of claim 10 , wherein the position and the scale of the physical object is based on the match between the 2D outline of the 3D model and the digital outline at the first rotation of the plurality of rotations of the physical object.

12. The computer-implemented method of claim 1 , further comprising:
partitioning, by the processor, an image into a plurality of segments, the image from the viewpoint;
applying, by the processor, a respective label to each of a plurality of pixels in each of the plurality of segments;
locating, by the processor, the physical object based on the labels; and
determining, by the processor, the digital outline of the physical object based on the labels.

13. A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a processor, cause the processor to:
identify a symmetry-based mismatch between a three-dimensional (3D) model of a physical object and a digital outline of the physical object at a viewpoint;
cause the 3D model of the physical object to match a position and a scale of the physical object at the viewpoint based on the symmetry-based mismatch;
overlay a 3D virtual object at a first location on an exterior portion of the physical object in augmented reality based on a predetermined locational relation between the 3D virtual object and the 3D model of the physical object; and
generate a composite view of the 3D virtual object overlaid on the exterior portion of the physical object.

14. The computer-readable storage medium of claim 13 , the symmetry-based mismatch based at least in part on symmetrical characteristics of the physical object.

15. The computer-readable storage medium of claim 13 , wherein the instructions further cause the processor to, prior to identifying the symmetry-based mismatch:
access the position and the scale of the physical object, wherein the position and the scale of the physical object based on an image, wherein the image is from the viewpoint.

16. The computer-readable storage medium of claim 13 , wherein the 3D virtual object is partially obstructed from view by the physical object at the viewpoint.

17. A computing apparatus comprising:
a processor; and
a memory storing instructions that, when executed by the processor, cause the processor to:
identify a symmetry-based mismatch between a three-dimensional (3D) model of a physical object and a digital outline of the physical object at a viewpoint;
cause the 3D model of the physical object to match a position and a scale of the physical object at the viewpoint based on the symmetry-based mismatch;
overlay a 3D virtual object at a first location on an exterior portion of the physical object in augmented reality based on a predetermined locational relation between the 3D virtual object and the 3D model of the physical object; and
generate a composite view of the 3D virtual object overlaid on the exterior portion of the physical object.

18. The computing apparatus of claim 17 , the symmetry-based mismatch based at least in part on symmetrical characteristics of the physical object.

19. The computing apparatus of claim 17 , wherein the instructions further cause the processor to, prior to identifying the symmetry-based mismatch:
access the position and the scale of the physical object, wherein the position and the scale of the physical object based on an image, wherein the image is from the viewpoint.

20. The computing apparatus of claim 17 , wherein the 3D virtual object is partially obstructed from view by the physical object at the viewpoint.

Assignees

• Capital One Services Llc

Inventors

• Price Micah
• Dagley Geoffrey
• Duckworth Staevan
• Tang Qiaochu
• Hoover Jason
• Wylie Stephen
• Awoyemi Olalekan

Classifications

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T7/12

  Edge-based segmentation · CPC title

• G06T7/13

  Edge detection · CPC title

• G06T7/60

  Analysis of geometric attributes · CPC title

• G06T7/75

  involving models · CPC title