Placing augmented reality objects in an image based on device position

Patent metadata

Field	Value
Publication number	US-2022012896-A1
Application number	US-201917283688-A
Country	US
Kind code	A1
Filing date	Oct 9, 2019
Priority date	Oct 9, 2018
Publication date	Jan 13, 2022
Grant date	—

Abstract

An electronic device places an augmented reality object in an image of a real environment based on a pose of the electronic device and based on image segmentation. The electronic device includes a camera that captures images of the real environment and sensors, such as an inertial measurement unit (IMU), that capture a pose of the electronic device. The electronic device selects an augmented reality (AR) object from a memory, segments a captured image of the real environment into foreground pixels and background pixels, and composites an image for display wherein the AR object is placed between the foreground pixels and the background pixels. As the pose of the electronic device changes, the electronic device maintains the relative position of the AR object with respect to the real environment in images for display.

First claim

1 . A method comprising:
segmenting, at a processor of an electronic device, a first image of a real environment of the electronic device captured at a camera of the electronic device into foreground pixels and background pixels; combining, at the processor, the first image with an augmented reality (AR) object based on the segmenting to generate a first combined image; and displaying, at a display of the electronic device, the first combined image.

2 . The method of claim 1 , wherein combining the first image with the AR object comprises:
rendering, at the processor, the first image; rendering, at the processor, the AR object over the first image; and rendering, at the processor, the foreground pixels over the AR object to occlude at least a portion of the AR object.

3 . The method of claim 1 , further comprising:
generating, at the processor, a virtual environment to model the real environment of the electronic device; estimating, at the electronic device, a first pose of the electronic device based on the first image and pose data captured from one or more pose sensors during a first time period; identifying an initial relationship between the first pose and an initial pose of the AR object in the virtual environment; estimating, at the electronic device, a second pose of the electronic device based on a second image captured during a second time period following the first time period and pose data captured from the one or more pose sensors during the second time period; identifying a transformation between the second pose and the first pose; applying a corresponding transformation to the initial pose of the AR object in the virtual environment to identify a current pose for the AR object; and combining the second captured image with the AR object to generate a second combined image, comprising rendering the AR object in the second combined image at a position in the second combined image corresponding to the identified current pose for the AR object.

4 . The method of claim 3 , further comprising limiting the corresponding transformation to at least one of a threshold velocity and a threshold rotation.

5 . The method of claim 3 , further comprising:
in response to identifying that the pose sensors have lost track of the pose of the electronic device, locking the pose of the electronic device to a most recent pose before tracking was lost; and in response to detecting that the pose sensors have regained tracking and have continuously tracked the pose of the electronic device for a threshold period of time, setting the pose of the AR object such that the AR object is in the same location relative to a current pose of the electronic device as the AR object was located relative to the most recent pose.

6 . The method of claim 3 , wherein the pose sensors comprise one or more accelerometers and are further configured to generate pose data based on feature identification and matching, and further comprising:
identifying the current pose of the AR object based on pose data generated by the one or more accelerometers and omitting pose data based on feature identification and matching in response to detecting that pose tracking based on feature identification and matching has been lost.

7 . The method of claim 1 , further comprising:
periodically updating a pose of the AR object in the first combined image, comprising:
detecting a largest face in the first combined image; and
positioning the AR object in the first combined image proximate to the largest face.

8 . The method of claim 7 , further comprising:
interpolating the pose of the AR object to an updated pose over a predetermined time period, comprising accelerating the AR object from an initial pose toward the updated pose and decelerating the AR object as the AR object approaches the updated pose.

9 . A method, comprising:
capturing, at an image sensor of an electronic device, a first image of a real environment of the electronic device; segmenting, at a processor of the electronic device, the first image into foreground pixels and background pixels; rendering, at the processor, the first image; rendering, at the processor, an augmented reality (AR) object over the first image; rendering, at the processor, the foreground pixels of the first image over the AR object to generate a first combined image in which at least part of the AR object is occluded by the foreground pixels; and displaying, at a display of the electronic device, the first combined image.

10 . The method of claim 9 , further comprising:
generating, at the processor, a virtual environment to model the real environment of the electronic device; estimating, at the electronic device, a first pose of the electronic device based on the first image and pose data captured from one or more pose sensors during a first time period; identifying an initial relationship between the first pose and an initial pose of the AR object in the virtual environment; estimating, at the electronic device, a second pose of the electronic device based on a second image captured during a second time period following the first time period and pose data captured from the one or more pose sensors during the second time period; identifying a transformation between the second pose and the first pose; applying a corresponding transformation to the initial pose of the AR object in the virtual environment to identify a current pose for the AR object; and combining the second captured image with the AR object to generate a second combined image, comprising rendering the AR object in the second combined image at a position in the second combined image corresponding to the identified current pose for the AR object.

11 . The method of claim 10 , further comprising limiting the corresponding transformation to at least one of a threshold velocity and a threshold rotation.

12 . The method of claim 10 , further comprising:
in response to identifying that the pose sensors have lost track of the pose of the electronic device, locking the pose of the electronic device to a most recent pose before tracking was lost; and in response to detecting that the pose sensors have regained tracking and have continuously tracked the pose of the electronic device for a threshold period of time, setting the pose of the AR object such that the AR object is in the same location relative to a current pose of the electronic device as the AR object was located relative to the most recent pose.

13 . The method of claim 9 , further comprising:
periodically updating a pose of the AR object in the first combined image, comprising:
detecting a largest face in the first combined image; and
positioning the AR object in the first combined image proximate to the largest face.

14 . The method of claim 13 , further comprising:
interpolating to an updated pose of the AR object over a predetermined time period, comprising accelerating the AR object from an initial pose toward the updated pose and decelerating the AR object as the AR object approaches the updated pose.

15 . An electronic device, comprising:
a processor configured to:
segment a first image of a real environment of the electronic device captured at a camera of the electronic device into foreground pixels and background pixels; and
combine the first image with an augmented reality (AR) object based on the segmenting to generate a first combined image; and
a display configured to display the first combined image.

16 . The electronic device of claim 15 , wherein the processor is further configured to:
render t

Assignees

• Google Llc

Inventors

• Bond David
• Dochtermann Mark

Classifications

• G06V40/16

  Human faces, e.g. facial parts, sketches or expressions · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T11/00 Primary

  Two-dimensional [2D] image generation · CPC title

• G06T7/70

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title

• G06T7/20 Primary

  Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) · CPC title