Markerless image analysis for augmented reality

Patent metadata

Field	Value
Publication number	US-10733801-B2
Application number	US-201916384059-A
Country	US
Kind code	B2
Filing date	Apr 15, 2019
Priority date	May 9, 2017
Publication date	Aug 4, 2020
Grant date	Aug 4, 2020

Abstract

Systems and methods for a markerless approach to displaying an image of a virtual object in an environment are described. A computing device is used to capture an image of a real-world environment; for example including a feature-rich planar surface. One or more virtual objects which do not exist in the real-world environment are displayed in the image, such as by being positioned in a manner that they appear to be resting on the planar surface, based at least on a sensor bias value and scale information obtained by capturing multiple image views of the real-world environment.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
determining a sensor bias value associated with at least one sensor of a computing device;
acquiring, using at least one camera of the computing device, a first image;
determining a first planar surface represented in the first image;
acquiring, using the at least one camera, additional image data including different views of the first planar surface;
determining a distance and orientation of the first planar surface based at least on the sensor bias value and positional data associated with the additional image data;
determining scale information of the first planar surface in relation to the at least one camera based on the positional data associated with the additional image data and the sensor bias value; and
displaying a representation of a first object on a screen of the computing device, the representation of the first object appearing to be situated on the first planar surface at a first size in accordance with the determined distance and orientation.

2. The computer-implemented method of claim 1 , further comprising:
displaying a region of interest indicator on the screen;
determining a portion of the first image corresponding to the region of interest indicator; and
determining the first planar surface based at least on the portion of the first image.

3. The computer-implemented method of claim 1 , wherein the positional data is based on a mapping between at least two portions of the different views in the additional image data.

4. The computer-implemented method of claim 1 , wherein the sensor bias value is associated with at least one of an accelerometer or gyroscope of the computing device, and wherein determining a sensor bias value further comprises:
determining that movement of the device during a first duration of time is below a threshold level;
receiving, during the first duration of time, sensor data from the at least one of an accelerometer or gyroscope; and
determining the sensor bias value based at least on an average of the sensor data over at least a portion of the first duration of time.

5. The computer-implemented method of claim 1 , further comprising:
displaying a region of interest indicator on the screen;
determining a second planar surface corresponding to the region of interest indicator;
determining that the second planar surface does not have a minimum threshold amount of features in the first planar surface; and
displaying a visual indication corresponding to the first planar surface,
wherein the first planar surface did not correspond to the region of interest indicator.

6. The computer-implemented method of claim 1 , further comprising:
determining a second planar surface having a minimum threshold amount of features in the first planar surface;
displaying a first visual indicator associated with the first planar surface;
displaying a second visual indicator associated with the second planar surface;
receiving an indication of a selection of the first object;
receiving an indication of a selection of the first visual indicator;
receiving an indication of a selection of a second object;
receiving an indication of a selection of the second visual indicator; and
displaying a representation of the second object in the camera view on the screen, the representation of the second object appearing to be situated on the second planar surface at a size in accordance with the determined distance and orientation.

7. The computer-implemented method of claim 1 , further comprising:
receiving the first representation of the first object based on an indication of a selection of a uniform resource locator (URL) corresponding to the first object in an electronic marketplace.

8. The computer-implemented method of claim 1 , further comprising:
determining a second planar surface having a minimum threshold amount of features in the first planar surface;
acquiring, using the at least one camera, at least a second set of additional image data, the second set of additional image data including different views of the second planar surface;
determining a second distance and second orientation of the second planar surface based at least on the sensor bias value and positional data associated with the second set of additional image data; and
displaying a second representation of the first object, the second representation of the first object appearing to be situated on the second planar surface at a second size in accordance with the determined second distance and second orientation,
wherein the first size and the second size are different.

9. The computer-implemented method of claim 1 , wherein the features comprise one or more of gradient, entropy, or patterns.

10. A system, comprising:
at least one processor; and
memory including instructions that, when executed by the at least one processor, cause the system to:
determine a sensor bias value associated with at least one sensor of a computing device;
acquire, using at least one camera of a computing device, a first image;
determine a first planar surface represented in the first image;
acquire, using the at least one camera, additional image data including different views of the first planar surface;
determine a distance and orientation of the first planar surface based at least on the sensor bias value and positional data associated with the additional image data;
determine scale information of the first planar surface in relation to the at least one camera based on the positional data associated with the additional image data and the sensor bias value; and
display a representation of a first object in a camera view on a screen of the computing device, the representation of the first object appearing to be situated on the first planar surface at a size in accordance with the determined distance and orientation.

11. The system of claim 10 , wherein the instructions when executed further cause the system to:
display a region of interest indicator on the screen;
determine a portion of the first image corresponding to the region of interest indicator; and
determine the first planar surface based at least on the portion of the first image.

12. The system of claim 10 , wherein at least one of the first image or the additional image data comprises frames of video data captured by the at least one camera.

13. The system of claim 10 , wherein the sensor bias value is associated with at least one of an accelerometer or gyroscope of the computing device, and wherein the instructions for determining a sensor bias value when executed further cause the system to:
determine that movement of the device during a first duration of time is below a threshold level;
receive, during the first duration of time, sensor data from the at least one of an accelerometer or gyroscope; and
determine the sensor bias value based at least on an average of the sensor data over at least a portion of the first duration of time.

14. The system of claim 10 , wherein the instructions when executed further cause the system to:
display a region of interest indicator on the screen;
determine a second planar surface corresponding to the region of interest indicator;
determine that the second planar surface does not have a minimum threshold amount of features in the first planar surface; and
display a visual indication corresponding to the first planar surface,
wherein the first planar surface did not correspond to the region of interest indicator.

15. The system of claim 10 , wherein the instructions when executed further cause the system to:
deter

Assignees

• A9 Com Inc

Inventors

• Corso Nicholas
• Cutter Michael Patrick
• Lou Yu
• Niu Sean
• Post Shaun Michael
• Taylor Colin Jon
• Waldo Mark Scott

Classifications

• G06V20/20

  in augmented reality scenes · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06Q30/0623

  by investigating goods or services · CPC title

• G06T7/73

  using feature-based methods · CPC title

• G06Q30/0643

  graphically representing goods, e.g. 3D product representation · CPC title