Systems and methods for activities solver development in augmented reality applications

Patent metadata

Field	Value
Publication number	US-9792732-B2
Application number	US-201514885684-A
Country	US
Kind code	B2
Filing date	Oct 16, 2015
Priority date	Jul 1, 2011
Publication date	Oct 17, 2017
Grant date	Oct 17, 2017

Abstract

Systems and methods for generating an augmented reality interface for generics activities are disclosed. The systems and methods may be directed to creating an augmented reality display for an activity performed on a surface. Given an image of the activity, an activity solver library and associated configuration information for the activity may be selected. The surface of the activity from the image may be rectified, forming a rectified image, from which activity state information may be extracted using the configuration information. The activity state information may be provided to the activity solver library to generate solution information, and elements indicating the solution information may be rendered in a perspective of the original image. By providing the configuration information associated with an activity solver library, an augmented reality interface can be generated for an activity by capturing an image of the activity.

First claim

What is claimed is:

1. A computer-implemented method comprising:
receiving an image;
determining an activity being performed in the received image, wherein the activity is configured to be performed on a surface;
selecting (i) a particular activity solver, from a set of activity solvers stored in memory, that provides solution data for the activity being performed in the received image and (ii) configuration data associated with the activity being performed in the received image;
based on the configuration data associated with the activity being performed in the received image, (i) generating a rectified image of the surface of the activity performed in the received image, and (ii) determining an activity state of the activity being performed in the received image;
wherein determining an activity state of the activity being performed in the received image is based on the rectified image in addition to the configuration data associated with the activity being performed in the received image;
providing data indicating the activity state to the particular activity solver, selected from the set of activity solvers, that provides solution data for the activity being performed in the received image;
in response to providing data indicating the activity state to the particular activity solver, obtaining solution data, from the selected particular activity solver, for the activity being performed in the received image in accordance with the data indicating the activity state provided to the particular activity solver; and
providing, for display, a representation of the solution data, obtained from the selected particular activity solver, for the activity being performed in the received image.

2. The method of claim 1 , wherein generating a rectified image of the surface comprises:
identifying a first perspective associated with the surface in the received image;
determining a second perspective associated with the rectified image of the surface, wherein the first perspective is different from the second perspective; and
generating the rectified image of the surface using the second perspective.

3. The method of claim 1 , wherein determining an activity being performed in the received image, the activity being configured to be performed on a surface, comprises identifying the surface using quadrilateral rectification.

4. The method of claim 1 , wherein determining an activity state of the activity being performed in the received image based on the rectified image and the configuration data comprises:
dividing the rectified image into a plurality of image regions;
for each respective image region of the plurality of image regions, determining state data associated with the respective image region; and
determining the activity state of the activity being performed in the received image based on the state data associated with the plurality of image regions.

5. The method of claim 1 , comprising:
detecting a relative arrangement of features within the received image based on one or more training images associated with the activity; and
determining a mapping between the received image and the rectified image.

6. The method of claim 1 , wherein determining an activity state of the activity being performed in the received image comprises determining the activity state using optical character recognition.

7. The method of claim 1 , wherein determining an activity state of the activity being performed in the received image comprises matching a portion of the received image with one or more training images associated with the activity.

8. A non-transitory computer-readable medium storing instructions, which, when executed by one or more computers, cause the one or more computers to perform operations of:
receiving an image;
determining an activity being performed in the received image, wherein the activity is configured to be performed on a surface;
selecting (i) a particular activity solver, from a set of activity solvers stored in memory, that provides solution data for the activity being performed in the received image and (ii) configuration data associated with the activity being performed in the received image;
based on the configuration data associated with the activity being performed in the received image, (i) generating a rectified image of the surface of the activity performed in the received image, and (ii) determining an activity state of the activity being performed in the received image;
wherein determining an activity state of the activity being performed in the received image is based on the rectified image in addition to the configuration data associated with the activity being performed in the received image;
providing data indicating the activity state to the particular activity solver, selected from the set of activity solvers, that provides solution data for the activity being performed in the received image;
in response to providing data indicating the activity state to the particular activity solver, obtaining solution data, from the selected particular activity solver, for the activity being performed in the received image in accordance with the data indicating the activity state provided to the particular activity solver; and
providing, for display, a representation of the solution data, obtained from the selected particular activity solver, for the activity being performed in the received image.

9. The computer-readable medium of claim 8 , wherein generating a rectified image of the surface comprises:
identifying a first perspective associated with the surface in the received image;
determining a second perspective associated with the rectified image of the surface, wherein the first perspective is different from the second perspective; and
generating the rectified image of the surface using the second perspective.

10. The computer-readable medium of claim 8 , wherein determining an activity being performed in the received image, the activity being configured to be performed on a surface, comprises identifying the surface using quadrilateral rectification.

11. The computer-readable medium of claim 8 , wherein determining an activity state of the activity being performed in the received image based on the rectified image and the configuration data comprises:
dividing the rectified image into a plurality of image regions;
for each respective image region of the plurality of image regions, determining state data associated with the respective image region; and
determining the activity state of the activity being performed in the received image based on the state data associated with the plurality of image regions.

12. The computer-readable medium of claim 8 , the operations further comprising:
detecting a relative arrangement of features within the received image based on one or more training images associated with the activity; and
determining a mapping between the received image and the rectified image.

13. The computer-readable medium of claim 8 , wherein determining an activity state of the activity being performed in the received image comprises determining the activity state using optical character recognition.

14. The computer-readable medium of claim 8 , wherein determining an activity state of the activity being performed in the received image comprises matching a portion of the received image with one or more training images associated with the activity.

15. A system comprising:
one or more processors and one or more computer storage media storing instructions that are operable, when executed by the one or more processors, to cause the one or more processors to p

Assignees

• Google Inc

Inventors

• Palm Leon G
• Adam Hartwig

Classifications

• G06T3/00 Primary

  Geometric image transformations in the plane of the image · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• A63F3/02

  Chess; Similar board games · CPC title