Mixed media reality recognition with image tracking

Patent metadata

Field	Value
Publication number	US-9972108-B2
Application number	US-201514746669-A
Country	US
Kind code	B2
Filing date	Jun 22, 2015
Priority date	Jul 31, 2006
Publication date	May 15, 2018
Grant date	May 15, 2018

Abstract

An MMR system integrating image tracking and recognition comprises a plurality of mobile devices, a pre-processing server or MMR gateway, and an MMR matching unit, and may include an MMR publisher. The MMR matching unit receives an image query from the pre-processing server or MMR gateway and sends it to one or more of the recognition units to identify a recognition result. Image tracking information also is provided for determining relative locations of images to each other. The mobile device includes an image tracker for providing at least a portion of the image tracking information. The disclosure also includes methods for image tracking-assisted recognition, recognition of multiple images using a single image query, and improved image tracking using MMR recognition.

First claim

What is claimed is:

1. A method comprising:
receiving a first image and a second image;
receiving image tracking information identifying a position, a rotation, and a scale of the first image relative to the second image, wherein the image tracking information includes sequence and relative timing information for the first image and the second image;
determining a first set of image features of the first image and a second set of image features of the second image;
combining the first set of image features and the second set of image features into a superset of image features using the image tracking information including the position, the rotation and the scale of the first image relative to the second image;
forming a single image query corresponding to the first image and the second image based on the superset of image features;
submitting the single image query corresponding to the first image and the second image; and
receiving a recognition result matching the single image query.

2. The method of claim 1 , wherein the image tracking information comprises instructions for the combining.

3. The method of claim 1 , further comprising:
combining the first image and the second image into a merged image based on the superset of image features.

4. The method of claim 3 , wherein the merged image is a combined image stitched together from the first image and the second image.

5. The method of claim 3 , wherein the merged image is a super-resolution image synthesized from the first image and the second image.

6. The method of claim 1 , wherein the first set of image features and the second set of image features for the same location are combined into consensus features.

7. The method of claim 6 , further comprising excluding from the superset of image features, features that are determined to be inconsistent with the consensus features.

8. The method of claim 1 , further comprising cropping the recognition result based upon the image tracking information indicating the relative locations of the first image and the second image to each other.

9. The method of claim 1 , wherein the first image and the second image are from a video stream.

10. A system comprising:
one or more processors; and
a memory coupled with the one or more processors, the memory storing instructions, which when executed, cause the one or more processors to:
receive a first image and a second image;
receive image tracking information identifying a position, a rotation, and a scale of the first image relative to the second image, wherein the image tracking information includes sequence and relative timing information for the first image and the second image;
determine a first set of image features of the first image and a second set of image features of the second image;
combine the first set of image features and the second set of image features into a superset of image features using the image tracking information including the position, the rotation and the scale of the first image relative to the second image;
form a single image query corresponding to the first image and the second image based on the superset of image features;
submit the single image query corresponding to the first image and the second image; and
receive a recognition result matching the single image query.

11. The system of claim 10 , wherein the instructions further cause the one or more processors to:
combine the first image and the second image into a merged image based on the superset of image features.

12. The system of claim 11 , wherein the merged image is a combined image stitched together from the first image and the second image.

13. The system of claim 11 , wherein the merged image is a super-resolution image synthesized from the first image and the second image.

14. A computer program product comprising a non-transitory computer useable medium including a computer readable program, wherein the computer readable program, when executed on a computer, causes the computer to:
receive a first image and a second image;
receive image tracking information identifying a position, a rotation, and a scale of the first image relative to the second image, wherein the image tracking information includes sequence and relative timing information for the first image and the second image;
determine a first set of image features of the first image and a second set of image features of the second image;
combine the first set of image features and the second set of image features into a superset of image features using the image tracking information including the position, the rotation and the scale of the first image relative to the second image;
form a single image query corresponding to the first image and the second image based on the superset of image features;
submit the single image query corresponding to the first image and the second image; and
receive a recognition result matching the single image query.

15. The computer program product of claim 14 , wherein the computer readable program further causes the computer to:
combine the first image and the second image into a merged image based on the superset of image features.

16. The computer program product of claim 15 , wherein the merged image is a combined image stitched together from the first image and the second image.

Assignees

• Moraleda Jorge
• Erol Berna
• Hull Jonathan J
• Ricoh Co Ltd

Inventors

• Moraleda Jorge
• Erol Berna
• Hull Jonathan J

Classifications

• G06V30/414 Primary

  Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text · CPC title

• G06T11/60 Primary

  Creating or editing images; Combining images with text · CPC title

• G06F18/25

  Fusion techniques · CPC title

• G06V10/16

  using multiple overlapping images; Image stitching · CPC title

• G06V10/24

  Aligning, centring, orientation detection or correction of the image · CPC title