Augmented reality system for communicating tagged video and data on a network

Patent metadata

Field	Value
Publication number	US-9280850-B2
Application number	US-201113291866-A
Country	US
Kind code	B2
Filing date	Nov 8, 2011
Priority date	Nov 8, 2010
Publication date	Mar 8, 2016
Grant date	Mar 8, 2016

Abstract

A system, method, and computer program product for automatically combining computer-generated imagery with real-world imagery in a portable electronic device by retrieving, manipulating, and sharing relevant stored videos, preferably in real time. A video is captured with a hand-held device and stored. Metadata including the camera's physical location and orientation is appended to a data stream, along with user input. The server analyzes the data stream and further annotates the metadata, producing a searchable library of videos and metadata. Later, when a camera user generates a new data stream, the linked server analyzes it, identifies relevant material from the library, retrieves the material and tagged information, adjusts it for proper orientation, then renders and superimposes it onto the current camera view so the user views an augmented reality.

First claim

What is claimed is:

1. A computer-implemented method for providing an augmented reality interface, comprising:
preloading a plurality of image snippets from a server based on a location information of a portable electronic device, each of the plurality of image snippets being a smaller image file of a corresponding stored image;
acquiring an image of a real-world scene and metadata with a camera of a portable electronic device;
overlaying one or more of the plurality of image snippets corresponding to a location and a direction at which a corresponding image was originally captured based on the location information and an orientation information of the portable electronic device without manipulating the plurality of image snippets based on corner features of the image of the real-world scene;
manipulating a retrieved image corresponding to an displayed image snippet based on matching corner feature information of the displayed image snippet generated and provided by the server and corner features of the image of the real-world scene, wherein the retrieved image is retrieved with metadata having selected features; and
combining the manipulated image with a currently observed real-world scene viewed with the portable electronic device.

2. The method of claim 1 , wherein the retrieving includes pushing a data stream to a network.

3. The method of claim 1 , wherein the retrieving includes pulling the at least one stored image from a network in response to a request.

4. The method of claim 1 , wherein the network includes at least one of a private network and the internet.

5. The method of claim 1 , wherein the image is at least one of a still photograph, at least one video frame, analog, digital, recorded, live, and communicated in a data stream.

6. The method of claim 1 , wherein the metadata describes the physical location and orientation of the camera during the acquiring, and is provided by at least one of a GPS system, a gyroscope, and an accelerometer.

7. The method of claim 1 , wherein at least one of the currently observed scene, images, and metadata are stored on at least one of a server and the portable electronic device.

8. The method of claim 1 , wherein the location information and the orientation information of the portable electronic device comprise best matching one of (a) a current physical location and orientation of the portable electronic device and (b) a predicted physical location and orientation of the portable electronic device.

9. The method of claim 1 , wherein the server searches for the selected features.

10. The method of claim 1 , wherein the retrieved image is in a second data stream.

11. The method of claim 1 , wherein the portable electronic device is at least one of a smartphone, a hand-held device, the camera, a second camera, a PDA, and a tablet computer.

12. The method of claim 1 , wherein the manipulating includes adjusting image orientation.

13. The method of claim 1 , wherein the combining includes superimposing the manipulated image on the currently observed scene.

14. The method of claim 5 , wherein the combining includes merging the data stream with a second data stream.

15. The method of claim 1 , wherein the combining includes displaying the manipulated image with the portable electronic device in one of a display and a viewfinder.

16. The method of claim 1 , wherein the method operates continuously and substantially in real time.

17. The method of claim 1 , wherein the method operates as the currently observed scene changes as the portable electronic device is moved, such motion including at least one of translating, tilting, panning, and zooming.

18. The computer-implemented method of claim 1 wherein the corner feature information of the retrieved image is determined using a markerless feature tracking algorithm performed by the server.

19. A system for providing an augmented reality interface, comprising:
a processor; and
a memory containing instructions that, when executed by the processor cause the processor to:
preload a plurality of video snippets from a server based on a location of a location information of a portable electronic device, each of the plurality of video snippets being a smaller video file of a corresponding stored video;
acquire a video of a real-world scene and metadata with a camera of a portable electronic device;
overlay one or more of the plurality of video snippets corresponding to a location and a direction at which a corresponding video was originally captured based on the location information and an orientation information of the portable electronic device without manipulating the plurality of video snippets based on corner features of the video of the real-world scene;
manipulate a retrieved video corresponding to an displayed video snippet based on matching corner feature information of the displayed video snippet generated and provided by the server and corner features of the video of the real-world scene, wherein the retrieved video is retrieved with metadata having selected features; and
combine the manipulated video with a currently observed real-world scene viewed with the portable electronic device.

20. A computer program product for providing an augmented reality interface, comprising a non-transitory computer readable medium tangibly embodying non-transitory computer-executable program instructions thereon that, when executed, cause a computing device to:
preload a plurality of video snippets from a server based on a location of a location information of a portable electronic device, each of the plurality of video snippets being a smaller video file of a corresponding stored video;
acquire a video of a real-world scene and metadata with a camera of a portable electronic device;
overlay one or more of the plurality of video snippets corresponding to a location and a direction at which a corresponding video was originally captured based on the location information and an orientation information of the portable electronic device without manipulating the plurality of video snippets based on corner features of the video of the real-world scene;
manipulate a retrieved video corresponding to an displayed video snippet based on matching corner feature information of the displayed video snippet generated and provided by the server and corner features of the video of the real-world scene, wherein the retrieved video is retrieved with metadata having selected features; and
combine the manipulated video with a currently observed real-world scene viewed with the portable electronic device.

21. A system for providing an augmented reality interface, comprising:
means for preloading a plurality of video snippets from a server based on a location of a location information of a portable electronic device, each of the plurality of video snippets being a smaller video file of a corresponding stored video;
means for acquiring a video of a real-world scene and metadata with a camera of a portable electronic device;
means for overlaying one or more of the plurality of video snippets corresponding to a location and a direction at which a corresponding video was originally captured based on the location information and an orientation information of the portable electronic device without manipulating the plurality of video snippets based on corner features of the video of the real-world scene;
means for manipulating a retrieved video corresponding to an displayed video snippet based on matching corner feature information of

Assignees

• Adhikari Suranjit
• Dunn Ted
• Hsiao Eric
• Sony Corp

Inventors

• Adhikari Suranjit
• Dunn Ted
• Hsiao Eric

Classifications

• G09G2340/14

  Solving problems related to the presentation of information to be displayed · CPC title

• G06Q30/0623 Primary

  by investigating goods or services · CPC title

• G09G5/00

  Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators (image data processing or generation, in general G06T) · CPC title

• G09G2354/00

  Aspects of interface with display user · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title