Augmentation of multimedia consumption

Patent metadata

Field	Value
Publication number	US-9854328-B2
Application number	US-201213542696-A
Country	US
Kind code	B2
Filing date	Jul 6, 2012
Priority date	Jul 6, 2012
Publication date	Dec 26, 2017
Grant date	Dec 26, 2017

Abstract

Disclosed are methods and apparatus for augmenting a user's multimedia consumption experience. The methods comprise while the user is consuming the multimedia presentation using a first device, that device provides (to one or more remote processors) information that may be used to identify a relevant location. The one or more processors use this information to identify the location and acquire a virtual environment. This virtual environment may be a virtual representation the location. The virtual environment is presented to the user on a second (companion) device. Using the second device, the user may explore the virtual environment and interact with virtual objects therein.

First claim

We claim:

1. A method of augmenting a user's multimedia consumption experience, the user being a user of a first user device and of a second user device, the user using the first device to consume a multimedia presentation, the method comprising:
providing, for use by one or more processors remote from the first device, information that may be used by the one or more processors to identify a location, the location being relevant to at least part of the multimedia presentation;
identifying, by the one or more processors, the location;
acquiring, by the one or more processors, a virtual environment, the virtual environment being representative of the identified location;
acquiring, by the one or more processors, one or more virtual objects, each virtual object relating to at least part of the multimedia presentation;
determining, by the one or more processors, for each virtual object, a position for that virtual object within the virtual environment;
for each virtual object, including that virtual object in the virtual environment at the position within the virtual environment determined for that virtual object; and
presenting to the user, using the second device, the virtual environment with the one or more virtual objects included therein;
wherein the virtual environment is to some extent explorable by the user using the second user device; and
wherein the virtual objects within the virtual environment are capable of being interacted with by the user.

2. A method according to claim 1 :
wherein each virtual object corresponds to an object within the multimedia presentation;
the method further comprising changing one or more properties of a virtual object of the one or more virtual objects within the virtual environment in response to a change, in the multimedia presentation, to one or more properties of the object that corresponds to that virtual object.

3. A method according to claim 1 :
wherein each virtual object corresponds to an object within the multimedia presentation;
the method further comprising, in response to the user interacting with a virtual object of the one or more virtual objects within the virtual environment, changing, in the multimedia presentation, one or more properties of the object that corresponds to that virtual object.

4. A method according to claim 1 wherein the location is a location at which events that occur in the multimedia presentation take place or that is referred to at some point in the multimedia presentation.

5. A method according to claim 4 wherein the location is a location at which events occurring in a portion of the multimedia presentation that is currently being consumed by the user take place or a location that is currently being referred to in a portion of the multimedia presentation that is currently being consumed by the user.

6. A method according to claim 1 wherein including comprises overlaying the one or more objects onto the virtual environment using an augmented-reality process.

7. A method according to claim 1 wherein the virtual environment is a three-dimensional virtual environment.

8. A method according to claim 1 :
wherein the virtual environment is an environment in which the user may interact with a further user;
wherein the further user is a user of a further device; and
wherein the virtual environment is being presented or has been presented, using the further device, to the further user.

9. A method according to claim 1 wherein the information that may be used by the one or more processors to identify the location comprises first identification data, the first identification data identifying the multimedia presentation, and second identification data, the second identification data identifying a point or range in the multimedia presentation that is currently being consumed by the user.

10. A method according to claim 1 wherein a watermark that is present in a portion of the multimedia presentation comprises information that may be used by the one or more processors to identify the location.

11. A method according to claim 1 wherein acquiring the virtual environment comprises selecting one or more images of the identified location and forming the virtual environment using the selected images.

12. A method according to claim 1 wherein the virtual environment comprises a panoramic representation of the location.

13. A method according to claim 1 wherein the multimedia presentation is selected from a group of multimedia presentations consisting of: a video presentation, a television program, and a movie.

14. A method according to claim 1 wherein the first device is the same as the second device.

15. A method according to claim 14 :
wherein the second device may be moved between having a first orientation and a second orientation;
wherein when the second device is positioned in the first orientation, the second device displays the multimedia presentation; and
wherein when the second device is positioned in the second orientation, the second device displays the virtual environment.

16. A method of augmenting a user's multimedia consumption experience, the user being a user of a first user device and of a second user device, the user using the first device to consume a multimedia presentation, the method comprising:
providing, for use by one or more processors remote from the first device, information that may be used by the one or more processors to identify a location, the location being relevant to at least part of the multimedia presentation;
identifying, by the one or more processors, the location;
acquiring, by the one or more processors, a virtual environment; and
presenting to the user, using the second device, the virtual environment;
wherein providing information that may be used by the one or more processors to identify a location comprises either:
the first device providing the information that may be used by the one or more processors to identify a location; or
the one or more processors receiving the information that may be used by the one or more processors to identify a location from one or more servers, the one or more servers storing information that may be used to identify a location;

wherein the virtual environment is representative of the identified location; and
wherein the virtual environment is to some extent explorable by the user using the second user device.

17. A method according to claim 16 further comprising presenting to the user, using the second device, one or more virtual objects, the virtual objects being presented as objects within the virtual environment, the virtual objects being capable of being interacted with by the user.

18. A method according to claim 17 :
wherein each virtual object corresponds to an object within the multimedia presentation;
the method further comprising:
changing one or more properties of a virtual object within the virtual environment in response to a change, in the multimedia presentation, to one or more properties of the object that corresponds to that virtual object; or
in response to the user interacting with a virtual object within the virtual environment, changing, in the multimedia presentation, one or more properties of the object that corresponds to that virtual object.

19. Apparatus for augmenting a user's multimedia consumption experience, the user being a user of a first user device and of a second user device, the user using the first device to consume a multimedia presentation, the apparatus comprising one or more processors arranged to:
use informa

Assignees

• Venkitaraman Narayanan
• Chaysinh Shirley A
• Mandalia Hiren M
• Metcalf Crysta J
• Novak Ashley B
• Kulka Isaac G
• Arris Entpr Inc

Inventors

• Venkitaraman Narayanan
• Chaysinh Shirley A
• Mandalia Hiren M
• Metcalf Crysta J
• Novak Ashley B
• Kulka Isaac G

Classifications

• H04N21/4722

  for requesting additional data associated with the content · CPC title

• H04N21/4307

  Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen · CPC title

• H04N21/8133 Primary

  specifically related to the content, e.g. biography of the actors in a movie, detailed information about an article seen in a video programme · CPC title

• H04N21/84

  Generation or processing of descriptive data, e.g. content descriptors {(systems specially adapted for using meta-information in broadcast systems H04H60/73)} · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title