Shared experience with contextual augmentation

Patent metadata

Field	Value
Publication number	US-9762851-B1
Application number	US-201615169453-A
Country	US
Kind code	B1
Filing date	May 31, 2016
Priority date	May 31, 2016
Publication date	Sep 12, 2017
Grant date	Sep 12, 2017

Abstract

Techniques for augmenting video content to enhance context of the video content are described herein. In some instances, a video may be captured at a first location and transmitted to a second location, where the video is output in real-time. A context surrounding a user that is capturing the video and/or a user that is viewing the video may be used to augment the video with additional content. For example, the techniques may process speech or other input associated with either user, a gaze associated with either user, a previous conversation for either user, an area of interest identified by either user, a level of understanding of either user, an environmental condition, and so on. Based on the processing, the techniques may determine augmentation content. The augmentation content may be displayed with the video in an overlaid manner to enhance the experience of the user viewing the video.

First claim

What is claimed is:

1. A method comprising:
receiving, by one or more computing devices, a panoramic video from a first mobile device, the panoramic video representing an environment in which the first mobile device is located;
determining, by the one or more computing devices, a geographic location of the first mobile device;
performing, by the one or more computing devices, image processing with the panoramic video to identify a feature in the panoramic video;
determining augmentation content that relates to the feature in the panoramic video and the geographic location of the first mobile device; and
causing, by the one or more computing devices, the augmentation content to be displayed via a second device along with the panoramic video to augment the panoramic video, the augmentation content being displayed in relation to a location of the feature in the panoramic video.

2. The method of claim 1 , further comprising:
receiving, from the first mobile device, speech input, the speech input being captured while the panoramic video is being captured; and
processing the speech input to determine that the speech input relates to the feature.

3. The method of claim 1 , further comprising:
receiving, from the second device, speech input, the speech input being captured while the panoramic video is being displayed; and
processing the speech input to determine that the speech input relates to the feature.

4. The method of claim 1 , further comprising: determining, based at least in part on gaze tracking data, that a user associated with the first mobile device is looking at the feature within the environment;
wherein the determining the augmentation content is based at least in part on determining that the user is looking at the feature within the environment.

5. The method of claim 1 , further comprising: determining that a user associated with the second device is looking at the location of the feature in the panoramic video;
wherein the determining the augmentation content is based at least in part on determining that the user is looking at the location of the feature in the panoramic video.

6. The method of claim 1 , further comprising: determining an interest of a user associated with the second device or a previous conversation that the user has had with an intelligent personal assistant; and
wherein the determining the augmentation content includes determining the augmentation content that relates to at least one of the interest of the user or the previous conversation that the user has had with the intelligent personal assistant.

7. The method of claim 1 , wherein the receiving the panoramic video from the first mobile device comprises receiving the panoramic video from a head-mounted device that captures the panoramic video.

8. A system comprising:
one or more processors; and
memory communicatively coupled to the one or more processors and storing executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
obtaining a video representing an environment in which a first user is located;
determining a geographic location of a mobile device associated with the first user;
performing image processing with the video to identify a feature in the video;
determining augmentation content based at least in part on the geographic location of the mobile device and identifying the feature in the video; and
providing the augmentation content and the video for display in substantially real-time via a device associated with a second user, the augmentation content to be displayed in an overlaid manner on the video.

9. The system of claim 8 , wherein the providing the augmentation content and the video for display includes:
designating the augmentation content to be displayed in an overlaid manner on the video at a location of the feature in the video; and
maintaining the augmentation content in relation to the feature as the location of the feature changes within the video.

10. The system of claim 8 , wherein the determining the augmentation content includes receiving, from the mobile device associated with the first user, annotation data to be used as the augmentation content.

11. The system of claim 8 , wherein the operations further comprise:
receiving input from the mobile device associated with the first user, the input identifying an area of interest;
wherein the performing image processing includes analyzing the video with respect to the area of interest to identify the feature.

12. The system of claim 8 , wherein the operations further comprise:
receiving input from the device associated with the second user, the input identifying an area of interest;
wherein the performing image processing includes analyzing the video with respect to the area of interest to identify the feature.

13. The system of claim 8 , wherein the operations further comprise:
determining a level of understanding of the second user with respect to content of the video based on at least one of a user profile for the second user or a previous conversation of the second user with an intelligent personal assistant; and
wherein the determining the augmentation content includes determining augmentation content that relates to the level of understanding of the second user with respect to content of the video.

14. The system of claim 8 , wherein the operations further comprise:
receiving, from the device associated with the second user, annotation data regarding the video;
receiving, from the device associated with the second user, a selection of a portion of the video;
determining a geographic location associated with the portion of the video; and
storing an association between the annotation data and the geographic location associated with the portion of the video.

15. The system of claim 8 , wherein the operations further comprise:
determining an environmental condition of the environment, the environmental condition including at least one of a location of the environment, a temperature of the environment, or a time of day at which the video is being captured at the environment;
and wherein the augmentation content includes content indicating the environmental condition.

16. The system of claim 8 , wherein the obtaining the video comprises receiving the video from a head-mounted device configured to capture the video of the environment.

17. A system comprising:
one or more processors; and
memory communicatively coupled to the one or more processors and storing executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
displaying a video representing an environment in which a first device is located;
receiving input regarding augmentation of the video;
identifying augmentation content that relates to a feature in the video and a geographic location of the first device, the augmentation content being based at least in part on the input; and
outputting the augmentation content as the video is displayed.

18. The system of claim 17 , wherein the receiving the input comprises receiving speech input via an intelligent personal assistant.

19. The system of claim 17 , wherein the identifying the augmentation content includes:
searching online to find content that is related to the feature in the video; and
retrieving the content from a source associated with the content.

20. The system of claim 17 , wherein the video comprises a panoram

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Baumert David
• Apacible Johnson T
• Bader Stephen David
• Behringer Adam Edwin
• Laurino Joseph
• Li Michelle
• Wang Hsiao-Lan

Classifications

• H04N7/147 Primary

  Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals (selecting H04Q) · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06V10/25

  Determination of region of interest [ROI] or a volume of interest [VOI] · CPC title

• G06F3/013

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title