Systems and methods for environment content sharing

Patent metadata

Field	Value
Publication number	US-9686466-B1
Application number	US-201313928639-A
Country	US
Kind code	B1
Filing date	Jun 27, 2013
Priority date	Jun 27, 2013
Publication date	Jun 20, 2017
Grant date	Jun 20, 2017

Abstract

Embodiments described herein may help to provide methods for sharing and viewing part of an environment of a computing device, such as a head-mountable device (HMD). An example method involves: (a) determining a still photo panorama of an environment, (b) receiving a video stream of a first portion of the environment from a video camera on a sharing device, (c) determining a registration data stream, where the registration data stream indicates a location and an orientation of the video stream within the still photo panorama of the environment; and (d) transmitting the video stream and the registration data stream to one or more viewing devices.

First claim

We claim:

1. A method, comprising:
receiving a continuous live video of a plurality of portions of an environment from a video camera on a sharing device while the video camera pans over the plurality of portions of the environment;
determining a still photo panorama of the environment by stitching together image frames of the plurality of portions of the environment from the continuous live video;
determining a registration data stream, wherein, for each respective image frame of the continuous live video, the registration data stream indicates coordinates at which to place the respective image frame of the continuous live video within the still photo panorama of the environment;
transmitting the still photo panorama, the continuous live video, and the registration data stream to a remote viewing device;
receiving, from the remote viewing device, live data indicative of current coordinates of a field of view of the remote viewing device within the still photo panorama, wherein the current coordinates of the field of view indicate a portion of the panorama currently viewed by the remote viewing device;
determining, based on the live data and the registration data stream, a current direction of the current coordinates of the field of view of the remote viewing device within the still photo panorama relative to current coordinates of the continuous live video within the still photo panorama; and
providing for display of visual cues indicating the determined current direction.

2. The method of claim 1 , wherein determining the still photo panorama comprises using one or more previously stored images of the environment.

3. The method of claim 1 , wherein determining the still photo panorama comprises updating one or more previously determined still photo panoramas of the environment.

4. The method of claim 1 , wherein determining the still photo panorama comprises using one or more images of the environment captured by one or more cameras on one or more additional sharing devices.

5. The method of claim 1 , wherein the still photo panorama comprises a 360 degree panorama.

6. The method of claim 1 , wherein the sharing device is a head-mountable device (HMD).

7. The method of claim 6 , wherein determining the still photo panorama comprises using one or more sensors on the HMD to determine head-orientation data indicative of head orientation.

8. The method of claim 6 , wherein determining the registration data stream comprises using one or more sensors on the HMD to determine head-orientation data indicative of head orientation.

9. The method of claim 6 , further comprising:
receiving a spatial audio signal from one or more audio sensors on the HMD;
transmitting the spatial audio signal to the remote viewing device.

10. The method of claim 6 , further comprising:
receiving a non-spatial audio signal from one or more audio sensors on the HMD; and
transmitting the non-spatial audio signal to the remote viewing device.

11. The method of claim 6 , further comprising receiving one or more audio signals from the remote viewing device.

12. The method of claim 1 , wherein the visual cues indicating the current direction indicate a direction for a user of the sharing device to move the sharing device to align the current coordinates of the continuous live video with the current coordinates of the field of view of the remote viewing device.

13. The method of claim 1 , further comprising:
receiving an additional continuous live video of an additional plurality of portions of the environment from an additional video camera on the sharing device;
determining an additional registration data stream for the additional continuous live video, wherein, for each respective image frame of the additional continuous live video, the additional registration data stream indicates coordinates at which to place the respective image frame of the additional continuous live video within the still photo panorama of the environment; and
transmitting the additional continuous live video and the additional registration data stream to the remote viewing device.

14. The method of claim 1 , further comprising:
receiving an additional continuous video of an additional plurality of portions of the environment from a video camera on an additional sharing device;
determining an additional registration data stream for the additional continuous live video, wherein, for each respective image frame of the additional continuous video, the additional registration data stream indicates coordinates at which to place the respective image frame of the additional continuous video within the still photo panorama of the environment; and
transmitting the additional continuous video and the additional registration data stream to the remote viewing device.

15. A method, comprising:
receiving a continuous video of a plurality of portions of an environment;
receiving a still photo panorama of the environment, wherein the still photo panorama comprises stitched-together image frames of the plurality of portions of the environment from the continuous video;
receiving a registration data stream, wherein, for each respective image frame of the continuous video, the registration data stream indicates coordinates at which to place the respective image frame of the continuous video within the still photo panorama of the environment to create an augmented panorama;
displaying a portion of the augmented panorama in a display of a viewing device, wherein the continuous video is moving within the augmented panorama based at least in part on the registration data stream;
moving, based on an input signal to the viewing device, a field of view within the augmented panorama to control the portion of the augmented panorama displayed on the display of the viewing device;
when the video stream is outside the field of view, determining, based on the registration data stream, a current direction of current coordinates of the continuous video stream within the augmented panorama relative to current coordinates of the field of view within the augmented panorama; and
displaying, in the display of the viewing device, visual cues indicating the current direction.

16. The method of claim 15 , wherein the field of view has a same size as the video stream.

17. The method of claim 15 , wherein the viewing device is a first viewing device, the method further comprising:
receiving one or more viewing directions, wherein each of the one or more viewing directions indicates a direction of current coordinates of a field of view of one or more other viewing devices within the augmented panorama relative to current coordinates of the field of view of the first viewing device within the augmented panorama; and
displaying, in the display of the first viewing device, visual cues indicating the one or more viewing directions.

18. The method of claim 15 , further comprising:
receiving an additional continuous video of a plurality of additional portions of the environment;
receiving an additional registration data stream, wherein, for each respective image frame of the additional continuous video, the additional registration data stream indicates coordinates at which to place the respective image frame of the additional continuous video within the augmented panorama; and
displaying the additional continuous video in the display of the viewing device, wherein the additional continuous video is moving within the augmented panorama based at least in part on the additional registration data stream.

19. The meth

Assignees

• Google Inc

Inventors

• Billinghurst Mark Nathan
• Tate Matthew David

Classifications

• H04N23/661 Primary

  Transmitting camera control signals through networks, e.g. control via the Internet · CPC title

• H04N23/698 Primary

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• H04N23/635

  Region indicators; Field of view indicators · CPC title

• H04N5/23238 Primary

  Electricity · mapped topic