360 degree image presentation

Patent metadata

Field	Value
Publication number	US-10097759-B1
Application number	US-201615282821-A
Country	US
Kind code	B1
Filing date	Sep 30, 2016
Priority date	Sep 30, 2015
Publication date	Oct 9, 2018
Grant date	Oct 9, 2018

Abstract

Techniques are disclosed for stabilizing a stream of spherical images captured by an image capture device to produce a stabilized spherical video sequence. The rotation of the image capture device during capture may be corrected in one or more desired axial directions in a way that is agnostic to the translation of the image capture device. The rotation of the image capture device may also be corrected in one or more desired axial directions in a way that is aware of the translation of the image capture device. For example, the assembled output spherical video sequence may be corrected to maintain the horizon of the scene at a constant location, regardless of the translation of the image capture device (i.e., a ‘translation-agnostic’ correction), while simultaneously being corrected to maintain the yaw of the scene in the direction of the image capture device's translation through three-dimensional space (i.e., a ‘translation-aware’ correction).

First claim

The invention claimed is:

1. A method to stabilize a stream of spherical images, comprising:
obtaining an input stream of spherical images from an image capture device, wherein each spherical image comprises corresponding metadata indicating a motion of the image capture device in one or more axial directions during capture of the respective spherical image;
stabilizing the input stream of spherical images based on each spherical image's corresponding metadata; and
storing the stream of stabilized spherical images in a memory,
wherein stabilizing the input stream of spherical images comprises correcting a rotational motion in at least a first axial direction of at least one of the spherical images, and
wherein the correction of the rotational motion in the at least a first axial direction of the at least one of the spherical images accounts for a translational motion of the image capture device.

2. The method of claim 1 , wherein stabilizing the input stream of spherical images further comprises correcting a rotational motion in at least a second axial direction of at least one of the spherical images, wherein the correction of the rotational motion in the at least a second axial direction of the at least one of the spherical images does not account for a translational motion of the image capture device.

3. The method of claim 1 , further comprising:
analyzing the input stream of spherical images to identify one or more translational stabilization points,
wherein the correction of the rotational motion in the at least a first axial direction of the at least one of the spherical images is further based, at least in part, on the one or more identified translational stabilization points.

4. The method of claim 3 , wherein the identified one or more translational stabilization points comprise one or more of the following: a reference roll angle, a reference pitch angle, or a reference yaw angle.

5. The method of claim 3 , wherein at least one of the identified one or more translational stabilization points comprises an object of interest.

6. The method of claim 1 , wherein the at least a first axial direction comprises a yaw axis.

7. The method of claim 1 , wherein the translational motion of the image capture device comprises a filtered motion path.

8. A non-transitory program storage device comprising instructions stored thereon to cause one or more processors to:
obtain an input stream of spherical images from an image capture device, wherein each spherical image comprises corresponding metadata indicating a motion of the image capture device in one or more axial directions during capture of the respective spherical image;
stabilize the input stream of spherical images based on each spherical image's corresponding metadata; and
store the stream of stabilized spherical images in a memory,
wherein stabilizing the input stream of spherical images comprises correcting a rotational motion in at least a first axial direction of at least one of the spherical images, and
wherein the correction of the rotational motion in the at least a first axial direction of the at least one of the spherical images accounts for a translational motion of the image capture device.

9. The non-transitory program storage device of claim 8 , wherein the instructions to stabilize the input stream of spherical images further comprise instructions to correct a rotational motion in at least a second axial direction of at least one of the spherical images, wherein the correction of the rotational motion in the at least a second axial direction of the at least one of the spherical images does not account for a translational motion of the image capture device.

10. The non-transitory program storage device of claim 8 , further comprising instructions stored thereon to cause the one or more processors to:
analyze the input stream of spherical images to identify one or more translational stabilization points,
wherein the correction of the rotational motion in the at least a first axial direction of the at least one of the spherical images is further based, at least in part, on the one or more identified translational stabilization points.

11. The non-transitory program storage device of claim 10 , wherein the identified one or more translational stabilization points comprise one or more of the following: a reference roll angle, a reference pitch angle, or a reference yaw angle.

12. The non-transitory program storage device of claim 10 , wherein at least one of the identified one or more translational stabilization points comprises an object of interest.

13. The non-transitory program storage device of claim 8 , wherein the at least a first axial direction comprises a yaw axis.

14. The non-transitory program storage device of claim 8 , wherein the translational motion of the image capture device comprises a filtered motion path.

15. An electronic system, comprising:
an image capture device;
a display element;
memory operatively coupled to the image capture device and the display element; and
one or more processors operatively coupled to the image capture device, the display element, and the memory, the memory having stored therein instructions that, when executed by the one or more processors, cause the electronic system to:
obtain an input stream of spherical images from the image capture device, wherein each spherical image comprises corresponding metadata indicating a motion of the image capture device in one or more axial directions during capture of the respective spherical image;
stabilize the input stream of spherical images based on each spherical image's corresponding metadata; and
store the stream of stabilized spherical images in the memory,
wherein the instructions that, when executed by the one or more processors, cause the electronic system to stabilize the input stream of spherical images further comprise instructions that, when executed by the one or more processors, cause the electronic system to correct a rotational motion in at least a first axial direction of at least one of the spherical images, and
wherein the correction of the rotational motion in the at least a first axial direction of the at least one of the spherical images accounts for a translational motion of the image capture device.

16. The electronic system of claim 15 , wherein the instructions that, when executed by the one or more processors, cause the electronic system to stabilize the input stream of spherical images further comprise instructions that, when executed by the one or more processors, cause the electronic system to correct a rotational motion in at least a second axial direction of at least one of the spherical images, wherein the correction of the rotational motion in the at least a second axial direction of the at least one of the spherical images does not account for a translational motion of the image capture device.

17. The electronic system of claim 15 , further comprising instructions stored in the memory that, when executed by the onr ore more processors, cause the electronic system to:
analyze the input stream of spherical images to identify one or more translational stabilization points,
wherein the correction of the rotational motion in the at least a first axial direction of the at least one of the spherical images is further based, at least in part, on the one or more identified translational stabilization points.

18. The electronic system of claim 17 , wherein the identified one or more translational stabilization points comprise one or more of the following:

Assignees

• Apple Inc

Inventors

• Motta Ricardo
• Kuo David D
• Krenn Matthaeus

Classifications

• H04N23/6811

  based on the image signal · CPC title

• H04N23/698 Primary

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• H04N23/683 Primary

  performed by a processor, e.g. controlling the readout of an image memory · CPC title

• H04N23/6812

  based on additional sensors, e.g. acceleration sensors · CPC title

• H04N5/23238

  Electricity · mapped topic