Dynamically rendering 360-degree videos using view-specific-filter parameters

Patent metadata

Field	Value
Publication number	US-11178374-B2
Application number	US-201916428201-A
Country	US
Kind code	B2
Filing date	May 31, 2019
Priority date	May 31, 2019
Publication date	Nov 16, 2021
Grant date	Nov 16, 2021

Abstract

This disclosure relates to methods, non-transitory computer readable media, and systems that generate and dynamically change filter parameters for a frame of a 360-degree video based on detecting a field of view from a computing device. As a computing device rotates or otherwise changes orientation, for instance, the disclosed systems can detect a field of view and interpolate one or more filter parameters corresponding to nearby spatial keyframes of the 360-degree video to generate view-specific-filter parameters. By generating and storing filter parameters for spatial keyframes corresponding to different times and different view directions, the disclosed systems can dynamically adjust color grading or other visual effects using interpolated, view-specific-filter parameters to render a filtered version of the 360-degree video.

First claim

We claim:

1. A non-transitory computer readable medium storing instruction thereon that, when executed by at least one processor, cause a computer system to:
determine a spatial-temporal-viewing location comprising spatial coordinates and a time coordinate within a 360-degree video corresponding to a current field of view on a display device;
identify one or more key-spatial-temporal locations comprising one or more spatial coordinates and one or more time coordinates within the 360-degree video that are proximate in direction and time to the spatial coordinates and the time coordinate of the spatial-temporal-viewing location, each key-spatial-temporal location having associated filter parameters;
based on identifying the one or more key-spatial-temporal locations, interpolate one or more filter parameters associated with the one or more key-spatial-temporal locations to generate view-specific-filter parameters for the current field of view; and
render a filtered version of the current field of view of the 360-degree video using the view-specific-filter parameters.

2. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computer system to:
determine the spatial-temporal-viewing location based on a view-direction vector at the time coordinate within the 360-degree video; and
identify the one or more key-spatial-temporal locations of the 360-degree video by identifying the one or more key-spatial-temporal locations based on a proximity in direction among the one or more spatial coordinates of the one or more key-spatial-temporal locations to the spatial coordinates of the spatial-temporal-viewing location in three-dimensional space and a proximity in time among the one or more time coordinates of the one or more key-spatial-temporal locations to time coordinate of the spatial-temporal-viewing location.

3. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computer system to:
repeatedly determine spatial-temporal-viewing locations corresponding to fields of view on the display device; and
for each determined spatial-temporal-viewing location corresponding to a detected field of view, render a new filtered version of a detected field of view of the 360-degree video using newly interpolated view-specific-filter parameters.

4. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computer system to determine the spatial-temporal-viewing location within a frame of the 360-degree video indicated by an orientation of the display device.

5. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computer system to interpolate the one or more filter parameters associated with the one or more key-spatial-temporal locations by interpolating one or more color-grading parameters reflecting adjusted lighting conditions for corresponding key-spatial-temporal locations of the 360-degree video.

6. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computer system to interpolate the one or more filter parameters associated with the one or more key-spatial-temporal locations by interpolating one or more blur-filter parameters, film-grain-filter parameters, vignette-filter parameters, or watercolor-filter parameters for corresponding key-spatial-temporal locations of the 360-degree video.

7. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computer system to interpolate the one or more filter parameters associated with the one or more key-spatial-temporal locations by tri-linearly interpolating the one or more filter parameters associated with the one or more key-spatial-temporal locations.

8. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computer system to:
determine an additional spatial-temporal-viewing location within the 360-degree video corresponding to an adjusted field of view on the display device comprising an object initially within the current field of view;
identify one or more additional key-spatial-temporal locations of the 360-degree video that are proximate in direction and time to the additional spatial-temporal-viewing location corresponding to the adjusted field of view;
based on identifying the one or more additional key-spatial-temporal locations, interpolate one or more additional filter parameters associated with the one or more additional key-spatial-temporal locations to generate adjusted view-specific-filter parameters for the adjusted field of view; and
render an adjusted filtered version of the adjusted field of view of the 360-degree video comprising the object using the adjusted view-specific-filter parameters.

9. A system comprising:
a display screen for a display device;
at least one processor;
at least one non-transitory computer readable medium comprising a 360-degree video, filter parameters corresponding to key-spatial-temporal locations, and instructions that, when executed by the at least one processor, cause the system to:
determine a spatial-temporal-viewing location comprising spatial coordinates and a time coordinate within a 360-degree video corresponding to a current field of view on the display device based on an orientation of the display screen;
identify a first key-spatial-temporal location and a second key-spatial-temporal location of the 360-degree video based on one or more spatial coordinates and one or more time coordinates of the first key-spatial-temporal location and the second key-spatial-temporal location being proximate in direction and time to the spatial coordinates and the time coordinate of the spatial-temporal-viewing location;
based on identifying the first key-spatial-temporal location and the second key-spatial-temporal location, generate view-specific-filter parameters for the current field of view by interpolating a first set of filter parameters corresponding to the first key-spatial-temporal location and a second set of filter parameters corresponding to the second key-spatial-temporal location; and
render a filtered version of the current field of view of the 360-degree video on the display screen using the view-specific-filter parameters.

10. The system of claim 9 , further comprising instructions that, when executed by the at least one processor, cause the system to access the first set of filter parameters and the second set of filter parameters from metadata within a video file for the 360-degree video, the metadata being separate from frames of the 360-degree video.

11. The system of claim 9 , wherein the first key-spatial-temporal location and the second key-spatial-temporal location each comprise a pan-angle coordinate for a view-direction vector, a tilt-angle coordinate for the view-direction vector, and a particular time coordinate corresponding to a spatial keyframe of the 360-degree video.

12. The system of claim 9 , further comprising instructions that, when executed by the at least one processor, cause the system to:
interpolate the first set of filter parameters and the second set of filter parameters by interpolating a first set of color-grading parameters reflecting adjusted lighting conditions corresponding to the first k

Assignees

• Adobe Inc

Inventors

• Diverdi Stephen
• Walker Seth
• Wang Oliver
• Nguyen Cuong

Classifications

• H04N9/74 Primary

  for obtaining special effects (H04N9/65 - H04N9/73 take precedence) · CPC title

• H04N23/698

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• H04N23/10

  for generating image signals from different wavelengths · CPC title

• H04N13/111 Primary

  Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation · CPC title

• H04N13/383

  for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes · CPC title