Processing and formatting video for interactive presentation

Patent metadata

Field	Value
Publication number	US-10623662-B2
Application number	US-201615201049-A
Country	US
Kind code	B2
Filing date	Jul 1, 2016
Priority date	Jul 1, 2016
Publication date	Apr 14, 2020
Grant date	Apr 14, 2020

Abstract

Systems and methods are described for receiving, at a computing device, a video comprising a plurality of frames and determining, by the computing device, that vertical cropping should be performed for the video. For each frame of the plurality of frames, the computing device processes the video by analyzing the frame to determine a region of interest in the frame, wherein the frame is a first frame, cropping the first frame based on the region of interest in the frame to produce a vertically cropped frame for the video, determining a second frame immediately preceding the first frame, and smoothing a trajectory from the second frame to the vertically cropped frame. The vertically cropped frame is displayed to a user instead of the first frame.

First claim

What is claimed is:

1. A method comprising:
causing, by a computing device, display of a video comprising a sequence of frames on a display of the computing device;
detecting, by the computing device, user interaction with the display of the computing device during display of the video comprising the sequence of frames, the user interaction detected during display of a current frame of the sequence of frames;
determining, by the computing device, that the user interaction indicates a region of interest in the current frame of the sequence of frames of the displayed video;
detecting, by the computing device, that the computing device has been moved from a first angle or orientation to a second angle or orientation during the display of the video comprising the sequence of frames;
determining, by the computing device, that the displayed video should be cropped according to the user interaction indicating the region of interest to display a portion of each frame of the sequence of frames;
cropping, by the computing device, the current frame of the sequence of frames being displayed to generate a cropped current frame displaying the region of interest;
enlarging the cropped current frame based on a display size corresponding to the second orientation or angle of the computing device;
causing the enlarged cropped current frame to be displayed as part of the sequence of frames of the displayed video;
for each frame of the sequence of frames after the current frame:
cropping, by the computing device, the frame to generate a cropped frame comprising a portion of the frame including the region of interest in the frame and enlarging the cropped frame based on the display size corresponding to the second angle or orientation of the computing device during display of the video; and
causing the enlarged cropped frame to replace the frame such that the enlarged cropped frame is displayed in the sequence of frames of the video on the display of the computing device instead of the frame.

2. The method of claim 1 , wherein before cropping the frame to generate a cropped frame comprising a portion of the frame including the region of interest in the frame the method comprises:
analyzing the frame to determine whether there is a scene change from a previous frame immediately preceding the frame;
based on determining there is no scene change from the previous frame. using a region of interest of the previous frame as a region of interest for the frame;
based on determining there is a scene change from the previous frame, determining a region of interest for the frame based a saliency analysis of the frame;
wherein using the region of interest of the previous frame as the region of interest for the frame comprises using a visual tracking method to determine the region of interest from the previous frame.

3. The method of claim 2 , wherein determining the region of interest based on a saliency analysis further comprises:
generating a saliency map of the frame;
projecting the saliency map on a horizontal axis;
analyzing the saliency map to determine a most salient window of a predetermined size; and
setting the region of interest to the most salient window of the predetermined size.

4. The method of claim 3 , wherein determining the most salient window of a predetermined size is based on an aspect ratio of the display of the computing device.

5. The method of claim 2 , wherein analyzing the frame to determine whether there is a scene change is based on matching a color histogram or matching a directional histogram.

6. The method of claim 2 , wherein using the regions of interest of the previous frame as the region of interest for the frame comprises using a tracking method comprising at least one of an image based tracking method, motion estimations, optical flows, particle filters, and deep learning methods.

7. The method of claim 2 , further comprising:
smoothing a trajectory from the previous frame to the enlarged cropped frame before causing the enlarged cropped frame to be displayed.

8. The method of claim 1 , wherein the video is a landscape video.

9. The method of claim 1 , wherein the enlarged cropped frame is displayed on the display of the computing device as a first part of a split screen and the frame is displayed as a second part of the split screen, wherein first part of the split screen is larger than the second part of the split screen.

10. The method of claim 1 , further comprising:
storing the enlarged cropped frame.

11. The method of claim 1 , further comprising:
splitting the display of the computing device into a first part and a second part based on the second angle or orientation;
causing the enlarged cropped frame to be displayed in the first part of the display of the computing device; and
causing an original frame to be displayed in the second part of the display of the computing device, the original frame the frame from which the enlarged cropped frame was generated.

12. The method of claim 1 , further comprising:
splitting the display of the computing device into a first part and a second part based on the second angle or orientation;
causing the enlarged cropped frame to be displayed in the first part of the display of the computing device; and
causing the video comprising the sequence of frames to be displayed in the second part of the display of the computing device.

13. The method of claim 12 , further comprising:
detecting a second user interaction with the second part of the display of the computing device during display of the sequence of frames, the user interaction detected during display of a second current frame;
determining that the second user interaction indicates a changed region of interest in the second current frame of the sequence of frames being displayed in the second part of the display of the computing device;
cropping the second current frame of the sequence of frames being displayed in the second part of the display of the computing device to generate a cropped second current frame displaying the region of interest;
enlarging the cropped second current frame based on a display size corresponding to the first part of the display of the computing device; and
causing the enlarged cropped second current frame to be displayed in the first part of the display of the computing device.

14. The method of claim 13 , further comprising:
for each frame of the sequence of frames after the second current frame:
cropping the frame to generate a cropped frame comprising a portion of the frame including the region of interest in the frame and enlarging the cropped frame based on the display size corresponding to first part of the display of the computing device during display of the video; and
causing the enlarged cropped frame to replace the frame such that the enlarged cropped frame is displayed in the sequence of frames of the video in the first part of the display of the computing device instead of the frame.

15. A computing device comprising:
a display;
at least one processor; and
a non-transitory computer readable medium coupled with the processor, the computer readable medium comprising instructions stored thereon that are executable by the at least one processor to cause a computing device to perform operations comprising:
causing display of a video comprising a sequence of frames on a display of the computing device;
detecting user interaction with the display of the computing device during display of the video comprising the sequence of frames, the user interaction detected during display of a current frame of the sequence of frames,

Assignees

• Snap Inc

Inventors

• Li Jia
• Litke Nathan
• Paredes Jose Jesus (Joseph)
• Sheth Rahul Bhupendra
• Szeto Daniel
• Xu Ning
• Yang Jianchao

Classifications

• H04N5/2628 Primary

  Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation · CPC title

• G06K9/6212

  Physics · mapped topic

• G06K9/00718

  Physics · mapped topic

• G06V10/758

  Involving statistics of pixels or of feature values, e.g. histogram matching · CPC title

• G06V20/41

  Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49) · CPC title