Block level update rate control based on gaze sensing

Patent metadata

Field	Value
Publication number	US-10123020-B2
Application number	US-201615395790-A
Country	US
Kind code	B2
Filing date	Dec 30, 2016
Priority date	Dec 30, 2016
Publication date	Nov 6, 2018
Grant date	Nov 6, 2018

Abstract

A method decodes video data based on gaze sensing. The method may decode an encoded video stream received from an encoder associated with a camera, and present the decoded video stream on a display of a device. The method may further detect a gaze point of an operator viewing the display, designate locations associated with the decoded video stream, based upon the gaze point, as skip block insertion points. The method may send the locations to the encoder, wherein the encoder reduces an update rate of inter-frame coded blocks corresponding to the skip block insertion points when encoding video data produced by the camera. An apparatus can implement the method to decode video data based on gaze sensing.

First claim

What is claimed is:

1. A method for decoding video data based on gaze sensing, comprising:
decoding an encoded video stream received from an encoder associated with a camera;
presenting the decoded video stream on a display of a device;
detecting a gaze point of an operator viewing the display;
designating locations associated with the decoded video stream, based upon the gaze point, as skip block insertion points; and
sending the locations to the encoder, wherein the encoder reduces an update rate of inter-frame coded blocks corresponding to the skip block insertion points when encoding video data produced by the camera.

2. The method of claim 1 , further comprising:
presenting the decoded video stream in a window having a primary focus of the operator on the display of the device;
determining that the gaze point of the operator is within the boundaries of the window having the primary focus of the operator;
determining a foveal vision area within the window having the primary focus of the operator; and
designating locations associated with the decoded video stream outside the foveal vision area as skip block insertion points.

3. The method of claim 1 , further comprising:
decoding at least one additional encoded video stream;
presenting the decoded video stream and the at least one additional decoded video stream each in separate windows from a plurality of windows on the display of the device, or on another display of the device;
determining, based upon the gaze point, a window from the plurality of windows having a primary focus of the operator; and
designating locations as skip block insertion points within the decoded video stream associated with the at least one window not having the primary focus of the operator.

4. The method of claim 3 , further comprising:
determining, based upon the gaze point, a foveal vision area within the window having the primary focus of the operator; and
designating locations outside the foveal vision area as skip block insertion points in the decoded video stream associated with the window having the primary focus of the operator.

5. The method of claim 3 , further comprising:
determining a group of pictures (GOP) length for a secondary decoded video stream associated with the at least one window not having the primary focus of the operator which is greater than the GOP length for the decoded video stream associated with the window having the primary focus of the operator;
sending the determined GOP length to an encoder associated with the secondary decoded video stream for encoding video associated with the at least one window not having the primary focus of the operator.

6. The method of claim 5 , further comprising:
determining a distance from the gaze point to the at least one window not having the primary focus of the operator; and
increasing the determined GOP length as the distance increases between the gaze point and the at least one window not having the primary focus of the operator.

7. The method of claim 2 , further comprising:
tracking a gaze point for a time period or a distance exceeding a predetermined threshold as the gaze point moves within the window having a primary focus of the operator;
correlating the movement of the gaze point and a moving object in the decoded video;
designating the moving object as an object of interest; and
preventing the designation of locations as skip block insertion points for locations associated with the object of interest in the decoded video stream.

8. The method of claim 7 , further comprising:
generating an identifier representing the designated object of interest; and
saving the identifier in a database containing metadata of the decoded video stream.

9. The method of claim 1 , further comprising:
determining that the gaze point is maintained at substantially the same position on the display for a time period exceeding a predetermined threshold; and
increasing a magnification of the decoded video stream in a predetermined area around the gaze point.

10. The method of claim 3 , further comprising:
determining that the gaze point is maintained for a time period exceeding a predetermined threshold on the window having the primary focus of the operator; and
increasing the magnification of the window having the primary focus of the operator.

11. The method of claim 1 , further comprising:
determining, as a result of blinking by the operator, that the gaze point disappears and reappears a predetermined number of times within a predetermined period of time, while maintaining substantially the same position on the display; and
executing a command associated with the decoded video stream in the area around the gaze point.

12. The method of claim 11 , wherein executing the command further comprises:
changing the magnification of the decoded video stream in the area around the gaze point, or saving an identifier in a database tagging the decoded video stream in the area around the gaze point.

13. The method of claim 1 , further comprising:
tracking positions of the gaze point over a period of time; and
predicting the next position of the gaze point based on the tracked positions of the gaze point.

14. The method of claim 1 , further comprising:
receiving a merged encoded video stream which includes a first component video stream having inter-frames which include skip blocks, and a second component video stream having a lower pixel density than the first component video stream sequence, wherein the second component video stream is temporally and spatially associated with the first component video stream;
identifying skip blocks in inter-frames of the first component video stream;
decoding inter-frames of the first component video stream in blocks which are not skip blocks;
decoding inter-frames of the second component video stream;
upscaling inter-frames in the decoded second component video stream to match the pixel density of the inter-frames in the decoded first component video stream;
identifying pixels in the upscaled decoded second component video stream which correspond to the skip blocks locations in the decoded first component video stream;
extracting the identified pixels in the decoded second component video stream; and
inserting the extracted pixels in corresponding locations of the skip blocks in the decoded first encoded bit stream.

15. A method for encoding video data based on gaze sensing, comprising
receiving video data captured by at least one sensor array;
receiving locations associated with a decoded video stream to designate skip block insertion points for encoding the received video data, wherein the locations are based on gaze points determined at a device;
identifying, based upon the received locations, skip block insertion points prior to encoding the received video data, wherein the skip block insertion points designate blocks within inter-frames having reduced update rates;
determining, for the identified skip block insertion points, a frequency for the reduced update rate; and
encoding inter-frames having blocks associated with the identified skip block insertion points based on the determined frequency.

16. The method of claim 15 , further comprising:
generating a first video sequence from the received video data;
generating a second video sequence from the received video data having a lower pixel density than the first video sequence, wherein the second video sequence is temporally and spatially similar to the first video sequence;
indicating pixels of relevance in the first video sequence, wherein the iden

Assignees

• Axis Ab

Inventors

• Ardo Bjorn
• Pihl Fredrik

Classifications

• H04N19/184

  the unit being bits, e.g. of the compressed video stream · CPC title

• H04N19/159

  Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction · CPC title

• H04N19/182

  the unit being a pixel · CPC title

• H04N19/167 Primary

  Position within a video image, e.g. region of interest [ROI] · CPC title

• H04N19/132

  Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking · CPC title