Method and system of video coding using display modification input

Patent metadata

Field	Value
Publication number	US-10666946-B2
Application number	US-201615282508-A
Country	US
Kind code	B2
Filing date	Sep 30, 2016
Priority date	Jul 1, 2016
Publication date	May 26, 2020
Grant date	May 26, 2020

Abstract

Techniques described herein are related to video coding using display modification input.

First claim

What is claimed is:

1. A computer-implemented method of video coding comprising:
obtaining temporal display content information of image data of frames of a frame sequence and comprising object data of at least one object being moved from frame to frame along the sequence due to user or automatic input to an application providing the temporal display content information so that the temporal display content information is more than color-related or brightness-related pixel image data values alone, and wherein the at least one object is defined before encoding and the temporal display content information is formed before encoding and for at least one reason that is not solely for encoding;
obtaining pixel image data of the frames to encode the frame sequence; and encoding the pixel image data comprising determining at least one motion vector to be used to compress the pixel image data of the at least one object and to transmit at least one frame of the frame sequence with the pixel image data, wherein the at least one motion vector indicates motion of at least part of the at least one object, and wherein the determining comprises either modifying or omitting a block matching search depending, at least in part, on the temporal display content information.

2. The method of claim 1 wherein the object data comprises at least: a first object position on a frame, and at least one of:
(1) coordinates of a second object position on another frame, and of the object having the first object position,
(2) x and y coordinates of the difference between the coordinates of the first and second object positions, and
(3) a motion direction value and a motion distance value from the first object position to the second object position.

3. The method of claim 1 wherein the object data comprises dimensions of the object and to be used to determine which pixels form the object.

4. The method of claim 1 wherein the object data comprises motion data of at least one object moved on a frame in response to a user entering motion commands via at least one of a mouse, keyboard, touchscreen, touchpad, track ball, pointing stick, game controller, microphone, and motion detector.

5. The method of claim 1 comprising wherein the object data comprises motion data of at least one object moved on a frame in response to an automatic input to an application that generated the automatic input and in response to at least one of: initial activation of the application, an input by the user, or an event on the display caused by the application.

6. The method of claim 1 wherein the object data is embedded in metadata of the frames and accompanies the frames as provided to the encoder.

7. The method of claim 1 wherein the encoder obtains the object data from an object data table or list in a memory.

8. The method of claim 1 comprising partitioning frames into blocks in order to encode the frames; determining which one or more of the blocks of a current frame form the at least one object; and individually providing the blocks that form the at least one object with a motion vector depending on the object data of the at least one object.

9. The method of claim 8 comprising forming the motion vector of a block without performing a block matching search on a reference frame.

10. The method of claim 9 comprising omitting the block matching when the object data comprises data that indicates which pixels form the object.

11. The method of claim 8 comprising modifying at least one parameter of a block matching search on a reference frame to determine a motion vector for the individual blocks.

12. The method of claim 11 wherein the search is modified when the object data does not provide an indication of the location of all pixels that form the object.

13. The method of claim 11 wherein the search is modified by at least one of:
reducing the area covered by the search,
reducing the number of sample points tested by the search, and
moving the search over or near a position indicated as an object position by the object data.

14. The method of claim 11 wherein the search is modified based, at least in part, on a hot zone near a pixel location on a frame that was selected by a user, wherein the dimensions of the hot zone are determined, at least in part, by an operating system or a display-based application providing the object data.

15. The method of claim 14 wherein the search is limited to within or near boundaries set as the hot zone boundaries.

16. The method of claim 1 comprising:
obtaining the object data based on user input formed by a user selecting at least one object on a hot zone on a display, wherein the hot zone is determined by an operating system and relative to the object on the display;
modifying a search for matching blocks between a reference frame and a current frame by limiting the search to near or within the hot zone;
determining a motion vector of individual blocks on the current frame at the hot zone and to be used to determine a prediction mode of the individual blocks, and
placing the motion vector in a bitstream with encoded data of the individual blocks and to be used to decode the encoded data.

17. A computer-implemented system comprising:
at least one display;
at least one memory to store frames of a video sequence and object data of at least one object on the frames;
at least one processor communicatively coupled to the at least one of the memory and display; and
an encoder operated by the at least one processor and operated by:
obtaining temporal display content information of image data of frames of a frame sequence and comprising object data of at least one object being moved from frame to frame along the sequence due to user or automatic input to an application providing the temporal display content information so that the temporal display content information is more than color-related or brightness-related pixel image data values alone, and wherein the at least one object is defined before encoding and the temporal display content information is formed before encoding and for at least one reason that is not solely for encoding;

obtaining pixel image data of the frames to encode the frame sequence; and encoding the pixel image data comprising determining at least one motion vector to be used to compress the pixel image data of the at least one object and to transmit at least one frame of the frame sequence with the pixel image data, wherein the at least one motion vector indicates motion of at least part of the at least one object, and wherein the determining comprises either modifying or omitting a block matching search depending, at least in part, on the temporal display content information.

18. The system of claim 17 wherein the object data comprises at least:
a first object position on a frame, and at least one of:
(1) coordinates of a second object position on another frame, and of the object having the first object position,
(2) x and y coordinates of the difference between the coordinates of the first and second object positions, and
(3) a motion direction value and a motion distance value from the first object position to the second object position.

19. The system of claim 17 wherein the object data comprises an identification of which frame the object is on, and shape data indicating the dimensions of the object sufficient to determine which pixels form the object.

20. The system of claim 17 wherein the object data comprises sufficient dimensions of the object to determine which pixels form t

Assignees

• Intel Corp

Inventors

• Lawrence Sean J
• Noraz Frederic J
• Boyce Jill M
• Mohan Sumit

Classifications

• H04N19/20

  using video object coding · CPC title

• H04N19/176

  the region being a block, e.g. a macroblock · CPC title

• H04N19/105

  Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction · CPC title

• H04N19/17

  the unit being an image region, e.g. an object · CPC title

• H04N19/51

  Motion estimation or motion compensation · CPC title