Saliency-weighted video quality assessment

Patent metadata

Field	Value
Publication number	US-10699396-B2
Application number	US-201815886776-A
Country	US
Kind code	B2
Filing date	Feb 1, 2018
Priority date	Nov 30, 2015
Publication date	Jun 30, 2020
Grant date	Jun 30, 2020

Abstract

Systems and methods are disclosed for weighting the image quality prediction of any visual-attention-agnostic quality metric with a saliency map. By accounting for the salient regions of an image or video frame, the disclosed systems and methods may dramatically improve the precision of the visual-attention-agnostic quality metric during image or video quality assessment. In one implementation, a method of saliency-weighted video quality assessment includes: determining a per-pixel image quality vector of an encoded video frame; determining per-pixel saliency values of the encoded video frame or a reference video frame corresponding to the encoded video frame; and computing a saliency-weighted image quality metric of the encoded video frame by weighting the per-pixel image quality vector using the per-pixel saliency values.

First claim

What is claimed is:

1. A method comprising:
receiving an encoded video;
determining a saliency-weighted image quality metric for each video frame of a plurality of video frames of the encoded video;
determining video frames of the plurality of video frames of the encoded video with a respective saliency-weighted image quality metric that is below a threshold; and
assembling a playlist including the video frames of the encoded video with the respective saliency-weighted image quality metric that is below the threshold.

2. The method of claim 1 , wherein determining the respective saliency-weighted image quality metric for a given video frame of the plurality of video frames of the encoded video comprises:
determining a per-pixel image quality vector of the given video frame;
determining per-pixel saliency values of the given video frame; and
computing the respective saliency-weighted image quality metric of the given video frame based on the per-pixel image quality vector and the per-pixel saliency values.

3. The method of claim 2 , wherein the respective saliency-weighted image quality metric of the given video frame is determined based on the equation:

q
^

=

1
N

⁢

(

q
·

s
w

)

,

wherein {circumflex over (q)} is the respective saliency-weighted image quality metric of the given video frame, (·) denotes an inner product, q is the per-pixel image quality vector of the given video frame, s is a vector of the per-pixel saliency values of the given video frame, w is a weight given to the vector s, and N is a number of pixels in the given video frame.

4. The method of claim 2 , wherein determining the per-pixel image quality vector of the given video frame comprises comparing the given video frame with a reference video frame.

5. The method of claim 2 , further comprising: displaying color-coded per-pixel visible difference visualization between the encoded video and a reference video.

6. The method of claim 5 , wherein the color-coded per-pixel visible difference visualization is displayed for the given video frame of the encoded video using at least the per-pixel image quality vector and the per-pixel saliency values determined for the given video frame.

7. The method of claim 5 , further comprising: providing a graphical user interface including one or more controls for selecting one of a plurality of video quality assessment methods to display the color-coded per-pixel visible difference visualization.

8. The method of claim 5 , wherein the color-coded per pixel visible difference visualization displays no color to denote no visible artifacts or color to denote visible artifacts.

9. The method of claim 1 , further comprising: displaying a quality prediction plot of the encoded video that provides video quality normalized as a function of time code or video frame number of the encoded video.

10. The method of claim 1 , wherein assembling the playlist comprises:
marking time codes or frame numbers of the video frames of the encoded video with the respective saliency-weighted image quality metric below the threshold; and
assembling the marked time codes or frame numbers into the playlist.

11. A system, comprising:
a non-transitory computer-readable medium operatively coupled to a processor and having instructions stored thereon that, when executed by the processor, cause the system to:
receive an encoded video;
determine a saliency-weighted image quality metric for each video frame of a plurality of video frames of the encoded video;
determine video frames of the plurality of video frames of the encoded video with a respective saliency-weighted image quality metric that is below a threshold; and
assemble a playlist including the video frames of the encoded video with the respective saliency-weighted image quality metric that is below the threshold.

12. The system of claim 11 , wherein determining the respective saliency-weighted image quality metric for a given video frame of the plurality of video frames of the encoded video comprises:
determining a per-pixel image quality vector of the given video frame;
determining per-pixel saliency values of the given video frame; and
computing the respective saliency-weighted image quality metric of the given video frame based on the per-pixel image quality vector and the per-pixel saliency values.

13. The system of claim 12 , wherein the respective saliency-weighted image quality metric of the given video frame is determined based on the equation:

q
^

=

1
N

⁢

(

q
·

s
w

)

,

wherein {circumflex over (q)} is the respective saliency-weighted image quality metric of the given video frame, (·) denotes an inner product, q is the per-pixel image quality vector of the given video frame, s is a vector of the per-pixel saliency values of the given video frame, w is a weight given to the vector s, and N is a number of pixels in the given video frame.

14. The system of claim 12 , wherein determining the per-pixel image quality vector of the given video frame comprises comparing the given video frame with a reference video frame.

15. The system of claim 12 , wherein the instructions, when executed by the processor, further cause the system to: display color-coded per-pixel visible difference visualization between the encoded video and a reference video.

16. The system of claim 15 , wherein the color-coded per-pixel visible difference visualization is displayed for the given video frame of the encoded video using at least the per-pixel image quality vector and the per-pixel saliency values determined for the given video frame.

17. The system of claim 15 , wherein the instructions, when executed by the processor, further cause the system to: provide a graphical user interface including one or more controls for selecting one of a plurality of video quality assessment methods to display the color-coded per-pixel visible difference visualization.

18. The system of claim 15 , wherein the color-coded per pixel visible difference visualization displays no color to denote no visible artifacts or color to denote visible artifacts.

19. The system of claim 11 , wherein the instructions, when executed by the processor, further cause the system to: display a quality prediction plot of the encoded video that provides video quality normal

Assignees

• Disney Entpr Inc

Inventors

• Aydin Tunc Ozan
• Stefanoski Nikolce
• Smolic Aljoscha
• Arana Mark

Classifications

• H04N17/004

  for digital television systems · CPC title

• H04N19/85

  using pre-processing or post-processing specially adapted for video compression · CPC title

• H04N19/00

  Methods or arrangements for coding, decoding, compressing or decompressing digital video signals · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T7/0002 Primary

  Inspection of images, e.g. flaw detection · CPC title