Method, electronic device, and computer program product for processing video

Patent metadata

Field	Value
Publication number	US-12299902-B2
Application number	US-202217668537-A
Country	US
Kind code	B2
Filing date	Feb 10, 2022
Priority date	Jan 21, 2022
Publication date	May 13, 2025
Grant date	May 13, 2025

Abstract

Embodiments of the present disclosure relate to a method, an electronic device, and a computer program product for processing a video. The method includes acquiring a video, where the video includes at least a current frame and a previous frame that are adjacent to each other. The method further includes determining, based on a first pixel value of a pixel in the current frame and a second pixel value of a corresponding pixel in the previous frame, whether the current frame has changed relative to the previous frame. The method further includes determining availability of the current frame for a computer vision task if it is determined that the current frame has changed relative to the previous frame. With the method, video data that needs to be processed is reduced, the task load of a computing device is lowered, system power consumption is improved, and data processing efficiency is improved.

First claim

What is claimed is:

1. A method comprising:
acquiring a video, in a first stage of a multi-stage video processing system executing on processor and memory resources of at least one computing device, the multi-stage video processing system comprising a plurality of separate sequential processing stages including the first stage and at least a second stage following the first stage, wherein the video comprises at least a current frame and a previous frame that are adjacent to each other;
determining, in the first stage of the multi-stage video processing system and based on a first pixel value of a pixel in the current frame and a second pixel value of a corresponding pixel in the previous frame, whether the current frame has changed relative to the previous frame, the determining of whether the current frame has changed relative to the previous frame being performed at least in part by at least one neural network of the first stage of the multi-stage video processing system;
determining, in the first stage of the multi-stage video processing system, availability of the current frame for a computer vision task that is implemented in the second stage of the multi-stage video processing system, by determining in the neural network of the first stage, utilizing pulse arrays generated as respective encodings of the first and second pixel values, whether or not the current frame has changed relative to the previous frame, wherein the current frame is determined as available for the computer vision task responsive to the current frame having changed relative to the previous frame, and the current frame is determined as not available for the computer vision task responsive to the current frame not having changed relative to the previous frame; and
controlling utilization of the second stage to perform the computer vision task for the current frame based on the determined availability of the current frame, wherein controlling utilization of the second stage to perform the computer vision task for the current frame comprises utilizing the second stage to perform the computer vision task for the current frame responsive to the current frame being determined as available, and bypassing utilization of the second stage to perform the computer vision task for the current frame responsive to the current frame being determined as not available;
wherein performance of the computer vision task implemented in the second stage is selectively bypassed for the current frame responsive to the controlling based on the determined availability of the current frame, to reduce a power consumption of the multi-stage video processing system in processing the video.

2. The method according to claim 1 , wherein determining whether the current frame has changed relative to the previous frame comprises:
acquiring the first pixel value and the second pixel value;
converting the first pixel value and the second pixel value into a first pulse array and a second pulse array; and
determining, responsive to determining that the first pulse array is different from the second pulse array, that the current frame has changed relative to the previous frame.

3. The method according to claim 2 , wherein determining whether the current frame has changed relative to the previous frame further comprises:
determining, responsive to determining that the first pulse array is the same as the second pulse array, that the current frame has not changed relative to the previous frame.

4. The method according to claim 1 , wherein determining whether the current frame has changed relative to the previous frame comprises:
inputting the first pixel value and the second pixel value to a change detection model to determine whether the current frame has changed relative to the previous frame, wherein the change detection model comprises a Spiking Neural Network (SNN) model trained to detect frame changes based on input adjacent video frames.

5. The method according to claim 1 , further comprising:
determining, responsive to determining that the current frame has not changed relative to the previous frame, that the current frame is not available for the computer vision task.

6. The method according to claim 1 , wherein determining the availability of the current frame for a computer vision task comprises:
detecting, responsive to determining that the current frame has changed relative to the previous frame, whether the current frame has a target object used for the computer vision task; and
determining, responsive to determining that the current frame has the target object, that the current frame is available for the computer vision task.

7. The method according to claim 6 , wherein determining the availability of the current frame for a computer vision task further comprises:
determining, responsive to determining that the current frame does not have the target object, that the current frame is not available for the computer vision task.

8. The method according to claim 1 , wherein determining the availability of the current frame for a computer vision task comprises:
determining, responsive to determining that the current frame has changed relative to the previous frame, that the current frame is available for the computer vision task.

9. An electronic device, comprising:
at least one processor; and
a memory coupled to the at least one processor and having instructions stored therein, wherein the instructions, when executed by the at least one processor, cause the electronic device to perform actions comprising:
acquiring a video, in a first stage of a multi-stage video processing system implemented by the at least one processor and the memory of the electronic device, the multi-stage video processing system comprising a plurality of separate sequential processing stages including the first stage and at least a second stage following the first stage, wherein the video comprises at least a current frame and a previous frame that are adjacent to each other;
determining, in the first stage of the multi-stage video processing system and based on a first pixel value of a pixel in the current frame and a second pixel value of a corresponding pixel in the previous frame, whether the current frame has changed relative to the previous frame, the determining of whether the current frame has changed relative to the previous frame being performed at least in part by at least one neural network of the first stage of the multi-stage video processing system;
determining, in the first stage of the multi-stage video processing system, availability of the current frame for a computer vision task that is implemented in the second stage of the multi-stage video processing system, by determining in the neural network of the first stage, utilizing pulse arrays generated as respective encodings of the first and second pixel values, whether or not the current frame has changed relative to the previous frame, wherein the current frame is determined as available for the computer vision task responsive to the current frame having changed relative to the previous frame, and the current frame is determined as not available for the computer vision task responsive to the current frame not having changed relative to the previous frame; and
controlling utilization of the second stage to perform the computer vision task for the current frame based on the determined availability of the current frame, wherein controlling utilization of the second stage to perform the computer vision task for the current frame comprises utilizing the second stage to perform the computer vision task for the current frame responsive to the current frame being determined as available, and bypassing utilization of the second stage to perform th

Assignees

• Dell Products Lp

Inventors

• Nie Yuhong
• Chen Tianxiang
• Li Sanping
• Hou Anzhou
• Jia Zhen

Classifications

• G06N3/049 Primary

  Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T7/20

  Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) · CPC title

• G06T2207/30196

  Human being; Person · CPC title