Reinforcement-learning based system for camera parameter tuning to improve analytics

Patent metadata

Field	Value
Publication number	US-12347143-B2
Application number	US-202217825519-A
Country	US
Kind code	B2
Filing date	May 26, 2022
Priority date	Jun 3, 2021
Publication date	Jul 1, 2025
Grant date	Jul 1, 2025

Abstract

A method for automatically adjusting camera parameters to improve video analytics accuracy during continuously changing environmental conditions is presented. The method includes capturing a video stream from a plurality of cameras, performing video analytics tasks on the video stream, the video analytics tasks defined as analytics units (AUs), applying image processing to the video stream to obtain processed frames, filtering the processed frames through a filter to discard low-quality frames and dynamically fine-tuning parameters of the plurality of cameras. The fine-tuning includes passing the filtered frames to an AU-specific proxy quality evaluator, employing State-Action-Reward-State-Action (SARSA) reinforcement learning (RL) computations to automatically fine-tune the parameters of the plurality of cameras, and based on the reinforcement computations, applying a new policy for an agent to take actions and learn to maximize a reward.

First claim

What is claimed is:

1. A method for automatically adjusting camera parameters to improve video analytics accuracy during continuously changing environmental conditions, the method comprising:
capturing a video stream from a plurality of cameras;
performing video analytics tasks on the video stream, the video analytics tasks defined as analytics units (AUs);
applying image processing to the video stream to obtain processed frames;
filtering the processed frames through a filter to discard low-quality frames; and
dynamically fine-tuning parameters of the plurality of cameras by:
passing the filtered frames to an AU-specific proxy quality evaluator;
employing State-Action-Reward-State-Action (SARSA) reinforcement learning (RL) computations to automatically fine-tune the parameters of the plurality of cameras; and

based on the reinforcement computations, applying a new policy for an agent to take actions and learn to maximize a reward, wherein the SARSA RL computations include a state, an action, and a reward, the state being a vector including current brightness, contrast, sharpness and color parameter values of a camera of the plurality of cameras and a measure of brightness, contrast, sharpness and color, the action is an increase or decrease of one of the brightness, contrast, sharpness or color parameter values or taking no action at all, and the reward is a AU-specific quality evaluator's output.

2. The method of claim 1 , wherein a virtual-camera (VC) is employed to:
apply different environmental characteristics and evaluate different camera settings on an exact same scene captured from the video stream; and
rapidly and independently train and test RL algorithms, and various reward functions used for RL.

3. The method of claim 2 , wherein the VC includes an offline profiling phase and an online phase.

4. The method of claim 3 , wherein, during the offline profiling phase, a VC-camera table and a mapping function are generated, the mapping function mapping a particular time in a day to its corresponding brightness, contrast, color-saturation, and sharpness feature values observed during that time.

5. The method of claim 3 , wherein, during the online phase, a frame for a different environmental condition corresponding to a time of day other than its capture time is simulated.

6. The method of claim 1 , wherein an analytic quality (AQ) metric specific for each of the AUs is employed to train an AU-specific quality evaluator.

7. The method of claim 6 , wherein, in the absence of ground truth in real-world deployments, the AU-specific quality evaluator is used as a proxy to evaluate the AUs.

8. A non-transitory computer-readable storage medium comprising a computer-readable program for automatically adjusting camera parameters to improve video analytics accuracy during continuously changing environmental conditions, wherein the computer-readable program when executed on a computer causes the computer to perform the steps of:
capturing a video stream from a plurality of cameras;
performing video analytics tasks on the video stream, the video analytics tasks defined as analytics units (AUS);
applying image processing to the video stream to obtain processed frames;
filtering the processed frames through a filter to discard low-quality frames; and
dynamically fine-tuning parameters of the plurality of cameras by:
passing the filtered frames to an AU-specific proxy quality evaluator;
employing State-Action-Reward-State-Action (SARSA) reinforcement learning (RL) computations to automatically fine-tune the parameters of the plurality of cameras; and

based on the reinforcement computations, applying a new policy for an agent to take actions and learn to maximize a reward, wherein a virtual-camera (VC) is employed to:
apply different environmental characteristics and evaluate different camera settings on an exact same scene captured from the video stream; and
rapidly and independently train and test RL algorithms, and various reward functions used for RL
wherein the VC includes an offline profiling phase and an online phase such that during the online phase, a frame for a different environmental condition corresponding to a time of day other than its capture time is simulated.

9. The non-transitory computer-readable storage medium of claim 8 , wherein, during the offline profiling phase, a VC-camera table and a mapping function are generated, the mapping function mapping a particular time in a day to its corresponding brightness, contrast, color-saturation, and sharpness feature values observed during that time.

10. The non-transitory computer-readable storage medium of claim 8 , wherein an analytic quality (AQ) metric specific for each of the AUs is employed to train an AU-specific quality evaluator.

11. The non-transitory computer-readable storage medium of claim 10 , wherein, in the absence of ground truth in real-world deployments, the AU-specific quality evaluator is used as a proxy to evaluate the AUs.

12. A system for automatically adjusting camera parameters to improve video analytics accuracy during continuously changing environmental conditions, the system comprising:
a memory; and
one or more processors in communication with the memory configured to:
capture a video stream from a plurality of cameras;
perform video analytics tasks on the video stream, the video analytics tasks defined as analytics units (AUs);
apply image processing to the video stream to obtain processed frames;
filter the processed frames through a filter to discard low-quality frames; and
dynamically fine-tune parameters of the plurality of cameras by:
passing the filtered frames to an AU-specific proxy quality evaluator;
employing State-Action-Reward-State-Action (SARSA) reinforcement learning (RL) computations to automatically fine-tune the parameters of the plurality of cameras; and

based on the reinforcement computations, applying a new policy for an agent to take actions and learn to maximize a reward, wherein a virtual-camera (VC) is employed to:
apply different environmental characteristics and evaluate different camera settings on an exact same scene captured from the video stream; and
rapidly and independently train and test RL algorithms, and various reward functions used for RL, wherein the VC includes an offline profiling phase and an online phase such that during the online phase, a frame for a different environmental condition corresponding to a time of day other than its capture time is simulated.

13. The system of claim 12 , wherein, during the offline profiling phase, a VC-camera table and a mapping function are generated, the mapping function mapping a particular time in a day to its corresponding brightness, contrast, color-saturation, and sharpness feature values observed during that time.

Assignees

• Nec Lab America Inc
• Nec Corp

Inventors

• Rao Kunal
• Coviello Giuseppe
• Sankaradas Murugan
• Po Oliver
• Chakradhar Srimat
• Paul Sibendu

Classifications

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T7/0002

  Inspection of images, e.g. flaw detection · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title