Techniques for context and performance adaptive processing in ultra low-power computer vision systems

Patent metadata

Field	Value
Publication number	US-2016173752-A1
Application number	US-201414566226-A
Country	US
Kind code	A1
Filing date	Dec 10, 2014
Priority date	Dec 10, 2014
Publication date	Jun 16, 2016
Grant date	—

Abstract

Techniques are disclosed to adaptively control an image sensor and image signal processor (ISP), collectively known as an image pipeline, such that a minimum amount of power is consumed by an electronic device while producing images that can be accurately analyzed by a computer vision application. The techniques disclosed herein can be implemented in various electronic devices capable of capturing and processing image data, such as, for example, smart phones, wearable computers, laptops, tablets, and other mobile computing or imaging systems. In an embodiment, the techniques disclosed herein are implemented in a system-on-chip device.

First claim

What is claimed is:

1 . An image processing system comprising:
an image sensor; an image signal processor communicatively coupled to the image sensor; a signal metrics processor communicatively coupled to the image sensor and the image signal processor, the signal metrics processor configured to measure a characteristic of captured image data; and a computer vision controller communicatively coupled to the signal metrics processor and configured to initiate capturing of image data in accordance with a selected computer vision task, wherein the computer vision controller adjusts a configuration parameter of at least one of the image signal processor and the image sensor based in part on the measured characteristic and the selected computer vision task.

2 . The image processing system of claim 1 , wherein the image sensor comprises a sensor configured to produce at least one of a color image signal (RGB), a color and depth image signal (RGBD), a YUV color image signal, and a stereo image signal (L/R RGB).

3 . The image processing system of claim 1 , wherein the measured characteristic comprises at least one of a sharpness value, a noise value, and a contrast value.

4 . The image processing system of claim 1 , wherein the selected computer vision task includes a corresponding computer vision task-specific configuration, the computer vision task-specific configuration comprising capture parameters including at least one of a resolution, a frame rate, and one or more post-capture image processing routines to apply.

5 . The image processing system of claim 1 , wherein the computer vision controller is configured to receive a performance indicator from a computer vision application during computer vision analysis of the captured image data.

6 . The image processing system of claim 5 , wherein the computer vision controller is configured to, in response to receiving a high confidence score from the computer vision application, store the measured characteristic in a best-case image signal profile for the selected computer vision task.

7 . The image processing system of claim 5 , wherein the computer vision controller is configured to, in response to receiving a low confidence score from the computer vision application, compare the measured characteristic to a corresponding metric in a best-case image signal profile for the selected computer vision task.

8 . The image processing system of claim 7 , wherein the computer vision controller determines if a difference between the measured characteristic and the corresponding metric exceeds a predetermined threshold, and wherein the computer vision controller is configured to attribute a signal deficiency as a cause for the low confidence score based on the difference.

9 . The image processing system of claim 8 , wherein the computer vision controller determines a corrective configuration such that the attributed signal deficiency is compensated for by the adjusted configuration parameter, wherein the corrective configuration comprises at least one of a change in shutter delay of the image sensor, a change in resolution of the image sensor, and a change in frame rate of the image sensor.

10 . The image processing system of claim 9 , wherein the corrective configuration further comprises at least one of enabling at least one post-processing routine in the image signal processor and disabling at least one post-processing routine in the image signal processor, and wherein the corrective configuration is selected based on a low-power profile.

11 . The image processing system of claim 9 , wherein the computer vision controller reprocesses previously captured image data using the corrective configuration.

12 . A computer-implemented method for image processing, the method comprising:
capturing image data by an image sensor in accordance with performance requirements of a selected computer vision task; measuring a characteristic of the captured image data; determining a difference between the measured characteristic and a corresponding metric; and adjusting a first configuration parameter of at least one of an image signal processor and the image sensor based in part on the difference.

13 . The method of claim 12 , further comprising comparing, in response to receiving a low confidence score from a computer vision application, the measured characteristic to a corresponding metric in a best-case image signal profile for the selected computer vision task.

14 . The method of claim 13 , further comprising attributing a signal deficiency as a cause for the low confidence score and determining a corrective configuration such that the attributed signal deficiency is compensated for by the adjusted first configuration parameter.

15 . The method of claim 12 , further comprising:
receiving motion estimation from a motion estimation module; and adjusting a second configuration parameter of at least one of the image signal processor and the image sensor based on the motion estimation.

16 . The method of claim 15 , wherein the second configuration parameter is a capture mode of the image sensor, and wherein adjusting the second configuration parameter comprises changing the capture mode of the image sensor to at least one of a burst-capture mode, a continuous-capture mode, and a single-shot capture mode.

17 . The method of claim 15 , wherein the second configuration parameter comprises one or more post-capture routines to apply to captured image data by the image signal processor, and wherein adjusting the second configuration parameter includes enabling an image stabilization routine.

18 . A non-transient computer program product encoded with instructions that when executed by one or more processors cause a process to be carried out, the process comprising:
capturing image data by an image sensor in accordance with performance requirements of a selected computer vision task; measuring a characteristic of the captured image data; determining a difference between the measured characteristic and a corresponding metric; and adjusting a first configuration parameter of at least one of an image signal processor and an image sensor based in part on the difference.

19 . The computer program product of claim 18 , wherein the selected computer vision task is dynamically changed in response detecting a different computer vision context, wherein the different computer vision context is determined based on recognizing at least one element in the captured image data.

20 . The computer program product of claim 19 , wherein the at least one recognized element comprises at least one of a human face, a character, a hand, an object, and a fiducial.

Assignees

• Intel Corp

Inventors

• Caviedes Jorge E
• Wu Sin Lin
• Zinevich Artem
• Ghenassia Frank
• Bitouk Dmitri

Classifications

• H04N23/651

  for reducing power consumption by affecting camera operations, e.g. sleep mode, hibernation mode or power off of selective parts of the camera · CPC title

• H04N23/80

  Camera processing pipelines; Components thereof · CPC title

• H04N1/00204 Primary

  with a digital computer or a digital computer system, e.g. an internet server (programmed control between transmitter and receiver or between image input and image output device H04N1/32561) · CPC title

• H04N2201/0084

  Digital still camera · CPC title

• H04N5/23203 Primary

  Electricity · mapped topic