Image stream pipeline controller for deploying image primitives to a computation fabric

Patent metadata

Field	Value
Publication number	US-9378534-B2
Application number	US-201113994013-A
Country	US
Kind code	B2
Filing date	Dec 28, 2011
Priority date	Dec 28, 2011
Publication date	Jun 28, 2016
Grant date	Jun 28, 2016

Abstract

According to some embodiments, an image pipeline controller may determine an image stream having a plurality of image primitives to be executed. Each image primitive may be, for example, associated with an image algorithm and a set of primitive attributes. The image pipeline controller may then automatically deploy the set of image primitives to an image computation fabric based at least in part on primitive attributes.

First claim

What is claimed is:

1. A method, comprising: determining, at an image pipeline controller, an image stream comprising a plurality of image primitives associated with different types of image algorithms to be executed for processing image data, each image primitive used by an imaging application that provides processing to create output image data wherein the plurality of image primitives are associated with an image primitive library including at least one of: (i) a sensor primitive; (ii) a calibration primitive, (iii) an optics primitive, (iv) a lighting primitive, (v) a segmentation primitive or (vi) a filter primitive; and
automatically deploying, by the image pipeline controller, the plurality of image primitives to an image computation fabric based at least in part on primitive attributes that describe aspects of an operation of the plurality of image primitives.

2. The method of claim 1 , wherein the image computation fabric comprises a plurality of image processing components, including at least one of: (i) a fixed function hardware image processing unit, (ii) a single instruction, multiple data image execution unit, (iii) a very long instruction word processing unit, or (iv) a general processing unit.

3. The method of claim 2 , further comprising:
executing a workload distribution algorithm to select an image processing component to receive one of the image primitives in the image stream.

4. The method of claim 3 , wherein the workload distribution algorithm selects one of a fixed function hardware image processing unit or a software emulation of the fixed function hardware image processing unit based on primitive attributes and an image processing component status.

5. The method of claim 4 , wherein said selection is further based on at least one of: (i) a power and performance policy, (ii) resource reservation priorities, (iii) pipeline priorities, or (iv) resource availability arbitration priorities.

6. The method of claim 3 , wherein the workload distribution algorithm reduces stalls and optimizes for power or performance associated with execution of the image primitives in the image computation fabric.

7. The method of claim 1 , wherein the image pipeline controller is associated with at least one of: (i) a hardware image pipeline controller, (ii) a software image pipeline controller, or (iii) a combination of hardware and software image pipeline controller components.

8. The method of claim 1 , wherein image primitives associated with a plurality of image streams are deployed.

9. The method of claim 8 , wherein different image streams are associated with different applications being executed by an operating system, and information about the image streams are associated with an application programming interface.

10. The method of claim 8 , further comprising:
executing a load-balancing algorithm between image primitives in different image streams.

11. The method of claim 1 , further comprising:
executing a sequencing algorithm to order the image primitives within the image stream for an in-order image primitive execution in a pipeline sequence.

12. The method of claim 11 , wherein the image primitives are associated with an original order and said executing is performed for at least some of the image primitives in an order different than the original order for an out-of-order image primitive execution in a pipeline sequence.

13. The method of claim 1 , wherein a subset of the image primitives are associated with an image stream segment.

14. The method of claim 13 , wherein an image stream segment is associated with at least one image stream attribute for: (i) workload distribution, (ii) stall reduction, (iii) power optimization, (iv) performance optimization, (v) load balancing, or (vi) a sequencing algorithm.

15. The method of claim 1 , wherein the image stream is associated with a set of image data, and further comprising:
determining whether a tile subset of the image data is to be deployed to the image computation fabric based at least in part on a primitive attribute.

16. The method of claim 1 , wherein the image attributes are defined by a developer and stored within a primitive attribute database.

17. The method of claim 16 , wherein at least one primitive attribute is associated with at least one of: (i) a number of computation units, (ii) a performance value, (iii) a power value, (iv) a thermal value, or (v) a rank preference.

18. The method of claim 1 , wherein the image stream is associated with at least one of: (i) a camera sensor, (ii) a video sensor, (iii) a display screen, (iv) a depth sensor or stereo image pair, or (v) a three dimensional image.

19. The method of claim 1 , wherein at least one image algorithm is associated with at least one of: (i) pixel correction, (ii) artifact removal, (iii) histogram information, (iv) a scaling function, (v) face recognition, (vi) visual object recognition, (vii) visual scene analysis, (viii) machine vision, (ix) gesture recognition, or (x) depth map calculation.

20. A non-transitory computer-readable storage medium having stored thereon instructions that when executed by a machine result in the following:
determining, at an image pipeline controller, an image stream comprising a plurality of image primitives associated with different types of image algorithms to be executed, each image primitive used by an imaging application that provides processing to create output image data wherein the plurality of image primitives are associated with an image primitive library including at least one of: (i) a sensor primitive; (ii) a calibration primitive, (iii) an optics primitive, (iv) a lighting primitive, (v) a segmentation primitive or (vi) a filter primitive; and
automatically deploying, by the image pipeline controller, the plurality of image primitives to an image computation fabric based at least in part on primitive attributes that describe aspects of an operation of the plurality of image primitives.

21. The medium of claim 20 , wherein the image computation fabric comprises a plurality of image processing components, including at least one of: (i) a fixed function hardware image processing unit, (ii) a single instruction, multiple data image execution unit, (iii) a very long instruction word processing unit, or (iv) a general processing unit.

22. The medium of claim 21 , further comprising: executing a workload distribution algorithm to select an image processing component to receive one of the image primitives in the image stream.

23. An apparatus, comprising:
an image computation fabric including a plurality of image processing components; and an image pipeline controller comprising a processor to: (a) determine an image stream comprising a plurality of image primitives associated with different types of image algorithms to be executed, each image primitive used by an imaging application that provides processing to create output image data wherein the plurality of image primitives are associated with an image primitive library including at least one of: (i) a sensor primitive; (ii) a calibration primitive, (iii) an optics primitive, (iv) a lighting primitive, (v) a segmentation primitive or (vi) a filter primitive, and (b) deploy the plurality of image primitives to the image computation fabric based at least in part on primitive attributes that describe aspects of an operation of the plurality of image primitives.

24. The apparatus of claim 23 , wherein subsets of the i

Assignees

• Krig Scott A
• Taylor Stewart N
• Intel Corp

Inventors

• Krig Scott A
• Taylor Stewart N

Classifications

• G06T7/0083

  Physics · mapped topic

• G06T1/20 Primary

  Processor architectures; Processor configuration, e.g. pipelining · CPC title

• G06K9/48

  Physics · mapped topic

• G06T15/005

  General purpose rendering architectures · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title