Virtual linebuffers for image signal processors

Patent metadata

Field	Value
Publication number	US-10516833-B2
Application number	US-201916376479-A
Country	US
Kind code	B2
Filing date	Apr 5, 2019
Priority date	Jan 22, 2015
Publication date	Dec 24, 2019
Grant date	Dec 24, 2019

Abstract

In a general aspect, an apparatus can include image processing logic (IPL) configured to perform an image processing operation on pixel data corresponding with an image having a width of W pixels and a height of H pixels to produce output pixel data in vertical slices of K pixels using K vertically overlapping stencils of S×S pixels, K being greater than 1 and less than H, S being greater than or equal to 2, and W being greater than S. The apparatus can also include a linebuffer operationally coupled with the IPL, the linebuffer configured to buffer the pixel data for the IPL. The linebuffer can include a full-size buffer having a width of W and a height of (S−1). The linebuffer can also include a sliding buffer having a width of SB and a height of K, SB being greater than or equal to S and less than W.

First claim

What is claimed is:

1. A processor comprising:
image processing logic configured to perform a kernel function on each of multiple partially overlapping stencils, each stencil being a two-dimensional region of image data of an image frame; and
a virtual linebuffer comprising a sliding buffer and a fixed buffer, wherein the fixed buffer has a height that is less than a height of the stencil and has a width that is at least a width of the image frame, wherein the sliding buffer has a height that is at least as tall as two stencils that partially overlap vertically and has a width that is at least as wide as two stencils that partially overlap horizontally and is less than a width of the image frame,
wherein the virtual linebuffer is configured to read data from the fixed buffer and the sliding buffer in order to provide multiple stencils that partially overlap horizontally and vertically to the image processing logic before updating data stored in the sliding buffer.

2. The processor of claim 1 , wherein the virtual linebuffer is configured to perform, for each slice of pixels, operations comprising:
storing a first portion of a first slice of a first slice row in the fixed buffer;
storing a second portion of the first slice in the sliding buffer; and
moving data within the first slice including overwriting the first portion of the first slice in the fixed buffer with a portion of the second portion of the first slice in the sliding buffer at a location in the fixed buffer where the first portion was stored after the first slice has been read, the moved data being data that is reused between slice rows that are adjacently processed in a sequence, each slice row comprising a plurality of rows of image data.

3. The processor of claim 2 , wherein moving the data within the first slice from the sliding buffer to the fixed buffer comprises moving a portion of data equal to an amount of overlap between stencils in a stencil slice.

4. The processor of claim 1 , wherein providing the multiple stencils that partially overlap horizontally and vertically comprises providing image data for four separate partially overlapping stencils to the image processing logic.

5. The processor of claim 1 , further comprising a consumer virtual linebuffer comprising a consumer sliding buffer and a consumer fixed buffer,
wherein the processor is configured to write a first portion of output of the kernel function to the consumer sliding buffer and a second portion of an output of the kernel function to the consumer fixed buffer.

6. The processor of claim 1 , wherein the virtual linebuffer is configured to support consecutive image processing stages having different stencil sizes.

7. The processor of claim 6 , wherein the consecutive image processing stages include a producer kernel and a consumer kernel, and wherein a first stencil size of the producer kernel is different than a second stencil size of the consumer kernel.

8. The processor of claim 7 , wherein output slices generated by the producer kernel are taller and narrower than input slices consumed by the consumer kernel.

9. The processor of claim 7 , wherein the virtual linebuffer is configured to provide an input slice of data to the consumer kernel only after multiple output slices of data have been written by the producer kernel.

10. A method performed by a processor comprising (i) image processing logic to perform a kernel function on each of multiple partially overlapping stencils, each stencil being a two-dimensional region of image data of an image frame, and (ii) a virtual linebuffer comprising a sliding buffer and a fixed buffer, wherein the fixed buffer has a height that is less than a height of the stencil and has a width that is at least a width of the image frame, wherein the sliding buffer has a height that is at least as tall as two stencils that partially overlap vertically and has a width that is at least as wide as two stencils that partially overlap horizontally and is less than a width of the image frame, the method comprising:
reading data from the fixed buffer and the sliding buffer;
providing multiple stencils that partially overlap horizontally and vertically to the image processing logic; and
after providing multiple stencils that partially overlap horizontally and vertically to the image processing logic, updating data stored in the sliding buffer.

11. The method of claim 10 , further comprising performing, by the image processing logic, a kernel function on each of the multiple stencils that partially overlap horizontally and vertically.

12. The method of claim 10 , further comprising writing, by the image processing logic, a result of the kernel function to a downstream virtual linebuffer.

13. The method of claim 12 , wherein the downstream virtual linebuffer is a consumer virtual linebuffer having a consumer sliding buffer and a consumer fixed buffer, and further comprising writing a first portion of an output of the kernel function to the consumer sliding buffer and a second portion of an output of the kernel function to the consumer fixed buffer.

14. The method of claim 10 , further comprising performing, by the virtual linebuffer for each slice of pixels, operations comprising:
storing a first portion of a first slice of a first slice row in the fixed buffer;
storing a second portion of the first slice in the sliding buffer; and
moving data within the first slice including overwriting the first portion of the first slice in the fixed buffer with a portion of the second portion of the first slice in the sliding buffer at a location in the fixed buffer where the first portion was stored after the first slice has been read, the moved data being data that is reused between slice rows that are adjacently processed in a sequence, each slice row comprising a plurality of rows of image data.

15. The method of claim 14 , wherein moving the data within the first slice from the sliding buffer to the fixed buffer comprises moving a portion of data equal to an amount of overlap between stencils in a stencil slice.

16. The method of claim 10 , wherein providing the multiple stencils that partially overlap horizontally and vertically comprises providing image data for four separate partially overlapping stencils to the image processing logic.

17. The method of claim 10 , wherein the virtual linebuffer is configured to support consecutive image processing stages having different stencil sizes.

18. The method of claim 17 , wherein the consecutive image processing stages include a producer kernel and a consumer kernel, and wherein a first stencil size of the producer kernel is different than a second stencil size of the consumer kernel.

19. The method of claim 18 , wherein output slices generated by the producer kernel are taller and narrower than input slices consumed by the consumer kernel.

20. The method of claim 18 , further comprising providing, by the virtual linebuffer, an input slice of data to the consumer kernel only after multiple output slices of data have been written by the producer kernel.

Assignees

• Google Llc

Inventors

• Zhu Qiuling
• Shacham Ofer
• Redgrave Jason Rupert
• Finchelstein Daniel Frederic
• Meixner Albert

Classifications

• G06T1/60 Primary

  Memory management · CPC title

• G06T1/20

  Processor architectures; Processor configuration, e.g. pipelining · CPC title

• H04N5/262 Primary

  Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects {; Cameras specially adapted for the electronic generation of special effects} · CPC title