Real-time low latency computer vision/machine learning compute accelerator with smart convolutional neural network scheduler

Patent metadata

Field	Value
Publication number	US-11816871-B2
Application number	US-202017138817-A
Country	US
Kind code	B2
Filing date	Dec 30, 2020
Priority date	Dec 30, 2020
Publication date	Nov 14, 2023
Grant date	Nov 14, 2023

Abstract

Methods and devices are provided for processing image data on a sub-frame portion basis using layers of a convolutional neural network. The processing device comprises memory and a processor. The processor is configured to receive frames of image data comprising sub-frame portions, schedule a first sub-frame portion of a first frame to be processed by a first layer of the convolutional neural network when the first sub-frame portion is available for processing, process the first sub-frame portion by the first layer and continue the processing of the first sub-frame portion by the first layer when it is determined that there is sufficient image data available for the first layer to continue processing of the first sub-frame portion. Processing on a sub-frame portion basis continues for subsequent layers such that processing by a layer can begin as soon as sufficient data is available for the layer.

First claim

What is claimed is:

1. An image processing method comprising:
receiving a plurality of frames of image data, the plurality of frames comprising sub-frame portions of image data;
scheduling a first sub-frame portion of a first frame to be processed by a first layer of a convolutional neural network when the first sub-frame portion is available for processing;
processing the first sub-frame portion by the first layer; and
continuing the processing of the first sub-frame portion by the first layer when it is determined that there is sufficient image data available for the first layer to continue processing of the first sub-frame portion.

2. The method of claim 1 , wherein the plurality of frames of image data are divided into a plurality of sub-frame portions, and
the sub-frame portions are one of slices and tiles.

3. The method of claim 1 , wherein the first sub-frame portion comprises data for a plurality of pixels, and
the first sub-frame portion is available for processing when data for each of the pixels of the first sub-frame portion is in memory.

4. The method of claim 1 , further comprising determining that there is sufficient image data available for the first layer to continue processing the first sub-frame portion when a second sub-frame portion is available for processing by the first layer.

5. The method of claim 4 , wherein the first layer of the convolutional neural network is a convolution layer, and
the processing of the first sub-frame portion by the convolutional layer is continued by convolving the first sub-frame portion and the second sub-frame portion.

6. The method of claim 1 , wherein the convolutional neural network comprises a plurality of different layers,
a sub-frame portion of the first frame is processed prior to any sub-frame portion of a second frame being processed, and
when the sub-frame portion of the first frame and a sub-frame portion of the second frame are concurrently available for processing by any of the different layers, the sub-frame portion of the first frame is scheduled for processing prior to scheduling the sub-frame portion of the second frame for processing.

7. The method of claim 6 , wherein when the sub-frame portion of the second frame is available for processing by the first layer, but no sub-frame portions of the first frame are available for processing by the first layer, the sub-frame portion of the second frame is processed by the first layer, and
after the processing of the sub-frame portion the second frame is completed and when an additional sub-frame portion of the first frame and an additional sub-frame portion of the second frame are concurrently available for processing, the additional sub-frame portion of the first frame is scheduled for processing by the first layer.

8. The method of claim 1 , wherein the convolutional neural network comprises a plurality of different layers which comprise a convolution layer, a max pooling layer and a rectified linear unit layer.

9. The method of claim 8 , further comprising:
processing the plurality of frames of image data on a sub-frame portion basis using the plurality of different layers; and
displaying the plurality of frames of image data.

10. An image processing device comprising:
memory; and
a processor configured to:
receive a plurality of frames of image data, the plurality of frames comprising sub-frame portions of image data;
schedule a first sub-frame portion of a first frame to be processed by a first layer of a convolutional neural network when the first sub-frame portion is available for processing;
process the first sub-frame portion by the first layer; and
continue the processing of the first sub-frame portion by the first layer when it is determined that there is sufficient image data available for the first layer to continue processing of the first sub-frame portion.

11. The image processing device of claim 10 , wherein the plurality of frames of image data are divided into a plurality of sub-frame portions, and
the sub-frame portions are one of slices and tiles.

12. The image processing device of claim 10 , wherein, wherein the first sub-frame portion comprises data for a plurality of pixels, and
the first sub-frame portion is available for processing when data for each of the pixels of the first sub-frame portion is in memory.

13. The image processing device of claim 10 , wherein the processor is configured to determine that there is sufficient image data available for the first layer to continue processing the first sub-frame portion when a second sub-frame portion is available for processing by the first layer.

14. The image processing device of claim 13 , wherein the first layer of the convolutional neural network is a convolution layer, and
the processing of the first sub-frame portion by the convolutional layer is continued by convolving the first sub-frame portion and the second sub-frame portion.

15. The image processing device of claim 10 , wherein the convolutional neural network comprises a plurality of different layers,
a sub-frame portion of the first frame is processed prior to any sub-frame portion of a second frame being processed, and
when the sub-frame portion of the first frame and a sub-frame portion of the second frame are concurrently available for processing by any of the different layers, the processor is configured to schedule the sub-frame portion of the first frame for processing prior to scheduling the sub-frame portion of the second frame for processing.

16. The image processing device of claim 15 , wherein when the sub-frame portion of the second frame is available for processing, but no sub-frame portions of the first frame are available for processing, the processor is configured to schedule the sub-frame portion of the second frame for processing by one of the different layers, and
after the processing of the sub-frame portion the second frame is completed and when an additional sub-frame portion of the first frame and an additional sub-frame portion of the second frame are concurrently available for processing, the processor is configured to schedule the additional sub-frame portion of the first frame for processing by one of the different layers.

17. The image processing device of claim 10 , wherein the convolutional neural network comprises a plurality of different layers which comprise a convolution layer, a max pooling layer and a rectified linear unit layer.

18. The image processing device of claim 17 , further comprising a display device,
wherein the plurality of frames of image data are processed on a sub-frame portion basis using the plurality of different layers; and
the plurality of frames of image data are displayed at the display device.

19. A non-transitory computer readable medium comprising instructions for causing a computer to execute a video encoding method comprising:
receiving a plurality of frames of image data, the plurality of frames comprising sub-frame portions of image data;
scheduling a first sub-frame portion of a first frame to be processed by a first layer of a convolutional neural network when the first sub-frame portion is available for processing;
processing the first sub-frame portion by the first layer; and
continuing the processing of the first sub-frame portion by the first layer when it is determined that there is sufficient image data available for the first layer to continue processing of the first sub-frame portion.

20. The non-transitory computer readable medium of claim 19 , wh

Assignees

• Advanced Micro Devices Inc
• Ati Technologies Ulc

Inventors

• Kwong Tung Chuen
• Porpino Sobreira Marques David
• Tam King Chiu
• Rajagopalan Shilpa
• Chan Benjamin Koon Pan
• Wu Vickie Youmin

Classifications

• G06T9/002 Primary

  using neural networks · CPC title

• G06N3/02

  Neural networks · CPC title

• G06T3/4046

  using neural networks · CPC title

• G01F1/66 Primary

  by measuring frequency, phase shift or propagation time of electromagnetic or other waves, e.g. using ultrasonic flowmeters · CPC title

• G01L9/0022

  of a piezoelectric element · CPC title