NPU and decoder capable of transceiving feature map

Patent metadata

Field	Value
Publication number	US-12556726-B2
Application number	US-202418808652-A
Country	US
Kind code	B2
Filing date	Aug 19, 2024
Priority date	Jun 29, 2022
Publication date	Feb 17, 2026
Grant date	Feb 17, 2026

Abstract

A neural processing unit (NPU) for decoding video and/or feature map may include at least one processing element (PE) for an artificial neural network (ANN), the at least one PE to receive and decode a bitstream. The bitstream is received in units of frames, and one frame includes a weight for an ANN model, data of a base layer, and data of a plurality of enhancement layers. An NPU for encoding video and/or feature map may include at least one processing element (PE) for an artificial neural network (ANN), the at least one PE to encode an input video or feature map and to transmit the encoded input video or feature map as a bitstream. The at least one PE transmits the bitstream in units of frames, and one frame includes a weight for an ANN model, data of a base layer, and data of a plurality of enhancement layers.

First claim

What is claimed is:

1 . A neural processing unit (NPU) for processing a feature map and a weight, the NPU comprising:
a plurality of processing elements (PEs) for performing operations of a neural network, the plurality of PEs being configured to process a bitstream, wherein the bitstream is received through a network, wherein the received bitstream includes a first feature map, and wherein the received bitstream further includes a first weight applied to the first feature map during the operations of the neural network.

2 . The NPU of claim 1 ,
wherein the received bitstream further includes: a second feature map and a second weight, and wherein the second weight is applied to the second feature map.

3 . The NPU of claim 2 , wherein the first feature map relates to a base layer and the second feature map relates to at least one of a plurality of enhancement layers.

4 . The NPU of claim 3 , wherein at least a portion of the plurality of enhancement layers in the received bitstream is configured to be selectively processed.

5 . The NPU of claim 3 , wherein at least a portion of the plurality of enhancement layers is configured to be selectively processed according to an available bandwidth of a transmission channel of the received bitstream or according to a preset machine analysis task.

6 . The NPU of claim 3 , wherein the plurality of PEs are configured to selectively process at least a portion of the plurality of enhancement layers according to a preset machine analysis task.

7 . The NPU of claim 3 , wherein a number of the plurality of enhancement layers included in one frame is varied according to a condition of a transmission channel.

8 . The NPU of claim 3 , wherein the NPU is configured to determine a number of the plurality of enhancement layers included in one frame according to a condition of a transmission channel and feedback to an encoder.

9 . The NPU of claim 3 , wherein the plurality of enhancement layers are included in one frame in ascending order according to indexes of layers of the plurality of enhancement layers.

10 . A neural processing unit (NPU) for processing a feature map and a weight, the NPU comprising:
a plurality of processing elements (PEs) for generating the feature map to be transmitted as a bitstream, wherein the bitstream is transmitted through a network, wherein the transmitted bitstream includes: a first feature map, and wherein the transmitted bitstream further includes a first weight, which is applied to the first feature map by a decoder during performing operations of a neural network.

11 . The NPU of claim 10 ,
wherein the transmitted bitstream further includes: a second feature map and a second weight, and wherein the second weight is applied to the second feature map.

12 . The NPU of claim 11 , wherein the first feature map relates to a base layer and the second feature map relates to at least one of a plurality of enhancement layers.

13 . The NPU of claim 12 , wherein a number of the plurality of enhancement layers of the bitstream is adjusted according to an available bandwidth of a transmission channel or adjusted for at least one frame interval.

14 . The NPU of claim 12 , wherein the plurality of PEs are configured to selectively process at least a portion of the plurality of enhancement layers according to a preset machine analysis task.

15 . The NPU of claim 12 , wherein the plurality of PEs are configured to process the base layer, a first enhancement layer, and a second enhancement layer according to a second machine analysis task.

16 . The NPU of claim 12 , wherein a number of the plurality of enhancement layers included in one frame is varied according to a condition of a transmission channel.

17 . The NPU of claim 12 , wherein the NPU is configured to receive feedback on a number of the plurality of enhancement layers included in one frame from a decoder.

18 . The NPU of claim 12 , wherein the plurality of enhancement layers are included in one frame in ascending order according to indexes of layers of the plurality of enhancement layers.

19 . A decoder for processing a feature map and a weight, the decoder comprising:
a plurality of processing elements (PEs) for performing operations of an neural network, the plurality of PEs being configured to process a bitstream, wherein the bitstream is received through a network, wherein the received bitstream includes a first feature map, and wherein the received bitstream further includes a first weight, which is applied to the first feature map during the operations of the neural network.

20 . The decoder of claim 19 ,
wherein the received bitstream further includes: a second feature map and a second weight, and wherein the second weight is applied to the second feature map.

Assignees

• Deepx Co Ltd

Inventors

• Im Iimyeong
• Lee Sunmi

Classifications

• G06N3/063 Primary

  using electronic means · CPC title

• H04N19/172

  the region being a picture, frame or field · CPC title

• H04N19/30

  using hierarchical techniques, e.g. scalability (H04N19/63 takes precedence) · CPC title

• G06N3/0464 Primary

  Convolutional networks [CNN, ConvNet] · CPC title

• H04N19/80

  Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation (H04N19/635, H04N19/86 take precedence) · CPC title