Scalable video coding for machine

Patent metadata

Field	Value
Publication number	US-2024056588-A1
Application number	US-202318486398-A
Country	US
Kind code	A1
Filing date	Oct 13, 2023
Priority date	Jun 29, 2022
Publication date	Feb 15, 2024
Grant date	—

Abstract

A neural processing unit (NPU) for decoding video or feature map is provided. The NPU may include at least one processing element (PE) for an artificial neural network, the at least one PE configured to receive and decode data included in a bitstream. The data included in the bitstream may include data of a base layer; or the data of the base layer and data of at least one enhancement layer. An NPU for encoding video or feature map is also provided. The encoder NPU may include at least one PE for an artificial neural network, the at least one PE configured to receive and encode a transmitted video or feature map, wherein the at least one PE may be further configured to output a bitstream including data of a base layer and data of at least one enhancement layer.

First claim

What is claimed is:

1 . A neural processing unit (NPU) for processing feature maps, the NPU comprising:
a first circuitry arranged as at least one processing element (PE) for performing operations of an artificial neural network, by using data included in a received bitstream, wherein the data included in the received bitstream comprises:
data of a base layer; or
the data of the base layer and data of at least one enhancement layer,
wherein the data of the base layer includes a first feature map used for a first machine analysis task, and wherein the data of the at least one enhancement layer includes a second feature map used for a second machine analysis task.

2 . The NPU of claim 1 , wherein at least a portion of the at least one enhancement layer of the received bitstream is configured to be selectively processed.

3 . The NPU of claim 1 , wherein at least a portion of the at least one enhancement layer is configured to be selectively processed according to an available bandwidth of a transmission channel of the received bitstream or according to a preset machine analysis task.

4 . The NPU of claim 1 , wherein
wherein the first and second feature maps in the received bitstream have been extracted at any intermediate layer of an artificial neural network model.

5 . The NPU of claim 1 , wherein an available bandwidth of a transmission channel of the received bitstream is configured to be detected.

6 . The NPU of claim 1 , wherein the at least one PE is configured to selectively process at least a portion of the at least one enhancement layer according to a preset machine analysis task.

7 . The NPU of claim 1 , wherein the at least one PE is configured to process the base layer and a first enhancement layer according to the first machine analysis task.

8 . The NPU of claim 1 , wherein the at least one PE is configured to process the base layer, a first enhancement layer, and the second enhancement layer according to the second machine analysis task.

9 . The NPU of claim 1 , wherein a number of the at least one enhancement layer included in one frame is varied according to a condition of a transmission channel.

10 . The NPU of claim 1 , wherein a number of the at least one enhancement layer included in one frame is determined according to a condition of a transmission channel and feedback information on the determined number is transmitted to an encoder.

11 . The NPU of claim 1 , wherein the at least one enhancement layer is included in one frame in ascending order according to indexes of layers of the at least one enhancement layer.

12 . A neural processing unit (NPU) for encoding feature maps, the NPU comprising:
a first circuitry arranged as at least one processing element (PE) for performing operations of an artificial neural network thereby outputting one or more feature maps, wherein the one or more feature maps are packed into a bitstream, wherein the bitstream includes data of a base layer; or the data of the base layer and data of at least one enhancement layer, wherein the data of the base layer includes a first feature map used for a first machine analysis task, and wherein the data of the at least one enhancement layer includes a second feature map used for a second machine analysis task.

13 . The NPU of claim 12 , wherein a number of the at least one enhancement layer of the bitstream is adjusted according to an available bandwidth of a transmission channel.

14 . The NPU of claim 12 , wherein a number of the at least one enhancement layer of the bitstream is adjusted for at least one frame interval.

15 . The NPU of claim 12 , wherein the first and second feature maps in the received bitstream are extracted at any intermediate layer of an artificial neural network model.

16 . The NPU of claim 12 , wherein a number of the at least one enhancement layer included in one frame is varied according to a condition of a transmission channel.

17 . The NPU of claim 12 , wherein the NPU is configured to receive feedback on a number of at least one enhancement layer included in one frame from a decoder.

18 . The NPU of claim 12 , wherein the at least one enhancement layer is included in one frame in ascending order according to indexes of layers of the at least one enhancement layer.

19 . A VCM decoder for processing feature maps, the VCM decoder comprising:
a first circuitry arranged as at least one processing element (PE) for performing operations of an artificial neural network, by using data included in a received bitstream, wherein the data included in the received bitstream comprises:
data of a base layer, or
the data of the base layer and data of at least one enhancement layer,
wherein the data of the base layer includes used for a first machine analysis task, and wherein the data of the at least one enhancement layer includes a second feature map used for a second machine analysis task.

20 . The VCM decoder of claim 19 , wherein the first and second feature maps in the received bitstream have been extracted at any intermediate layer of an artificial neural network model.

Assignees

• Deepx Co Ltd

Inventors

• Im Ilmyeong
• Lee Sunmi

Classifications

• H04N19/42 Primary

  characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation (H04N19/635 takes precedence) · CPC title

• H04N19/30 Primary

  using hierarchical techniques, e.g. scalability (H04N19/63 takes precedence) · CPC title

• H04N19/184 Primary

  the unit being bits, e.g. of the compressed video stream · CPC title

• H04N19/70

  characterised by syntax aspects related to video coding, e.g. related to compression standards · CPC title

• G06N20/20

  Ensemble learning · CPC title