Enhanced computer vision application programming interface

Patent metadata

Field	Value
Publication number	US-12530732-B2
Application number	US-202318178059-A
Country	US
Kind code	B2
Filing date	Mar 3, 2023
Priority date	Apr 26, 2022
Publication date	Jan 20, 2026
Grant date	Jan 20, 2026

Abstract

An image processing system includes one or more processors operative to receive a graph application programming interface (API) call to add a complex node to a graph. The graph includes at least the complex node connected to other nodes by edges that are directed and acyclic. The one or more processors are further operative to process, by a graph compiler at compile time, the complex node by iteratively expanding the complex node into multiple nodes with each node corresponding to one operation in an image processing pipeline. The system further includes one or more target devices to execute executable code compiled from each node to perform operations of the image processing pipeline. The system further includes memory to store the graph compiler and the executable code.

First claim

What is claimed is:

1 . A method for image processing, comprising:
receiving a graph application programming interface (API) call that specifies a complex node representing a multi-layered neural network model used in an image processing pipeline; adding the complex node to a graph that contains other nodes representing computer vision operations, wherein the graph is directed and acyclic; attaching a buffer to the complex node to store code and parameters of the complex node; generating, by a graph compiler processing contents of the buffer at compile time, a sequence of nodes from the complex node to add to the graph, each node in the sequence corresponding to one operation layer of the neural network model; and executing executable code compiled from each node in the graph on one or more target devices to perform computer vision operations and neural network operations in the image processing pipeline.

2 . The method of claim 1 , further comprising:
adding to the graph another complex node that represents a subgraph of nodes corresponding to a plurality of computer vision operations.

3 . The method of claim 1 , further comprising:
adding to the graph another complex node that corresponds to customer-defined code for image processing.

4 . The method of claim 1 , wherein operations corresponding to the complex node are selected from a first library of functions, and operations corresponding to the other nodes in the graph are selected from a second library of functions, and wherein the first library and the second library are based on different programming models optimized for different types of operations.

5 . The method of claim 4 , wherein the first library of functions is provided by a deep learning framework, and the second library of functions is provided by a computer vision framework.

6 . The method of claim 1 , wherein the graph API call identifies a graph-based programming model used by operations corresponding to the complex node.

7 . The method of claim 1 , further comprising:
receiving another graph API call that identifies another complex node as one of following: a proprietary computer vision (CV) node, a customer-defined node, a neural network model node, and a VX graph node based on OpenVX.

8 . A system for image processing, comprising:
one or more processors to:
receive a graph application programming interface (API) call that specifies a complex node representing a multi-layered neural network model used in an image processing pipeline;
add a complex node to a graph that contains other nodes representing computer vision operations, wherein the graph is directed and acyclic;
attach a buffer to the complex node to store code and parameters of the complex node; and
generate, by a graph compiler processing contents of the buffer at compile time, a sequence of nodes from the complex node to add to the graph, each node in the sequence corresponding to one operation layer of the neural network model;
one or more target devices to execute executable code compiled from each node in the graph to perform computer vision operations and neural network operations in the image processing pipeline; and memory to store the graph compiler and the executable code.

9 . The system of claim 8 , wherein the one or more processors are further operative to add to the graph another complex node that represents a subgraph of nodes corresponding to a plurality of computer vision operations.

10 . The system of claim 8 , wherein the one or more processors are further operative to add to the graph another complex node that corresponds to customer-defined code for image processing.

11 . The system of claim 8 , wherein operations corresponding to the complex node are selected from a first library of functions, and operations corresponding to the other nodes in the graph are selected from a second library of functions, and wherein the first library and the second library are based on different programming models optimized for different types of operations.

12 . The system of claim 11 , wherein the first library of functions is optimized for deep learning, and the second library of functions is optimized for computer vision.

13 . The system of claim 8 , wherein the graph API call identifies a graph-based programming model used by operations corresponding to the complex node.

14 . The system of claim 8 , wherein the one or more processors are further operative to receive another graph API call that identifies another complex node as one of following: a proprietary computer vision (CV) node, a customer-defined node, a neural network model node, and a VX graph node based on OpenVX.

Assignees

• Mediatek Inc

Inventors

• Lin Yu-Chieh
• Liu Hungchun
• Jeng Po-Yuan
• Chiu Yungchih
• Chang Chia-Yu
• Hsieh Cheng-Hsun
• Chen Lei
• Chen Li-Ming
• Wang Taichi

Classifications

• G06T11/26

  Drawing of charts or graphs · CPC title

• G06F8/36

  Software reuse · CPC title

• G06T7/00

  Image analysis · CPC title

• G06T1/00

  General purpose image data processing · CPC title

• G06T2207/20072

  Graph-based image processing · CPC title