Continuous integration and continuous delivery of artificial intelligence machine learning components using metamorphic relations

Patent metadata

Field	Value
Publication number	US-2024168748-A1
Application number	US-202218057001-A
Country	US
Kind code	A1
Filing date	Nov 18, 2022
Priority date	Nov 18, 2022
Publication date	May 23, 2024
Grant date	—

Abstract

A system for CI/CD of AI/ML based components includes a cloud infrastructure which receives multiple new requirements from a vehicle designer or a developer, with the new requirements adapted for artificial intelligence/machine learning (AI/ML) based components of a vehicle. A dataset is provided. A metamorphic relations (MR) module receives input information from the dataset and sends MR information to the dataset. A components requirements database includes the new requirements in addition to existing requirements for the AI/ML based components. The MR module also receives components requirements data from the components requirements database and sends the MR information to the components requirements database. An AI/ML algorithm analyzes the input information from the dataset and prepares an updated component dataset.

First claim

What is claimed is:

1 . A system for (continuous integration continuous deployment) CI/CD of (artificial intelligence/machine learning) AI/ML based components, comprising:
a cloud infrastructure which receives multiple new requirements from a vehicle designer or a developer, with the new requirements adapted for artificial intelligence/machine learning (AI/ML) based components of a vehicle; a dataset; a metamorphic relations (MR) module which receives input information from the dataset and sends MR information to the dataset; a components requirements database which includes the new requirements in addition to existing requirements for the AI/ML based components; the MR module also receiving components requirements data from the components requirements database and sending the MR information to the components requirements database; and an AI/ML algorithm which analyzes the input information from the dataset and prepares an updated component dataset.

2 . The system for CI/CD of AI/ML based components of claim 1 , further including multiple test cases data also received from and sent to the MR module.

3 . The system for CI/CD of AI/ML based components of claim 2 , including a vehicle requirements database having multiple edge cases from multiple vehicles.

4 . The system for CI/CD of AI/ML based components of claim 1 , including a cloud-based architecture supporting CI/CD of the AI/ML based components.

5 . The system for CI/CD of AI/ML based components of claim 4 , including a requirement capturing module (RCM) receiving the new requirements as input from the vehicle designer or the developer.

6 . The system for CI/CD of AI/ML based components of claim 5 , including an asset refinement engine (ARE) which receives refined requirements from the RCM.

7 . The system for CI/CD of AI/ML based components of claim 6 , including an edge case capturing component (ECC) which receives requirements data from a vehicle requirements database and generates inputs to the AI/ML based components for situations when the AI/ML based components may perform poorly due to images in low light conditions, images with overlapping traffic participants, or images with long shadows or degraded lane markers, with the inputs to the ECC from the vehicle requirements database being generated directly from real-life scenarios.

8 . The system for CI/CD of AI/ML based components of claim 7 , including an AI/ML development platform (ADP) receiving an output of the ARE and from the dataset, the ADP being used for all AI/ML based components; and wherein an output from the ADP is forwarded to the AI/ML algorithms.

9 . The system for CI/CD of AI/ML based components of claim 6 , wherein the ARE: employs processes and methods for validation, generation, and refinement of requirements, datasets and test cases; provides a process and method for simultaneous refinement of requirements, datasets and test cases; provides representation of requirements as metamorphic relations (MRs), and extraction of MRs from the requirements, datasets and test cases; and converts MRs as satisfiability modulo theory (SMT) formulae; and wherein an output of the ARE is forwarded to the dataset.

10 . The system for CI/CD of AI/ML based components of claim 1 , including a vehicle requirements database, wherein the updated component dataset is forwarded as an over-the-air (OTA) update to the vehicle requirements database.

11 . A method for continuous integration continuous deployment (CI/CD) of artificial intelligence/machine learning (AI/ML) based components using a cloud-based architecture, comprising:
performing CI/CD of AI/ML component development; entering input data from a vehicle requirements database into an edge case capturing component (ECC) and applying the ECC to generate inputs to the AI/ML component development; inputting design data as design requirements from a vehicle designer or developer into a requirements capturing module (RCM); providing the design requirements from the RCM to an asset refinement engine (ARE) to validate, generate, and refine the design requirements; and analyzing data retrieved from a dataset and preparing an updated component dataset using an AI/ML algorithm.

12 . The method of claim 11 , further including providing representation of the design requirements as metamorphic relations (MRs) using the ARE.

13 . The method of claim 12 , further including extracting the MRs from the design requirements using the ARE.

14 . The method of claim 13 , further including converting the MRs as satisfiability modulo theory (SMT) formulae using the ARE.

15 . The method of claim 11 , further including operating a metamorphic relations module receiving data from and sending data to a components requirements database to check consistency of behaviors across variations and to generate new test cases.

16 . The method of claim 11 , further including sending an output of the ARE and data from the dataset to an AI/ML development platform (ADP).

17 . The method of claim 11 , further including operating the ECC to perform a pre-analysis and filter the inputs to the AI/ML based components.

18 . A method to continuously integrate/continuously deploy artificial intelligence/machine learning based components, comprising:
performing continuous integration/continuous deployment (CI/CD) of artificial intelligence/machine learning (AI/ML) component development; entering input data from a vehicle requirements database into an edge case capturing component (ECC) and applying the ECC to generate inputs to the AI/ML component development; inputting design data as design requirements from a vehicle designer or developer into a requirements capturing module (RCM); providing the design requirements from the RCM to an asset refinement engine (ARE) to validate and generate the design requirements; refining the design requirements applying a satisfiability modulo theory (SMT) formulae solver; and extracting and analyzing data retrieved from a dataset and sending the data to a dataset generator to generate a new dataset.

19 . The method of claim 18 , further including:
extracting behaviors from real life driving situations; retrieving images from observers of a vehicle; and pre-analyzing and filtering edge cases using the ECC.

20 . The method of claim 18 , further including:
extracting MR relations from the data retrieved from the dataset; converting the MR relations to SMT formulae; and preparing an updated component dataset using an AI/ML algorithm.

Assignees

• Gm Global Tech Operations Llc

Inventors

• Sethu Ramesh
• Debouk Rami Ismail
• Giusto Paolo
• Sarwar Azeem

Classifications

• G06F8/60

  Software deployment · CPC title

• G06F8/10

  Requirements analysis; Specification techniques · CPC title

• G06V10/774

  Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06F8/34

  Graphical or visual programming · CPC title

• H04W4/44

  for communication between vehicles and infrastructures, e.g. vehicle-to-cloud [V2C] or vehicle-to-home [V2H] · CPC title