Governance based validation framework for JSON data using machine learning

Patent metadata

Field	Value
Publication number	US-11853272-B2
Application number	US-202117249077-A
Country	US
Kind code	B2
Filing date	Feb 19, 2021
Priority date	Feb 19, 2021
Publication date	Dec 26, 2023
Grant date	Dec 26, 2023

Abstract

A method, a computer system, and a computer program product for data validation in a microservice environment is provided. Embodiments of the present invention may include receiving a request based on an application configuration validation. Embodiments of the present invention may include determining a schema is not defined based on the request. Embodiments of the present invention may include generating the schema using machine learning. Embodiments of the present invention may include using the generated schema for a plurality of validations.

First claim

What is claimed is:

1. A method comprising:
receiving input data, wherein the input data is used to train, test and validate a machine learning model;
receiving a request based on an application configuration validation;
determining a request schema is undefined based on one or more constraints included with the request;
generating a schema using the machine learning model, wherein the schema generated by the machine learning model is a set of constraints based on the input data received;
using the schema generated by the machine learning model for a plurality of validations, wherein a validation status for each of the plurality of validations of a first microservice are shared across one or more additional microservices enabling the plurality of validations to be skipped for the one or more additional microservices;
updating the machine learning model based on updated input data or additional data received;
generating an updated schema using an updated machine learning model; and
using the updated schema for the plurality of validations.

2. The method of claim 1 , further comprising:
performing the application configuration validation;
determining the application configuration validation is enabled; and
transmitting the request to a common validation framework.

3. The method of claim 1 , wherein the plurality of validations further comprises:
performing a whitelist validation;
performing a custom injectable validation, wherein the custom injectable validation is performed upon determining that a constraint is defined as overridable;
performing a schema based validation for a payload; and
performing a custom schema injectable validation for the payload.

4. The method of claim 1 , wherein the request is received by a validation framework that includes an input JSON payload validation component, an additional business validation component, the schema generated by the machine learning model and a custom validation injection component, wherein the validation framework is a common library that each incoming request received from each of a plurality of microservices must pass through.

5. The method of claim 1 , wherein the set of constraints are generated by the machine learning model to reduce an amount of input data to be analyzed in generating the schema.

6. The method of claim 1 , identifying mandatory data for analysis within the input data received based on a completeness of the input data for each attribute type.

7. The method of claim 1 , further comprising:
generating one or more new constraint types in addition to the set of constraints generated by the machine learning model based on one or more rules for regulatory compliance.

8. The method of claim 1 , wherein the machine learning model is trained according to semi-supervised machine learning procedures and wherein updating the machine learning model further comprises:
removing a data classification or a data label from a labeled dataset within the input data.

9. The method of claim 1 , the first microservice is a trusted source and has a trusted relationship with each of the one or more additional microservices.

10. The method of claim 1 , wherein the schema generated using the machine learning model is utilized in building a default model schema, wherein the default model schema is managed using the machine learning model to update versions of a representational state transfer (REST) application program interface (API) automatically such that there is no requirement to implement additional validations.

11. A computer system comprising:
one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage media, and program instructions stored on at least one of the one or more computer-readable tangible storage media for execution by at least one of the one or more processors via at least one of the one or more computer-readable memories, wherein the computer system is capable of performing a method comprising:
receiving input data, wherein the input data is used to train, test and validate a machine learning model;
receiving a request based on an application configuration validation;
determining a request schema is undefined based on one or more constraints included with the request;
generating a schema using the machine learning model, wherein the schema generated by the machine learning model is a set of constraints based on the input data received;
using the schema generated by the machine learning model for a plurality of validations, wherein a validation status for each of the plurality of validations of a first microservice are shared across one or more additional microservices enabling the plurality of validations to be skipped for the one or more additional microservices;
updating the machine learning model based on updated input data or additional data received;
generating an updated schema using an updated machine learning model; and
using the updated schema for the plurality of validations.

12. The computer system of claim 11 , further comprising:
performing the application configuration validation;
determining the application configuration validation is enabled; and
transmitting the request to a common validation framework.

13. The computer system of claim 11 , wherein the plurality of validations further comprises:
performing a whitelist validation;
performing a custom injectable validation, wherein the custom injectable validation is performed upon determining that a constraint is defined as overridable;
performing a schema based validation for a payload; and
performing a custom schema injectable validation for the payload.

14. The computer system of claim 11 , wherein the request is received by a validation framework that includes an input JSON payload validation component, an additional business validation component, the schema generated by the machine learning model and a custom validation injection component, wherein the validation framework is a common library that each incoming request received from each of a plurality of microservices must pass through.

15. A computer program product comprising:
one or more computer-readable tangible storage media and program instructions stored on at least one of the one or more computer-readable tangible storage media, the program instructions executable by a processor to cause the processor to perform a method comprising:
receiving input data, wherein the input data is used to train, test and validate a machine learning model;
receiving a request based on an application configuration validation;
determining a request schema is undefined based on one or more constraints included with the request;
generating a schema using the machine learning model, wherein the schema generated by the machine learning model is a set of constraints based on the input data received;
using the schema generated by the machine learning model for a plurality of validations, wherein a validation status for each of the plurality of validations of a first microservice are shared across one or more additional microservices enabling the plurality of validations to be skipped for the one or more additional microservices;
updating the machine learning model based on updated input data or additional data received;
generating an updated schema using an updated machine learning model; and
using the updated schema for the plurality of validations.

16. The computer program product of claim 15 , further comprising:
performing the application configuration validation;
determining the application configuration vali

Assignees

• IBM

Inventors

• Hiremath Pushpalatha M
• Naganna Soma Shekar
• Khatibi Mohammad
• Wang Mark

Classifications

• G06F16/213 Primary

  with details for schema evolution support · CPC title

• G06F16/288

  Entity relationship models · CPC title

• G06F18/2155

  characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling · CPC title

• G06N20/00

  Machine learning · CPC title

• G06F16/211 Primary

  Schema design and management · CPC title