Method, system, and computer program product for operating dynamic shadow testing environments

Patent metadata

Field	Value
Publication number	US-11928048-B2
Application number	US-202318121214-A
Country	US
Kind code	B2
Filing date	Mar 14, 2023
Priority date	Oct 21, 2019
Publication date	Mar 12, 2024
Grant date	Mar 12, 2024

Abstract

Described are a method, system, and computer program product for operating dynamic shadow testing environments for machine-learning models. The method includes generating a shadow testing environment operating at least two transaction services. The method also includes receiving a plurality of transaction authorization requests. The method further includes determining a first percentage associated with a first testing policy of the first transaction service and a second percentage associated with a second testing policy of the second transaction service. The method further includes replicating in the shadow testing environment, in real-time with processing the payment transactions, a first portion of the plurality of transaction authorization requests and a second portion of the plurality of transaction authorization requests. The method further includes testing the first transaction service using the first set of replicated transaction data and the second transaction service using the second set of replicated transaction data.

First claim

What is claimed is:

1. A computer-implemented method comprising:
generating, with at least one processor, a shadow testing environment operating at least two transaction services, wherein the at least two transaction services comprise a first transaction service executed at least partly with a first machine-learning model and a second transaction service executed at least partly with a second machine-learning model, and wherein the first transaction service is a different version of a same service as the second transaction service;
receiving, with the at least one processor at a transaction processing system of a transaction service provider in an electronic payment processing network, a plurality of transaction authorization requests, each transaction authorization request of the plurality of transaction authorization requests comprising transaction data of a payment transaction associated with a payment device, wherein the transaction processing system is configured to receive the plurality of transaction authorization requests from at least one merchant and process payment transactions associated with the plurality of transaction authorization requests as said payment transactions are initiated by at least one point-of-sale terminal of the at least one merchant;
determining, with at least one processor, a first parameter defining a first testing policy of the first transaction service and a second parameter defining a second testing policy of the second transaction service, wherein the first parameter defines a percent of actively processed data on which to test the first transaction service, and wherein the second parameter defines a percent of actively processed data on which to test the second transaction service;
replicating in the shadow testing environment, with the at least one processor in real-time with the transaction processing system processing the payment transactions, a first portion of the plurality of transaction authorization requests to produce a first set of replicated transaction data, and a second portion of the plurality of transaction authorization requests to produce a second set of replicated transaction data, wherein the first portion is based on the first parameter and the second portion is based on the second parameter; and
testing, with the at least one processor, the first transaction service using the first set of replicated transaction data as input, and the second transaction service using the second set of replicated transaction data as input.

2. The method of claim 1 , further comprising storing, with the at least one processor and in a database, the first testing policy and the second testing policy, wherein the first testing policy comprises the first parameter and a first identifier of the first transaction service, and wherein the second testing policy comprises the second parameter and a second identifier of the second transaction service.

3. The method of claim 2 , wherein the percent of actively processed data defined by the first parameter is a higher than the percent of actively processed data defined by the second parameter.

4. The method of claim 2 , further comprising:
detecting, with the at least one processor, a modification of the first testing policy or the second testing policy by a user through a computer interface;
determining, with the at least one processor, that the modification comprises at least one of a new identifier of at least one new machine-learning model and a new identifier of at least one new transaction service; and
regenerating, with the at least one processor, the shadow testing environment to perform at least one of the following: operate the at least one new transaction service, and use the at least one new machine-learning model.

5. The method of claim 4 , further comprising, in response to determining that at least one of the at least one new transaction service and the at least one new machine-learning model has not yet been stored, prompting, with the at least one processor, the user through the computer interface to provide at least one of the at least one new transaction service and the at least one new machine-learning model.

6. The method of claim 2 , wherein generating the shadow testing environment further comprises:
identifying, with the at least one processor, a set of computer resources available for operating shadow testing environments;
determining, with the at least one processor, a resource requirement of the first testing policy and the second testing policy;
selecting, with the at least one processor based at least partly on the resource requirement, a subset of the set of computer resources to operate the shadow testing environment; and
initiating, with the at least one processor, the shadow testing environment using the subset of the set of computer resources.

7. The method of claim 6 , further comprising, in response to detecting a modification of the first testing policy or the second testing policy:
determining, with the at least one processor and based on the modification, a new resource requirement of the first testing policy and the second testing policy;
selecting, with the at least one processor based at least partly on the new resource requirement, a new subset of the set of computer resources to operate the shadow testing environment; and
initiating, with the at least one processor, the shadow testing environment using the new subset of the set of computer resources.

8. A system comprising a server comprising at least one processor, the server being at least one of programmed and configured to:
generate a shadow testing environment operating at least two transaction services, wherein the at least two transaction services comprise a first transaction service executed at least partly with a first machine-learning model and a second transaction service executed at least partly with a second machine-learning model, and wherein the first transaction service is a different version of a same service as the second transaction service;
receive, at a transaction processing system of a transaction service provider in an electronic payment processing network, a plurality of transaction authorization requests, each transaction authorization request of the plurality of transaction authorization requests comprising transaction data of a payment transaction associated with a payment device, wherein the transaction processing system is configured to receive the plurality of transaction authorization requests from at least one merchant and process payment transactions associated with the plurality of transaction authorization requests as said payment transactions are initiated by at least one point-of-sale terminal of the at least one merchant;
determine a first parameter defining a first testing policy of the first transaction service and a second parameter defining a second testing policy of the second transaction service, wherein the first parameter defines a percent of actively processed data on which to test the first transaction service, and wherein the second parameter defines a percent of actively processed data on which to test the second transaction service;
replicate in the shadow testing environment, in real-time with the transaction processing system processing the payment transactions, a first portion of the plurality of transaction authorization requests to produce a first set of replicated transaction data, and a second portion of the plurality of transaction authorization requests to produce a second set of replicated transaction data, wherein the first portion is based on the first parameter and the second portion is based on the second parameter; and
test the first transaction service using the first set of replicated transaction data as input

Assignees

• Visa Int Service Ass

Inventors

• Lu Ranglin
• GU Yu
• Cheng Yinhe

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06F11/3698 Primary

  Environments for analysis, debugging or testing of software · CPC title

• G06F11/3664 Primary

  Physics · mapped topic

• G06F11/362

  Debugging of software · CPC title

• G06F11/3696

  Methods or tools to render software testable · CPC title