System, Method, and Platform for Auto Machine Learning Via Optimal Hybrid AI Formulation from Crowd

Patent metadata

Field	Value
Publication number	US-2021174130-A1
Application number	US-201916703430-A
Country	US
Kind code	A1
Filing date	Dec 4, 2019
Priority date	Dec 4, 2019
Publication date	Jun 10, 2021
Grant date	—

Abstract

Aspects of the subject disclosure may include, for example, receiving a plurality of proposed machine learning solutions to a machine learning problem including receiving, for each respective proposed machine learning solution of the plurality of proposed machine learning solutions, one or more of a machine learning model, a dataset and a data pipeline output; automatically determining hybrid solutions to the machine learning problem, including combining, by the processing system, at least one of a first component from a first proposed machine learning solution with at least one of a second component from a second proposed machine learning solution; and ranking the hybrid solutions including determining a log loss score for each hybrid solution and sorting the hybrid solutions according to the log loss score for each hybrid solution. Other embodiments are disclosed.

First claim

What is claimed is:

1 . A device, comprising:
a processing system including a processor; and a memory that stores executable instructions that, when executed by the processing system, facilitate performance of operations, the operations comprising: receiving information defining a machine learning problem to be solved; developing an initial machine learning solution to the machine learning problem; scoring the initial machine learning solution; receiving information about a plurality of other machine learning solutions to the machine learning problem, each respective machine learning solution of the plurality of machine learning solutions comprising two or more components; scoring each respective machine learning solution of the plurality of machine learning solutions to determine respective scores; ranking the respective machine learning solutions and the initial machine learning solution; developing one or more hybrid machine learning solutions including at least one component of the initial machine learning solution and at least one component of the plurality of machine learning solutions; scoring the one or more hybrid machine learning solutions; ranking the one or more hybrid machine learning solutions with the respective machine learning solutions and the initial machine learning solution to produce a leaderboard ranking; providing a human readable indication of the leaderboard ranking; and receiving a human input, the human input indicating a selected machine learning solution among the one or more hybrid machine learning solutions, the respective machine learning solutions, the initial machine learning solution.

2 . The device of claim 1 , wherein the operations further comprise:
selecting a machine learning solution of the plurality of machine learning solutions; selecting a component of the selected machine learning solution; and developing a hybrid machine learning solution by combining the component of the selected machine learning solution with components of the initial machine learning solution.

3 . The device of claim 2 , wherein the selecting the component of the selected machine learning solution comprises selecting one of a data set, feature set, a machine learning algorithm, a parameter set, and a validation technique of the selected machine learning solution.

4 . The device of claim 2 , wherein the operations further comprise:
scoring the hybrid machine learning solution; and comparing a score for the hybrid machine learning solution with scores for the respective machine learning solutions and a score for the initial machine learning solution to produce the leaderboard ranking.

5 . The device of claim 2 , wherein the operations further comprise:
iteratively selecting each machine learning solution of the plurality of machine learning solutions; iteratively selecting each component of the selected machine learning solution; developing the hybrid machine learning solution by combining the selected component of the selected machine learning solution with components of the initial machine learning solution or other components of one or more other machine learning solutions of the plurality of machine learning solutions; and scoring the hybrid machine learning solution.

6 . The device of claim 1 , wherein the operations further comprise:
conducting a fraud detection competition among the one or more hybrid machine learning solutions, wherein the fraud detection conducting the competition comprises:
providing, by an automatic process forming a judge of the fraud detection competition, authentication event data to a competitor, wherein the competitor is a hybrid machine learning solution of the one or more hybrid machine learning solutions;
providing. by the automatic process, transaction event data to the competitor, wherein the transaction event data comprises information about a potentially fraudulent transaction;
pausing, by the automatic process, a predetermined amount of time, wherein the pausing comprises suspending provision of additional authentication event data and additional transaction event data to the competitor during the predetermined amount of time to prevent the competitor from receiving future data;
receiving, by the automatic process, a fraud risk score from the competitor;
scoring, by the automatic process, the competitor based at least in part on the fraud risk score; and
if the fraud risk score is not received from the competitor within the predetermined amount of time, penalizing, by the automatic process, the competitor in the fraud detection competition.

7 . The device of claim 1 , wherein the operations further comprise:
receiving the information defining a machine learning problem to be solved from a user; providing at least some of the information defining a machine learning problem to be solved to a plurality of other users; receiving from the other users information about a plurality of other machine learning solutions to the machine learning problem, wherein the plurality of other machine learning solutions are developed by the other users; providing to the user and the other users the human readable indication of the leaderboard ranking; and receiving from the user the human input to select a machine learning solution satisfying requirements of the user.

8 . The device of claim 1 , wherein the operations further comprise:
receiving, from the other users, information about revised machine learning solutions, wherein the revised machine learning solutions are based on the human readable indication of the leaderboard ranking; developing one or more new hybrid machine learning solutions based on the information about revised machine learning solutions; scoring the one or more new hybrid machine learning solutions; ranking the one or more new hybrid machine learning solutions to produce an updated leaderboard ranking; and providing a human readable indication of the updated leaderboard ranking.

9 . A machine-readable medium, comprising executable instructions that, when executed by a processing system including a processor, facilitate performance of operations, the operations comprising:
providing information about a machine learning problem to be solved, the providing comprising presenting the information about the machine learning problem to be solved to a web accessible interface; receiving, from a plurality of users, information about proposed solutions to the machine learning problem, wherein the receiving comprises receiving the information about the proposed solutions from the web accessible interface; ranking the proposed solutions, wherein the ranking comprises sorting the proposed solutions according to a logarithmic loss determination for each respective proposed solution of the proposed solutions; forming a plurality of hybrid solutions, wherein the forming comprises selecting one or more first components of a first respective proposed solution and one or more second components of a second respective proposed solution and combining the one or more first components with the one or more second components to form a hybrid solution; ranking the plurality of hybrid solutions with the proposed solutions, wherein the ranking comprises sorting the plurality of hybrid solutions with the proposed solutions according to a logarithmic loss determination for each respective hybrid solution and for each respective proposed solution of the proposed solutions; providing information about the ranking of the plurality of hybrid solutions with the proposed solutions; and receiving a user selection of a selected machine learning solution of the plurality of hybrid solutions and the proposed solutions, wherein the user sel

Assignees

• At & T Ip I Lp

Inventors

• Austin Mark
• Dabholkar Abhay
• Nagarajan Vijayan
• Savage Lauren
• Whitney Joshua

Classifications

• G06V10/96

  Management of image or video recognition tasks · CPC title

• G06V10/80

  Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level (multimodal speaker identification or verification G10L17/10) · CPC title

• G06F18/2113

  by ranking or filtering the set of features, e.g. using a measure of variance or of feature cross-correlation · CPC title

• G06N20/20 Primary

  Ensemble learning · CPC title

• G06N5/01

  Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound · CPC title