Targeted training of inductive multi-organization recommendation models for enterprise applications

Patent metadata

Field	Value
Publication number	US-11983649-B2
Application number	US-202117510523-A
Country	US
Kind code	B2
Filing date	Oct 26, 2021
Priority date	Oct 26, 2021
Publication date	May 14, 2024
Grant date	May 14, 2024

Abstract

An enterprise system server, a computer-readable storage medium, and a method for targeted training of inductive multi-organization recommendation models for enterprise applications are described herein. The method includes receiving enterprise application data from remote organization computing systems executing the enterprise application, training per-organization recommendation models for a subset of the organizations, and validating each per-organization recommendation model on enterprise application data corresponding to one or more other organizations. The method also includes calculating a transferability metric for each per-organization recommendation model based on results obtained during validation, determining a specified number of organizations including the best-transferring per-organization recommendation models based on the calculated transferability metrics, and training an inductive multi-organization recommendation model using the enterprise application data from the specified number of organizations. The method further includes utilizing the trained inductive multi-organization recommendation model to provide user recommendations to the remote organization computing systems during execution of the enterprise application.

First claim

What is claimed is:

1. A method implemented in a computing system comprising a processor, and wherein the method comprises:
receiving, via a network, enterprise application data from remote organization computing systems executing an enterprise application, wherein each remote organization computing system corresponds to an organization that subscribes to the enterprise application;
training, via the processor, a per-organization recommendation machine learning model for each of a subset of the organizations that subscribe to the enterprise application using at least a portion of the enterprise application data received from the corresponding remote organization computing systems;
validating each per-organization recommendation machine learning model on at least a portion of the enterprise application data corresponding to at least one other organization;
calculating a transferability metric for each per-organization recommendation machine learning model based on results obtained during the validation of the per-organization recommendation machine learning model, wherein the calculated transferability metric includes a mean average precision (MAP) for the results obtained during the validation of each per-organization recommendation machine learning model;
determining a specified number of organizations comprising best-transferring per-organization recommendation machine learning models based on the calculated transferability metrics;
training an inductive multi-organization recommendation machine learning model using at least a portion of the enterprise application data from the specified number of organizations comprising the best-transferring per-organization recommendation machine learning models;
transmitting, via the network, user recommendations to the remote organization computing systems during execution of the enterprise application, the user recommendations being derived based on the trained inductive multi-organization recommendation machine learning model; and
receiving new enterprise application data from at least one of the remote organization computing systems, wherein the new enterprise application data is used to update the inductive multi-organization recommendation machine learning model.

2. The method of claim 1 , further comprising training the per-organization recommendation machine learning models and the inductive multi-organization recommendation machine learning model using at least one of network topology features or temporal features, but excluding content-based features.

3. The method of claim 1 , further comprising periodically updating the inductive multi-organization recommendation machine learning model as new enterprise application data are received from the specified number of organizations comprising the best-transferring per-organization recommendation machine learning models.

4. The method of claim 1 , wherein the transferability metric comprises a ranking metric.

5. The method of claim 1 , wherein the per-organization recommendation machine learning models and the inductive multi-organization recommendation machine learning model comprise at least one of a logistic regression model, a graph convolutional network (GCN) model, or an inductive graph-based matrix completion (IGMC) model.

6. The method of claim 1 , wherein training the inductive multi- organization recommendation machine learning model using at least the portion of the enterprise application data from the specified number of organizations comprises:
disregarding the best-transferring per-organization recommendation machine learning models; and
training the inductive multi-organization recommendation machine learning model using only at least the portion of the enterprise application data from the specified number of organizations comprising the best-transferring per-organization recommendation machine learning models.

7. The method of claim 1 , wherein training the inductive multi-organization recommendation machine learning model using at least the portion of the enterprise application data from the specified number of organizations comprises aggregating the best-transferring per-organization recommendation machine learning models.

8. The method of claim 1 , wherein the enterprise application comprises a communication/collaboration application, and wherein the enterprise application data comprise at least one of user-user data or user-item data.

9. The method of claim 1 , wherein the new enterprise application data is used to iteratively update the inductive multi-organization recommendation machine learning model.

10. The method of claim 1 , wherein the new enterprise application data is used to periodically or aperiodically update the inductive multi-organization recommendation machine learning model.

11. A computer-readable storage medium comprising computer-executable instructions that, when executed by a processor, cause the processor to:
receive, via a network, enterprise application data from remote organization computing systems executing an enterprise application, wherein each remote organization computing system corresponds to an organization that subscribes to an enterprise application;
train a per-organization recommendation machine learning model for each of a subset of the organizations that subscribe to the enterprise application using at least a portion of the enterprise application data received from the corresponding remote organization computing systems;
validate each per-organization recommendation machine learning model on at least a portion of the enterprise application data corresponding to at least one other organization;
calculate a transferability metric for each per-organization recommendation machine learning model based on results obtained during the validation of the per-organization recommendation machine learning model, wherein the calculated transferability metric includes a mean average precision (MAP) for the results obtained during the validation of each per-organization recommendation machine learning model;
determine a specified number of organizations comprising best-transferring per-organization recommendation machine learning models based on the calculated transferability metrics;
train an inductive multi-organization recommendation machine learning model using at least a portion of the enterprise application data from the specified number of organizations comprising the best-transferring per-organization recommendation machine learning models;
transmitting, via the network, user recommendations to the remote organization computing systems during execution of the enterprise application, the user recommendations being derived based on the trained inductive multi-organization recommendation machine learning model; and
receive new enterprise application data from at least one of the remote organization computing systems, wherein the new enterprise application data is used to update the inductive multi-organization recommendation machine learning model.

12. The computer-readable storage medium of claim 11 , wherein the computer-executable instructions further cause the processor to train the per-organization recommendation machine learning models and the inductive multi-organization recommendation machine learning model using at least one of network topology features or temporal features, but excluding content-based features.

13. The computer-readable storage medium of claim 11 , wherein the computer-executable instructions further cause the processor to periodically update the inductive multi-organization recommendation machine learning model as new enterprise application data are received from the specified nu

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Tomlinson Kiran
• Yang Longqi
• Wan Mengting
• Lu Cao
• Hecht Brent Jaron
• Teevan Jaime

Classifications

• G06Q10/063 Primary

  Operations research, analysis or management · CPC title

• G06N5/022

  Knowledge engineering; Knowledge acquisition · CPC title

• G06N5/04

  Inference or reasoning models · CPC title

• G06Q10/067 Primary

  Enterprise or organisation modelling · CPC title

• G06N3/042

  Knowledge-based neural networks; Logical representations of neural networks · CPC title