Automatic routing of user inquiries using natural language and image recognition models

Patent metadata

Field	Value
Publication number	US-12354151-B2
Application number	US-202218064129-A
Country	US
Kind code	B2
Filing date	Dec 9, 2022
Priority date	Dec 9, 2022
Publication date	Jul 8, 2025
Grant date	Jul 8, 2025

Abstract

A system or a method for using machine learning to automatically route user inquiries to a retailer are presented. The system receives an inquiry from a client device associated with a user. The inquiry includes text content and an image. The system uses a natural language model to analyze the received text to identify a first category of items. The system applies the received image to an image recognition model to identify a second category of items contained in the received image. The system then identifies a retailer that carries items in at least one of the first or second category of items, and suggests the retailer to the user via the client device associated with the user. A retail associate at the retailer can then respond to the inquiry via a client device associated with the retailer.

First claim

What is claimed is:

1. A method comprising, at a computer system comprising a processor and a computer-readable medium:
receiving an inquiry from a client device associated with a user, the inquiry comprising text content and an image;
using a natural language model to analyze the received text content to identify a first category of items;
applying the received image to an image recognition model to identify a second category of items contained in the received image;
identifying a retailer that carries items in at least one of the first or second category of items by applying an item availability model to the first or second category of items of a plurality of retailers, wherein the item availability model is trained using historical inventory records from the plurality of retailers;
in response to identifying the retailer that carries the items in the at least one of the first or second category of items, sending, to the client device associated with the user, suggestion information that causes the client device associated with the user to display one or more user interfaces that suggest the retailer to the user;
identifying a shopper associated with the retailer who is a subject matter expert in the first or second category of items based on an expertise score of the shopper, wherein the expertise score of the shopper is determined by applying a shopper expertise model trained using data associated with historical orders completed by the shopper, the shopper expertise model is trained by
accessing data about a plurality of shoppers labeled with expertise scores, the data about each of the plurality of shoppers including data about historical orders completed by the shopper, the historical orders including items in the first or second category;
applying parameters of the shopper expertise model to the data about each of the plurality of shoppers to predict an expertise score; and
in response to the predicted expertise score differing from the labeled expertise score, adjusting the parameters of the shopper expertise model to reduce the difference between the labeled expertise score and the predicted expertise score; and

in response to identifying the shopper associated with the retailer who is a subject matter expert of the first or second category of items, transmitting the user's inquiry to the shopper's client device for display, thereby enabling direct communication between the user and the shopper.

2. The method of claim 1 , further comprising:
using the natural language model to analyze the received text content to identify one or more first items in the first category of items; and
applying the received image to the image recognition model to identify one or more second items in the second category of items, wherein identification of the retailer is further based on the one or more first items or the one or more second items.

3. The method of claim 2 , further comprising:
using a machine-learning item availability model to predict a likelihood of the one or more first items or one or more second items being available at the retailer;
selecting a retailer with a higher likelihood of the one or more first items or one or more second items being available; and
routing the received inquiry to a client device of the selected retailer.

4. The method of claim 2 , further comprising:
selecting a retailer that is within a predetermined distance from a delivery address of the user; and
routing the received inquiry to a client device of the selected retailer.

5. The method of claim 1 , further comprising:
selecting a retailer that the user has historically made an order with based on a profile of the user;
routing the received inquiry to a client device of the selected retailer.

6. The method of claim 1 , further comprising:
identifying a shopper among a plurality of shoppers based on profiles of the plurality of shoppers; and
routing the received inquiry to a client device of the identified shopper.

7. The method of claim 6 , further comprising:
receiving a recommendation from the client device of the identified shopper, recommending the retailer; and
responsive to the recommendation, rerouting the received inquiry to the client device of the retailer.

8. The method of claim 6 , further comprising:
receiving a recommendation from the client device of the identified shopper, recommending a second shopper; and
responsive to the recommendation, rerouting the received inquiry to a client device of the second shopper.

9. The method of claim 6 , wherein the profiles of the plurality of shoppers include (1) historical orders that the plurality of shoppers have fulfilled, or (2) answers that the plurality of shoppers have provided responsive to historical inquiries from users.

10. The method of claim 6 , wherein the profiles of the plurality of the shoppers include expertise scores associated with the first or second category of items.

11. The method of claim 10 , further comprising:
for each of the plurality of shoppers,
generating an expertise score associated with at least one of a plurality of categories of items; and
storing the expertise score in a shopper information database, the expertise score being stored relationally with other shopper information of the corresponding shopper.

12. The method of claim 11 , wherein the expertise score of a shopper associated with a particular category of items is generated based on at least one of (1) a number of successful orders completed for a particular retailer that carries the particular category of items, (2) a frequency of successful orders completed for a particular retailer that carries the particular category of items, (3) a number of successful orders completed for the particular category, (4) a frequency of successful orders completed for the particular category, (5) a number of shopper recommendations associated with the particular category of items that are approved by users, or (6) a frequency of shopper recommendations associated with the particular category of items are approved by users.

13. A computer program item comprising a non-transitory computer readable storage medium having instructions encoded thereon that, when executed by a processor of a computing system, cause the computing system to:
receive an inquiry from a client device associated with a user, the inquiry comprising text content and an image;
use a natural language model to analyze the received text content to identify a first category of items;
apply the received image to an image recognition model to identify a second category of items contained in the received image;
identify a retailer that carries items in at least one of the first or second category of items by applying an item availability model to the first or second category of items of a plurality of retailers, wherein the item availability model is trained using historical inventory records from the plurality of retailers;
in response to identifying the retailer that carries the items in the at least one of the first or second category of items, send, to the client device associated with the user, suggestion information that causes the client device associated with the user to display one or more user interfaces that suggest the retailer to the user;
identify a shopper associated with the retailer who is a subject matter expert in the first or second category of items based on an expertise score of the shopper, wherein the expertise score of the shopper is determined by applying a shopper expertise model trained using data associated with historical orders completed by the shopper, the shopper expertise model is trained by
accessi

Assignees

• Maplebear Inc Dba Instacart
• Maplebear Inc

Inventors

• Maharaj Shaun Navin
• Scheibelhut Brent
• Oberemk Mark

Classifications

• G06F40/279

  Recognition of textual entities · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06Q30/0631 Primary

  Recommending goods or services · CPC title