Machine learning based models for object recognition

Patent metadata

Field	Value
Publication number	US-11715290-B2
Application number	US-202117558416-A
Country	US
Kind code	B2
Filing date	Dec 21, 2021
Priority date	Nov 19, 2019
Publication date	Aug 1, 2023
Grant date	Aug 1, 2023

Abstract

Machine learning based models recognize objects in images. Specific features of the object are extracted from the image using machine learning based models. The specific features extracted from the image assist deep learning based models in identifying subtypes of a type of object. The system recognizes the objects and collections of objects and determines whether the arrangement of objects violates any predetermined policies. For example, a policy may specify relative positions of different types of objects, height above ground at which certain types of objects are placed, or an expected number of certain types of objects in a collection.

First claim

What is claimed is:

1. A computer-implemented method for enforcing policies associated with storing of physical objects, the method comprising:
receiving an image displaying one or more collections of physical objects stored on one or more shelves, wherein a shelf stores objects of a plurality of object types, wherein a collection stores objects of a particular object type;
accessing a plurality of machine learning based models, wherein one or more of the plurality of machine learning based models are configured to predict features of the image;
providing the predicted features as input to an object recognition model, the object recognition model configured to predict a property of the collections of physical objects displayed in the received image;
determining based on the predicted property, whether a collection of physical objects displayed in the image violates a policy associated with storing the physical objects on shelves; and
responsive to determining a violation of the policy, sending an alert describing the violation of the policy.

2. The computer-implemented method of claim 1 , further comprising, generating a report describing the violation of the policy.

3. The computer-implemented method of claim 1 , wherein the policy represents a constraint based on a property of the collection of physical objects.

4. The computer-implemented method of claim 1 , wherein the policy specifies relative positions of different types of objects on the one or more shelves.

5. The computer-implemented method of claim 1 , wherein the policy specifies a height above ground at which objects of a particular type are placed on the one or more shelves.

6. The computer-implemented method of claim 1 , wherein the policy specifies a number of objects of a particular type that should be available on the one or more shelves.

7. The computer-implemented method of claim 1 , wherein the policy specifies an order in which objects of a particular type should be placed on the one or more shelves.

8. The computer-implemented method of claim 1 , wherein the policy is specified as a constraint represented as an expression and determining whether the collection of physical objects displayed in the image violates a policy comprises evaluating the expression.

9. The computer-implemented method of claim 1 , further comprising:
executing a machine learning based model to determine a bounding box within the image; and
extracting a feature based on a portion of the image within the bounding box.

10. A non-transitory computer readable storage medium, storing instructions that when executed by one or more computer processors, cause the one or more computer processors to perform steps comprising:
receiving an image displaying one or more collections of physical objects stored on one or more shelves, wherein a shelf stores objects of a plurality of object types, wherein a collection stores objects of a particular object type;
accessing a plurality of machine learning based models, wherein one or more of the plurality of machine learning based models are configured to predict a features of the image;
providing the predicted features as input to an object recognition model, the object recognition model configured to predict a property of the collections of physical objects displayed in the received image;
determining based on the predicted property, whether a collection of physical objects displayed in the image violates a policy associated with storing the physical objects on shelves; and
responsive to determining a violation of the policy, sending an alert describing the violation of the policy.

11. The non-transitory computer readable storage medium of claim 10 , further comprising, generating a report describing the violation of the policy.

12. The non-transitory computer readable storage medium of claim 10 , wherein the policy represents a constraint based on the property of the collection of physical objects.

13. The non-transitory computer readable storage medium of claim 10 , wherein the policy specifies relative positions of different types of objects on the one or more shelves.

14. The non-transitory computer readable storage medium of claim 10 , wherein the policy specifies a height above ground at which objects of a particular type are placed on the one or more shelves.

15. The non-transitory computer readable storage medium of claim 10 , wherein the policy specifies a number of objects of a particular type that should be available on the one or more shelves.

16. The non-transitory computer readable storage medium of claim 10 , wherein the policy specifies an order in which objects of a particular type should be placed on the one or more shelves.

17. The non-transitory computer readable storage medium of claim 10 , wherein the policy is specified as a constraint represented as an expression and determining whether the collection of physical objects displayed in the image violates a policy comprises evaluating the expression.

18. The non-transitory computer readable storage medium of claim 10 , wherein the instructions further cause the one or more computer processors to perform steps comprising:
executing a machine learning based model to determine a bounding box within the image; and
extracting a feature based on a portion of the image within the bounding box.

19. A computer system comprising:
one or more computer processors; and
a non-transitory computer readable storage medium, storing instructions that when executed by the one or more computer processors, cause the one or more computer processors to perform steps comprising:
receiving an image displaying one or more collections of physical objects stored on one or more shelves, wherein a shelf stores objects of a plurality of object types, wherein a collection stores objects of a particular object type;
accessing a plurality of machine learning based models, wherein one or more of the plurality of machine learning based models are configured to predict a features of the image;
providing the predicted features as input to an object recognition model, the object recognition model configured to predict a property of the collections of physical objects displayed in the received image;
determining based on the predicted property, whether a collection of physical objects displayed in the image violates a policy associated with storing the physical objects on shelves; and
responsive to determining a violation of the policy, sending an alert describing the violation of the policy.

20. The computer system of claim 19 , wherein the instructions further cause the one or more computer processors to perform steps comprising:
executing a machine learning based model to determine a bounding box within the image; and
extracting a feature based on a portion of the image within the bounding box.

Assignees

• Salesforce Com Inc
• Salesforce Inc

Inventors

• Mustafi Joy
• Kumar Lakshya
• Dua Rajdeep
• Rama Rao Machiraju Pakasasana

Classifications

• G06V10/82 Primary

  using neural networks · CPC title

• G06F18/217

  Validation; Performance evaluation; Active pattern learning techniques · CPC title

• G06F18/2148

  characterised by the process organisation or structure, e.g. boosting cascade · CPC title

• G06F18/24765

  Rule-based classification · CPC title

• G06V10/806

  of extracted features · CPC title