Machine learning based models for object recognition

Patent metadata

Field	Value
Publication number	US-11210562-B2
Application number	US-202016751078-A
Country	US
Kind code	B2
Filing date	Jan 23, 2020
Priority date	Nov 19, 2019
Publication date	Dec 28, 2021
Grant date	Dec 28, 2021

Abstract

Machine learning based models recognize objects in images. Specific features of the object are extracted from the image using machine learning based models. The specific features extracted from the image assist deep learning based models in identifying subtypes of a type of object. The system recognizes the objects and collections of objects and determines whether the arrangement of objects violates any predetermined policies. For example, a policy may specify relative positions of different types of objects, height above ground at which certain types of objects are placed, or an expected number of certain types of objects in a collection.

First claim

We claim:

1. A computer implemented method for recognizing objects, the method comprising:
receiving an image displaying one or more collections of physical objects, the image further including information describing each collection of physical objects;
accessing a plurality of machine learning based models, each machine learning based model configured to predict a feature of the image;
for each of the plurality of machine learning based models:
executing the machine learning based model to determine coordinates of a bounding box within the image; and
extracting a feature based on a portion of the image within the bounding box;

providing the features extracted as input to an object recognition model, the object recognition model configured to predict a property of the collections of physical objects displayed in the image; and
determining based on the predicted property, whether a collection of objects displayed in the image violates a policy, wherein the policy represents a constraint based on the property of the collection; and
responsive to determining a violation of a policy, sending an alert describing the policy violation.

2. The method of claim 1 , further comprising:
generating a report describing compliance of the collections of objects with respect to a set of rules, wherein at least some of the rules specify placement of collections of objects.

3. The method of claim 1 , wherein the property represents a size of the collection, wherein the constraint represented by the policy specifies that the size of the collection should exceed a threshold value, the method comprising
responsive to determining that a size of a collection of physical objects is below the threshold value, sending an alert.

4. The method of claim 1 , wherein extracting a feature comprises:
performing optical character recognition on a portion of the image within a bounding box; and
determining a feature describing a result of the optical character recognition.

5. The method of claim 1 , wherein extracting a feature comprises:
recognizing a logo displayed within a portion of the image within a bounding box; and
determining a feature describing the logo.

6. The method of claim 1 , wherein a feature comprises a position of the bounding box with respect to a physical object displayed within the image.

7. The method of claim 1 , wherein extracting a feature comprises a size of the bounding box relative to a size of a physical object displayed within the image.

8. The method of claim 1 , further comprising:
determining a collection of related bounding boxes associated with a label describing a collection of physical objects, wherein a feature comprises a relative position of a bounding box within the label.

9. The method of claim 1 , further comprising:
determining a location of a bounding box within a physical object; and
wherein a feature represents a position of the bounding box within the physical object.

10. A non-transitory computer readable storage medium, storing instructions that when executed by a computer processor, cause the computer processor to perform steps comprising:
receiving an image displaying one or more collections of physical objects (e.g., shelves of a physical store), the image further displaying information describing each collection of physical objects;
accessing a plurality of machine learning based models, each machine learning based model configured to predict a feature of the image;
for each of the plurality of machine learning based models:
executing the machine learning based models to determine coordinates of a bounding box within the image; and
extracting a feature based on a portion of the image within the bounding box;

providing the features extracted as input to an object recognition model, the object recognition model configured to predict a property of the collections of physical objects displayed in the image; and
determining based on the predicted property, whether a collection of objects displayed in the image violates a policy, wherein the policy represents a constraint based on the property of the collection; and
responsive to determining a violation of a policy, sending an alert describing the policy violation.

11. The non-transitory computer readable storage medium of claim 10 , wherein extracting a feature causes the computer processor to perform steps comprising:
performing optical character recognition on a portion of the image within a bounding box; and
determining a feature describing a result of the optical character recognition.

12. The non-transitory computer readable storage medium of claim 10 , wherein the property represents a size of the collection, wherein the constraint represented by the policy specifies that the size of the collection should exceed a threshold value, wherein the instructions further cause the computer processor to perform steps comprising
responsive to determining that a size of a collection of physical objects is below the threshold value, sending an alert.

13. The non-transitory computer readable storage medium of claim 10 , wherein the instructions further cause the computer processor to perform steps comprising:
determining a collection of related bounding boxes associated with a label describing a collection of physical objects, wherein a feature comprises a relative position of a bounding box within the label.

14. The non-transitory computer readable storage medium of claim 10 , wherein the instructions further cause the computer processor to perform steps comprising:
determining a location of a bounding box within a physical object; and
wherein a feature represents a position of the bounding box within the physical object.

15. A computer system comprising:
a computer processor; and
a non-transitory computer readable storage medium, storing instructions that when executed by the computer processor, cause the computer processor to perform steps comprising:
receiving an image displaying one or more collections of physical objects (e.g., shelves of a physical store), the image further displaying information describing each collection of physical objects;
accessing a plurality of machine learning based models, each machine learning based model configured to predict a feature of the image;
for each of the plurality of machine learning based models:
executing the machine learning based models to determine coordinates of a bounding box within the image; and
extracting a feature based on a portion of the image within the bounding box;

providing the features extracted as input to an object recognition model, the object recognition model configured to predict a property of the collections of physical objects displayed in the image; and
determining based on the predicted property, whether a collection of objects displayed in the image violates a policy, wherein the policy represents a constraint based on the property of the collection; and
responsive to determining a violation of a policy, sending an alert describing the policy violation.

16. The computer system of claim 15 , wherein extracting a feature causes the computer processor to perform steps comprising:
performing optical character recognition on a portion of the image within a bounding box; and
determining a feature describing a result of the optical character recognition.

17. The computer system of claim 15 , wherein the property represents a size of the collection, wherein the constraint represented by the policy specifies that the size of the collection should exceed a threshold value, wherein the

Assignees

• Salesforce Com Inc

Inventors

• Mustafi Joy
• Kumar Lakshya
• Dua Rajdeep
• Rama Rao Machiraju Pakasasana

Classifications

• G06V10/806

  of extracted features · CPC title

• G06V30/18057

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title

• G06V30/19173

  Classification techniques · CPC title

• G06V10/82 Primary

  using neural networks · CPC title

• G06F18/2148

  characterised by the process organisation or structure, e.g. boosting cascade · CPC title