System and method for computer vision driven applications within an environment

Patent metadata

Field	Value
Publication number	US-10339595-B2
Application number	US-201715590467-A
Country	US
Kind code	B2
Filing date	May 9, 2017
Priority date	May 9, 2016
Publication date	Jul 2, 2019
Grant date	Jul 2, 2019

Abstract

A system and method for computer vision driven applications in an environment that can include collecting image data across an environment; maintaining an environmental object graph from the image data whereby maintaining the environmental object graph is an iterative process that includes: classifying objects, tracking object locations, detecting interaction events, instantiating object associations in the environmental object graph, and updating the environmental object graph by propagating change in at least one object instance across object associations; and inspecting object state for at least one object instance in the environmental object graph and executing an action associated with the object state. The system and method can be applied to automatic checkout, inventory management, and/or other system integrations.

First claim

We claim:

1. A method for expediting a checkout process comprising:
collecting image data across an environment;
maintaining an environmental object graph from the image data wherein the environmental object graph is a data representation of computer vision classified objects in space and time across the environment, the environmental object graph comprising at least a subset of objects having object associations;
wherein maintaining the environmental object graph comprises at least one instance of:
in a first region captured in the image data, classifying a first object and at least a shopper object,
in the first region, detecting an interaction event between the first object and the shopper object, and
updating the environmental object graph whereby the first object is associated with the shopper object;

inspecting objects that are associated with the shopper object and thereby generating a checkout list.

2. The method of claim 1 , further comprising, in a second region captured in the image data, detecting the shopper object and initiating a checkout process with the checkout list.

3. The method of claim 2 , further comprising accessing an account associated with the shopper object; and wherein initiating the checkout process comprises charging the checkout list to the account after entering the second region.

4. The method of claim 3 , wherein initiating the checkout process comprises communicating the checkout list to a checkout processing system in the second region.

5. The method of claim 4 , further comprising, at the checkout processing system, populating the checkout processing system with items from the checkout list.

6. The method of claim 5 , further comprising biasing product input of the checkout processing system for items in the checkout list.

7. The method of claim 2 , wherein the first object is associated with the shopper object with an initial confidence level; and wherein generating the checkout list comprises adding objects to the checkout list with confidence levels satisfying a confidence threshold.

8. The method of claim 2 , wherein the first region is captured in image data from a first camera and the second region is captured in image data from a second camera.

9. The method of claim 2 , wherein the first region and the second region are captured in image data from one camera.

10. The method of claim 2 , wherein the first object is associated with the shopper object as a possessed object with a first confidence level; and wherein maintaining the environmental object graph in a second instance comprises:
in a third region captured in the image data, classifying a second object and the shopper object,
associating the second object as a possessed object with the shopper object with a second confidence level, and
updating the environmental object graph wherein the first confidence level is altered at least partially in response to the second confidence level.

11. The method of claim 1 , further comprising accessing an account associated with the shopper object; and in an application instance of the account, presenting the checkout list.

12. The method of claim 1 , wherein maintaining the environmental object graph in a second instance after the first instance comprises: in the first region, classifying the first object, and updating the environmental object graph and thereby removing the association of the first object with the shopper object in the first instance.

13. The method of claim 1 , wherein maintaining the environmental object graph in a second instance comprises:
in a second region captured in the image data, classifying a second object and the shopper object, wherein the second object is a compound object with contained objects,
in the second region, detecting an interaction event between the first object and the shopper object, and
updating the environmental object graph whereby the second object and the contained objects of the second object are associated with the shopper object.

14. The method of claim 1 , wherein detecting the interaction event between the first object and the shopper object comprises detecting proximity between the first object and the shopper object satisfying a proximity threshold.

15. The method of claim 1 , wherein classifying the first object and the shopper object comprises applying computer-vision driven processes during classification, the computer-vision driven processes including at least image feature extraction and classification and an application of neural networks.

16. A method comprising:
collecting image data across an environment;
maintaining an environmental object graph from the image data whereby maintaining the environmental object graph is an iterative process comprising:
classifying objects and storing corresponding object instances in the environmental object graph,
tracking object locations and establishing an association of object instances in an object path,
detecting interaction events and, for event instances of a subset of detected interaction events, generating an object association of at least two object instances involved in the interaction event, and
updating the environmental object graph comprising propagating change in at least one object instance across object associations; and

inspecting object state for at least one object instance in the environmental object graph and executing an action associated with the object state.

17. The method of claim 16 , wherein collecting imaging data comprises collecting imaging data from multiple image capture devices distributed across an environment.

18. The method of claim 17 , wherein collecting image data from multiple image capture devices distributed across an environment comprises collecting imaging data from a set of image capture devices that include at least two image capture configurations selected from: an inventory storage capture configuration, an interaction capture configuration, an object identification capture configuration, and a movable capture configuration.

19. The method of claim 16 , wherein detecting the interaction events comprises detecting proximity between a first object of a first classification and at least a second object of a second classification satisfying a proximity threshold.

20. The method of claim 16 , wherein detecting the interaction events comprises: for at least one interaction event, detecting an object proximity event, and, for at least a second interaction event, detecting an object transformation event.

21. The method of claim 20 , wherein detecting the object transformation event can comprise detecting an object transformation selected from the set of: an object appearance event, object disappearance event, and object classification mutation event.

22. The method of claim 16 , wherein classifying an object can include identifying an object as one of a set of possible object types, wherein possible object types comprise at least a product object, a compound object, and a person object.

23. The method of claim 16 , wherein maintaining the environmental object graph further comprises instantiating an object instance in the environmental object graph as probabilistically having a hierarchical association with at least one object.

24. The method of claim 16 , wherein maintaining the environmental object graph further comprises instantiating at least two probabilistically possible states of an object instance.

25. The metho

Assignees

• Grabango Co

Inventors

• Glaser William
• Van Osdol Brian

Classifications

• G06Q30/0643 Primary

  graphically representing goods, e.g. 3D product representation · CPC title

• G06Q30/0635

  replenishment orders; recurring orders · CPC title

• G06Q20/203

  Inventory monitoring · CPC title

• H04N7/181 Primary

  for receiving images from a plurality of remote sources · CPC title

• G07G1/0036

  Checkout procedures · CPC title