Directional impression analysis using deep learning

Patent metadata

Field	Value
Publication number	US-11544866-B2
Application number	US-202017089012-A
Country	US
Kind code	B2
Filing date	Nov 4, 2020
Priority date	Aug 7, 2017
Publication date	Jan 3, 2023
Grant date	Jan 3, 2023

Abstract

Systems and techniques are provided for detecting gaze direction of subjects in an area of real space. The system receives a plurality of sequences of frames of corresponding fields of view in the real space. The system uses sequences of frames in a plurality of sequences of frames to identify locations of an identified subject and gaze directions of the subject in the area of real space over time. The system includes logic having access to a database identifying locations of items in the area of real space. The system identifies items in the area of real space matching the identified gaze directions of the identified subject.

First claim

What is claimed is:

1. A system for detecting gaze direction of subjects in an area of real space, comprising:
a plurality of sensors, sensors in the plurality of sensors producing respective sequences of frames of corresponding fields of view in the area of real space; and
a processing system coupled to the plurality of sensors and including a database, the processing system including:
logic that uses the sequences of frames produced by sensors in the plurality of sensors to identify locations of an identified subject and gaze directions of the identified subject in the area of real space over time;
logic having access to a database identifying locations of items in the area of real space and that identifies items in the area of real space matching the identified gaze directions of the identified subject;
logic that stores the identified items matching the identified gaze directions of the identified subject as an entry in an item-subject correlations database and accumulating entries in the item-subject correlations database over a period of time for identified subjects, the entry in the accumulated entries including an item identifier, a subject identifier, and a location of the item in the area of real space; and
logic that generates a heat map of the area of real space, wherein the heat map identifies locations in the area of real space, wherein the locations identified by the heat map have a number of directional impressions that are above a threshold indicating an interest in items positioned at locations in the area of real space.

2. The system of claim 1 , further including logic that calculates a unique visitor directional impression from the accumulated entries in the item-subject correlations database, wherein the unique visitor directional impression identifies a number of unique or distinct subjects who looked at the item identified by the item identifier.

3. The system of claim 2 , further including logic that calculates unique visitor directional impressions over a period of time.

4. The system of claim 2 , further including logic that determines lengths of time for which the identified subject maintains respective gaze directions and stores the determined lengths of time in respective entries in the accumulated entries storing the identified items matching the identified gaze directions of the identified subject.

5. The system of claim 4 , further including logic that calculates lengths of time for which items are in the gaze directions of a particular identified subject during the particular identified subject's presence in the area of real space.

6. The system of claim 5 , further including logic that determines locations in the area of real space at which a particular item is in the gaze directions of the identified subjects and displays a view of the locations in the area of real space where the particular item was in the gaze directions of the identified subjects.

7. The system of claim 4 , further including logic that calculates distances of the identified subject from items having locations matching the identified gaze directions and said logic that determined the lengths of time utilizes the calculated distances.

8. The system of claim 7 , wherein the logic that calculates the distances of the identified subject from items in the gaze directions uses a location of a neck joint of the identified subject and a location of inventory display structures in which the items are positioned.

9. The system of claim 1 , further including logic to update a store planogram using the heat map of the area of real space and use the updated store planogram to stock items in the area of real space.

10. A method of detecting gaze direction of subjects in an area of real space, the method including:
receiving a plurality of sequences of frames of corresponding fields of view in the area of real space;
using the sequences of frames in the plurality of sequences of frames to identify locations of an identified subject and gaze directions of the identified subject in the area of real space over time;
accessing a database identifying locations of items in the area of real space, and using the locations to identify items in the area of real space matching the identified gaze directions of the identified subject;
storing the identified items matching the identified gaze directions of the identified subject as an entry in an item-subject correlations database and accumulating entries in the item-subject correlations database over a period of time for identified subjects, the entry in the accumulated entries including an item identifier, a subject identifier, and a location of the item in the area of real space; and
generating a heat map of the area of real space, wherein the heat map identifies locations in the area of real space, wherein the locations identified by the heat map have a number of directional impressions that are above a threshold indicating an interest in items positioned at locations in the area of real space.

11. The method of claim 10 , further including calculating a unique visitor directional impression from the accumulated entries in the item-subject correlations database, wherein the unique visitor directional impression identifies a number of unique or distinct subjects who looked at the item identified by the item identifier.

12. The method of claim 11 , further including calculating unique visitor directional impressions over a period of time.

13. The method of claim 11 , further including determining lengths of time for which the identified subject maintains respective gaze directions and storing the determined lengths of time in respective entries in the accumulated entries storing the identified items matching the identified gaze directions of the identified subject.

14. The method of claim 13 , further including calculating lengths of time for which items are in the gaze directions of a particular identified subject during the particular identified subject's presence in the area of real space.

15. The method of claim 14 , further including determining locations in the area of real space at which a particular item is in the gaze directions of the identified subjects and displaying a view of the locations in the area of real space where the particular item was in the gaze directions of the identified subjects.

16. The method of claim 13 , further including calculating distances of the identified subject from items having locations matching the identified gaze directions and said determining of the lengths of time utilizing the calculated distances.

17. The method of claim 16 , wherein the calculating of the distances of the identified subject from items in the gaze directions uses a location of a neck joint of the identified subject and a location of inventory display structures in which the items are positioned.

18. The method of claim 10 , further including updating a store planogram using the heat map of the area of real space and using the updated store planogram to stock items in the area of real space.

19. A non-transitory computer readable storage medium impressed with computer program instructions to detect gaze direction of subjects in an area of real space, the instructions, when executed on a processor, implement a method comprising:
receiving a plurality of sequences of frames of corresponding fields of view in the area of real space;
using the sequences of frames in the plurality of sequences of frames to identify locations of an identified subject and gaze directions of the identified subject in the

Assignees

• Standard Cognition Corp

Inventors

• Fisher Jordan E
• Suswal Michael S
• Green Warren

Classifications

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06F16/53

  Querying · CPC title

• G06V40/197

  Matching; Classification · CPC title

• G06T7/70 Primary

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title