Identifying object of interest for handicapped individuals based on eye movement patterns

Patent metadata

Field	Value
Publication number	US-12165192-B2
Application number	US-202318539687-A
Country	US
Kind code	B2
Filing date	Dec 14, 2023
Priority date	Mar 25, 2020
Publication date	Dec 10, 2024
Grant date	Dec 10, 2024

Abstract

In some embodiments, apparatuses and methods are provided to support individuals in a retail environment. Some embodiments, provide systems, comprising: a plurality of cameras positioned in a retail store; an image processing control circuit configured to determine a customer field of view; a product identification control circuit configured to identify a set of multiple products within the identified customer field of view, assign a viewing probability to each product without the first customer having to touch the first product; a product selection control circuit configured to determine a purchase probability of the first product; add the first product to a virtual electronic cart when the purchase probability of the first product exceeds a purchase probability threshold; and communicate instructions to control the worker personal computing device to instruct a worker to retrieve the first product.

First claim

What is claimed is:

1. A system comprising:
a plurality of cameras distributed along each aisle of a plurality of aisles of a retail environment, wherein each camera of the plurality of cameras is oriented to capture images relative to users moving along a respective aisle of the plurality of aisles;
an image processing control circuit communicatively coupled with each of the plurality of cameras, wherein the image processing control circuit is configured to process the images from each of the plurality of cameras, determine a reliability confidence of each camera of a subset of cameras of the plurality of cameras, and determine a user field of view of a first user within a camera field of view of one or more of the cameras of the subset of cameras based on multiple images from the subset of cameras, an orientation of each camera of the subset of cameras, the camera field of view of each camera of the subset of cameras and the reliability confidences of each camera of the subset of cameras;
a product identification control circuit communicatively coupled with the image processing control circuit, wherein the product identification control circuit is configured to identify, based on product location information, a set of multiple products within the user field of view, assign a viewing probability to each product of the set of the multiple products as a function of a distance from a determined center of the user field of view and identify a first product of the set of multiple products that the first user is viewing, prior to the first user touching the first product, as a function of the viewing probability and a user history associated with the first user;
a product selection control circuit communicatively coupled with the product identification control circuit, wherein the product selection control circuit is configured to:
determine a purchase probability that the first product is a product the first user intends to purchase as a function of the user history associated with the first user and prior to the first user touching the first product;
add the first product to a virtual electronic cart as a selected set of one or more products that the first user is predicted to intend to purchase when the purchase probability of the first product exceeds a purchase probability threshold; and
control a worker computing device to instruct a first worker to retrieve the first product for purchase by the first user and to be supplied to the first user.

2. The system of claim 1 , wherein the image processing control circuit, in determining the user field of view of the first user, is configured to track eye movement of the first user over time and determine the user field of view of the first user as a function of the tracked eye movement over time.

3. The system of claim 1 , wherein the image processing control circuit, in determining the user field of view of the first user, is configured to track user position and/or user orientation over time and determine the user field of view of the first user as a function of the tracked user position and/or user orientation over time.

4. The system of claim 1 , wherein the image processing control circuit, in determining the reliability confidence of each camera of the subset of cameras, is configured to identify an orientation of the first user relative to the camera field of view of each of the cameras of the subset of cameras, and assign the reliability confidence of each camera of the subset of cameras as a function of the orientation of the first user relative to the camera field of view of a respective one of the cameras of the subset of cameras.

5. The system of claim 4 , wherein the image processing control circuit is further communicatively coupled with an optical head mounted display system that is at least temporarily associated with the first user, wherein the image processing control circuit receives eye tracking information from the optical head mounted display system;
wherein the image processing control circuit, in determining the user field of view of the first user, is configured to determine the user field of view based on the orientations of each camera of the subset of cameras, the fields of view of each camera of the subset of cameras, the reliability confidences of each camera of the subset of cameras and the eye tracking information relative to a location of the first user.

6. The system of claim 1 , wherein the image processing control circuit is further configured to identify facial features of the first user from one or more images from at least one of the cameras of the subset of cameras, identify the first user within a respective aisle of the plurality of aisles based at least on the facial features, access a user profile associated with the first user based on the identification of the first user, and access the user history through the user profile associated with the first user.

7. The system of claim 1 , wherein the image processing control circuit in determining the user field of view is configured to determine, based on image processing of the multiple images from the cameras of the subset of cameras, a head orientation of a head of the first user and a facial orientation of a face of the first user, and determine the user field of view as a function of the head orientation and the facial orientation.

8. The system of claim 1 , wherein the product selection control circuit, in determining the purchase probability that the first user intends to purchase the first product, obtains a duration of a gaze of the first user on the first product, obtains estimated emotions of the first user based on facial expressions occurring within a first threshold time of the duration of the gaze of the first user on the first product, and obtains body language data occurring within a second threshold time of the duration of the gaze of the first user on the first product; and
determine the purchase probability as a function of the user history associated with the first user, the duration of the gaze, the estimated emotions, and the body language.

9. The system of claim 8 , further comprising a model training control circuit configured to receive notification that the first user declined to actually purchase the first product, and adjust a modeling relative to the first user based at least on the received notification and the purchase probability of the first product.

10. The system of claim 1 , wherein the product selection control circuit is further configured to determine a quantity of the first product predicted to be purchased by the first user based on the user history of the first user, and in communicating the instructions to the worker computing device is further configured to instruct the quantity of the first product and control the worker computing device to instruct the first worker to retrieve the quantity of the first product for purchase by the first user.

11. A method comprising:
processing, by an image processing control circuit, images from each of a plurality of cameras, wherein the plurality of cameras are positioned and distributed along both sides of each aisle of a plurality of aisles of a retail environment, wherein each camera of the plurality of cameras is oriented to capture images relative to customers moving along a respective one of the plurality of aisles;
determining a reliability confidence of each camera of a set of multiple cameras of the plurality of cameras;
determining a customer field of view of a first customer within a camera field of view of one or more of the cameras of the set of multiple cameras based on multiple images from the set of multiple cameras of the plurality of cameras, an orientation of each camer

Assignees

• Walmart Apollo Llc

Inventors

• Aggarwal Parul
• Kumar Rahul
• Wadhonkar Mangesh N Kulkarni
• Jhunjhunwala Amit
• Mishra Rajiv

Classifications

• G06V40/19

  Sensors therefor · CPC title

• G06V20/52

  Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) · CPC title

• G06V10/25

  Determination of region of interest [ROI] or a volume of interest [VOI] · CPC title

• G06Q30/0633

  Managing shopping lists, e.g. compiling or processing purchase lists (shipping orders G06Q10/083; order filling G06Q10/087) · CPC title

• G06V40/174

  Facial expression recognition · CPC title