User pose and item correlation

Patent metadata

Field	Value
Publication number	US-10460468-B2
Application number	US-201715844179-A
Country	US
Kind code	B2
Filing date	Dec 15, 2017
Priority date	Dec 15, 2017
Publication date	Oct 29, 2019
Grant date	Oct 29, 2019

Abstract

In aspects of user pose and item correlation, a mobile device includes a correlation module that receives an indication a user of the mobile device has positioned a hand proximate an item based on a pose of the user. The mobile device includes wireless radio systems to scan for wireless-enabled devices corresponding to items tagged with the wireless-enabled devices. The correlation module of the mobile device is implemented to receive the indication that the hand of the user of the mobile device is positioned proximate the item, and initiate a scan from the mobile device for the wireless-enabled devices proximate the user based on the indication of the hand of the user positioned proximate the item. The correlation module receives identifying data of the item that is proximate the hand of the user responsive to the scan of the wireless-enabled devices, and can correlate the item with the user.

First claim

The invention claimed is:

1. A method, comprising:
determining that a user of a mobile device has positioned a hand proximate an item based on a pose of the user;
scanning from the mobile device for wireless-enabled devices proximate the user based on the determining that the hand of the user is positioned proximate the item;
receiving identifying data of the item that is proximate the hand of the user responsive to the scanning;
detecting that the user has contacted the item based on skeletal tracking data; and
determining that the user put the item down back where the item was contacted based on a lack of detected head movement of the user.

2. The method as recited in claim 1 , wherein the determining that the hand of the user is positioned proximate the item comprises detecting the pose of the user based on a pose estimation that indicates the user is holding the item and has a head orientation towards the item.

3. The method as recited in claim 2 , wherein the pose estimation is based on captured digital images of the user.

4. The method as recited in claim 1 , wherein the determining that the hand of the user is positioned proximate the item comprises detecting a pickup gesture by the user based on the skeletal tracking data that indicates the pose and head movement in a direction towards the item.

5. The method as recited in claim 4 , wherein the skeletal tracking data is derived from captured digital images of the user moving to pick up the item.

6. The method as recited in claim 1 , wherein the correlating the item with the user is based on an identifier of the mobile device that is associated with the user.

7. The method as recited in claim 1 , further comprising:
receiving the identifying data corresponding to additional items proximate the user in a region where the hand of the user is positioned proximate the item; and
correlating the additional items with the user.

8. The method as recited in claim 1 , further comprising:
determining that the user put the item down for portability based on detected head movement of the user to a head orientation that is different than an initial head orientation when the item was contacted.

9. The method as recited in claim 1 , further comprising:
detecting that the user has moved location to a region other than where the user contacted the item;
rescanning from the mobile device for the wireless-enabled devices proximate the user;
receiving the identifying data of the item contacted by the user responsive to the rescanning at the moved location; and
determining that the user has retained possession of the item based on the receiving the identifying data of the item at the moved location.

10. The method as recited in claim 1 , wherein:
a wireless-enabled device that corresponds to the picked up item is a RFID tag that identifies the item; and
the scanning from the mobile device comprises interrogating the RFID tag by a RFID reader of the mobile device to request the identifying data stored on the RFID tag.

11. A mobile device, comprising:
wireless radio systems to scan for wireless-enabled devices corresponding to items that are tagged with the wireless-enabled devices;
a correlation module implemented at least partially in hardware with a processor to:
receive an indication that a user of the mobile device has positioned a hand proximate an item based on a pose of the user;
initiate a scan from the mobile device for the wireless-enabled devices proximate the user based on the indication that the hand of the user is positioned proximate the item;
receive identifying data of the item that is proximate the hand of the user responsive to the scan of the wireless-enabled devices;
receive another indication that the user has contacted the item based on skeletal tracking data; and
determine that the user put the item down for portability based on detected head movement of the user to a head orientation that is different than an initial head orientation when the item was contacted.

12. The mobile device as recited in claim 11 , wherein:
the indication that the hand of the user is positioned proximate the item is based on detection of the pose from a pose estimation that indicates the user is holding the item and has a head orientation towards the item; and
the pose estimation is based on captured digital images of the user.

13. The mobile device as recited in claim 11 , wherein:
the indication that the hand of the user is positioned proximate the item is based on detection of a pickup gesture by the user from the skeletal tracking data that indicates the pose and head movement in a direction towards the item; and
the skeletal tracking data is derived from captured digital images of the user moving to pick up the item.

14. The mobile device as recited in claim 11 , wherein the correlation module is implemented to:
receive the identifying data corresponding to additional items proximate the user in a region where the hand of the user is positioned proximate the item; and
correlate the additional items with the user.

15. The mobile device as recited in claim 11 , wherein the correlation module is implemented to:
determine that the user put the item down back where the item was contacted based on a lack of detected head movement of the user.

16. The mobile device as recited in claim 11 , wherein the correlation module is implemented to:
determine that the user has moved location to a region other than where the user contacted the item;
initiate a rescan from the mobile device for the wireless-enabled devices proximate the user;
receive the identifying data of the item contacted by the user responsive to the rescan at the moved location; and
determine that the user has retained possession of the item based on the identifying data of the item received at the moved location.

17. The mobile device as recited in claim 11 , wherein:
the wireless-enabled device that corresponds to the item is a RFID tag that identifies the item; and
the mobile device further comprising a RFID reader to interrogate the RFID tag as the scan to request the identifying data stored on the RFID tag.

18. A system, comprising:
a server computing device to determine that a hand of a person is positioned proximate an item based on a pose of the person;
a mobile device associated with the person, the mobile device configured to:
receive a communication from the server computing device indicating that the hand of the person is positioned proximate the item;
initiate a scan for wireless-enabled devices proximate the person based on the communication from the server computing device;
receive identifying data of the item that is proximate the hand of the person responsive to the scan of the wireless-enabled devices;
correlate the item with the person;
determine that the person has moved location to a region other than where the person contacted the item; and
determine that the person has retained possession of the item based on the identifying data of the item received at the moved location.

19. The system as recited in claim 18 , wherein the server computing device determines that the hand of the person is positioned proximate the item based on detection of the pose from skeletal tracking data that is derived from captured digital images of the person moving to pick up the item.

20. The system as recited in claim 19 , wherein the mobile device is configured to:
initiate a rescan for the wireless-enabled devices proximate the person responsive to the

Assignees

• Motorola Mobility Llc

Inventors

• Song Miao
• Wang Yunming

Classifications

• G06K7/10009

  sensing by radiation using wavelengths larger than 0.1 mm, e.g. radio-waves or microwaves · CPC title

• G06T2207/30244

  Camera pose · CPC title

• G06Q30/0613

  using intermediate agents · CPC title

• G06K7/10297

  arrangements for handling protocols designed for non-contact record carriers such as RFIDs NFCs, e.g. ISO/IEC 14443 and 18092 (cryptographic protocols H04L9/00; network security protocols H04L63/00; real-time communication protocols in data switching networks H04L65/00; network protocols for data switching network services H04L67/00) · CPC title

• G06T2207/30196

  Human being; Person · CPC title