Activity assistance system

Patent metadata

Field	Value
Publication number	US-11798272-B2
Application number	US-202117333155-A
Country	US
Kind code	B2
Filing date	May 28, 2021
Priority date	Sep 17, 2019
Publication date	Oct 24, 2023
Grant date	Oct 24, 2023

Abstract

An activity assistance system includes a video camera arranged to acquire video of a person performing an activity, an output device configured to output human-perceptible prompts, and an electronic processor programmed to execute an activity script. The script comprises a sequence of steps choreographing the activity. The execution of each step includes presenting a prompt via the output device and detecting an event or sequence of events subsequent to the presenting of the prompt. Each event is detected by performing object detection on the video to detect one or more objects depicted in the video and applying one or more object-oriented image analysis functions to detect a spatial or temporal arrangement of one or more of the detected objects. Each event detection triggers an action comprising at least one of presenting a prompt via the output device and and/or going to another step of the activity script.

First claim

The invention claimed is:

1. An activity assistance system comprising:
a video camera arranged to acquire video of an activity being performed by a person;
an output device configured to output human-perceptible prompts; and
an electronic processor programmed to execute an activity script comprising a sequence of steps choreographing the activity wherein the execution of each step includes presenting a prompt via the output device and detecting an event or sequence of events subsequent to the presenting of the prompt;
wherein events are detected by performing object detection on the video to detect one or more objects depicted in the video and applying one or more object-oriented image analysis functions to detect a spatial or temporal arrangement of one or more of the detected objects; and
each event detection triggers an action comprising at least one of presenting a prompt via the output device and and/or going to another step of the activity script.

2. The activity assistance system of claim 1 further comprising:
a wireless tag with a radio frequency (RF) transmitter attached to at least one object of the one or more objects; and
an array of wireless RF receivers arranged to receive RF signals from the RF transmitter of the wireless tag;
wherein the electronic processor is programmed to compare differences in phase angles of the RF signals received at the wireless receivers and calculate a position of the at least one object from the differences in phase angles, at least one event being detected further based on the calculated position of the at least one object.

3. The activity assistance system of claim 2 wherein the wireless tag further includes an inertial measurement unit (IMU) and the electronic processor is programmed to determine an orientation of the at least one object based on readings of the IMU transmitted via the RF transmitter to the RF array of wireless receivers, at least one event being detected further based on the determined orientation of the at least one object.

4. The activity assistance system of claim 1 wherein the electronic processor is further programmed to determine an aspect of the person based on the video of the person performing the activity, wherein the electronic processor is programmed to execute the activity script based in part on the determined aspect of the person.

5. The activity assistance system of claim 1 further comprising a microphone, wherein the electronic processor is further programmed to execute the activity script based in part on audio acquired by the microphone.

6. The activity assistance system of claim 1 wherein the electronic processor is further programmed to detect assistance from a person other than the person performing the activity based on detection of a body part of the person other than the person performing the activity in the video of the person performing the activity, wherein the electronic processor is programmed to execute the activity script based in part on the detected assistance.

7. The activity assistance system of claim 1 wherein at least one event is detected further based on one or more of:
identifying attention of the person by applying an attention mask;
identifying attention of the person by applying gaze monitoring using the video camera;
identifying a facial expression of the person; and/or
performing voice semantic analysis using a microphone.

8. The activity assistance system of claim 1 wherein at least one event is detected by:
applying attention masks to at least one frame of the video to assess attention of the person to corresponding objects; and
identifying the at least one event based on a weighted combination of outputs of the applied attention masks.

9. An activity assistance method comprising:
using a video camera, acquiring video of an activity being performed by a person;
using an electronic processor, executing an activity script comprising a sequence of steps choreographing the activity wherein the execution of each step includes presenting a prompt via an output device and detecting an event or sequence of events subsequent to the presenting of the prompt, wherein each event is detected by performing object detection on the video to detect one or more objects depicted in the video and applying one or more object-oriented image analysis functions to detect a spatial or temporal arrangement of one or more of the detected objects; and
responsive to each event detection, performing an action comprising at least one of presenting a prompt via the output device and and/or going to another step of the activity script.

10. The activity assistance method of claim 9 further comprising:
tracking at least one object of the one or more objects using a wireless tag attached to the at least one object;
wherein the executing of the activity script is based in part on the tracking.

11. The activity assistance method of claim 10 wherein the wireless tag includes an inertial measurement unit (IMU) and the tracking of the at least one object includes tracking an orientation of the at least one object based on readings of the IMU wirelessly transmitted from the wireless tag attached to the at least one object.

12. The activity assistance method of claim 9 further comprising:
using the electronic processor, determining an aspect of the person based on the video of the person performing the activity, wherein the executing of the activity script is based in part on the determined aspect of the person.

13. The activity assistance method of claim 9 further comprising:
recording audio using a microphone;
wherein the executing of the activity script is based in part on the recorded audio.

14. The activity assistance method of claim 9 further comprising:
using the electronic processor, detecting assistance from a person other than the person performing the activity based on detection of a body part of the person other than the person performing the activity in the video of the person performing the activity, wherein the executing of the activity script is based in part on the detected assistance.

15. The activity assistance method of claim 9 wherein the video camera is mounted on a bathroom wall and the output device comprises a display integrated into a bathroom mirror.

16. The activity assistance method of claim 9 wherein the output device includes at least one of a display of a computer, a display of smart glasses, a loudspeaker of the computer, and/or a loudspeaker of the smart glasses.

17. A non-transitory storage medium storing instructions readable and executable by an electronic processor to perform an activity assistance method comprising:
receiving, from a video camera, video of an activity being performed by a person;
executing an activity script comprising a sequence of steps choreographing the activity wherein the execution of each step includes presenting a prompt via an output device comprising one or more of a display and/or a loudspeaker and detecting an event or sequence of events subsequent to the presenting of the prompt, wherein each event is detected by performing object detection on the video to detect one or more objects depicted in the video and applying one or more object-oriented image analysis functions to detect a spatial or temporal arrangement of one or more of the detected objects; and
responsive to each event detection, performing an action comprising at least one of presenting a prompt via the output device and and/or going to another step of the activity script.

18. The non-transitory storage medium of c

Assignees

• Battelle Memorial Institute

Inventors

• Colachis Samuel
• Sharma Gaurav
• Dunlap Collin
• Boyd Douglas
• Branch Joshua

Classifications

• G06V10/82 Primary

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06V20/00

  Scenes; Scene-specific elements (control of digital cameras H04N23/60) · CPC title

• G06V20/20

  in augmented reality scenes · CPC title

• G06V20/52

  Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) · CPC title