Activity assistance system

Patent metadata

Field	Value
Publication number	US-11037670-B2
Application number	US-202017023577-A
Country	US
Kind code	B2
Filing date	Sep 17, 2020
Priority date	Sep 17, 2019
Publication date	Jun 15, 2021
Grant date	Jun 15, 2021

Abstract

An activity assistance system includes a video camera arranged to acquire video of a person performing an activity, an output device configured to output human-perceptible prompts, and an electronic processor programmed to execute an activity script. The script comprises a sequence of steps choreographing the activity. The execution of each step includes presenting a prompt via the output device and detecting an event or sequence of events subsequent to the presenting of the prompt. Each event is detected by performing object detection on the video to detect one or more objects depicted in the video and applying one or more object-oriented image analysis functions to detect a spatial or temporal arrangement of one or more of the detected objects. Each event detection triggers an action comprising at least one of presenting a prompt via the output device and and/or going to another step of the activity script.

First claim

The invention claimed is:

1. An activity assistance system comprising:
a video camera arranged to acquire video of a person performing an activity;
an output device configured to output human-perceptible prompts; and
an electronic processor programmed to execute an activity script comprising a sequence of steps choreographing the activity wherein the execution of each step includes presenting a prompt via the output device and detecting an event or sequence of events subsequent to the presenting of the prompt;
wherein each event is detected by performing object detection on the video to detect one or more objects depicted in the video delineated by bounding boxes or by pixel boundaries of the respective one or more objects and applying one or more object-oriented image analysis functions to detect a spatial or temporal arrangement of one or more of the detected objects; and
each event detection triggers an action comprising at least one of presenting a prompt via the output device and and/or going to another step of the activity script.

2. The activity assistance system of claim 1 wherein the activity script choreographs an activity of daily living (ADL) and steps of the activity script include presenting a congratulatory prompt when an event detection indicates the step is successfully completed.

3. The activity assistance system of claim 1 wherein the activity script choreographs a rehabilitation activity and steps of the activity script include presenting a congratulatory prompt when an event detection indicates the step is successfully completed.

4. The activity assistance system of claim 1 wherein the activity script includes a step which includes:
presenting a prompt via the output device asking a person to place a set of objects in a correct order; and
applying an object order detection function to detect whether a spatial order of the set of objects is in the correct order;
wherein detection by the order detection function that the spatial order of the set of objects is not in the correct order triggers presenting a prompt indicating the incorrect order and asking that the order be corrected.

5. The activity assistance system of claim 4 wherein the output device comprises a display and the prompt asking the person to place the set of objects in a correct order includes displaying an image or graphical representation of the set of objects in the correct order on the display.

6. The activity assistance system of claim 1 wherein the activity script includes a step which includes:
presenting a prompt via the output device asking a person to place a specified object in a specified location; and
applying an object location in-range function to detect an object in the specified location;
wherein detection by the object in-range function that an object other than the specified object is in the specified location triggers presenting a prompt indicating the incorrect object has been placed in the specified location and asking that the specified object be placed in the specified location.

7. The activity assistance system of claim 6 wherein the output device comprises a display and the prompt asking the person to place the specified object in the specified location includes displaying an image or graphical representation of the specified location with the specified object in the specified location on the display.

8. The activity assistance system of claim 1 wherein the activity script includes a step which includes:
presenting a prompt via the output device asking a person to dispense a substance onto a specified object; and
applying an object overlap detection function to detect the substance overlapping an object;
wherein detection by the object overlap function that the substance overlaps an object other than the specified object triggers presenting a prompt indicating the substance has been applied to an incorrect object and asking that the substance be applied to the specified object.

9. The activity assistance system of claim 8 wherein the output device comprises a display and the prompt asking the person to dispense the substance onto the specified object includes displaying an image or graphical representation of the substance dispensed onto the specified object on the display.

10. The activity assistance system of claim 1 wherein the activity script includes a step which includes:
presenting a prompt via the output device asking a person to cause an interaction of a first object and a second object; and
applying an object overlap detection function to detect whether the first object and the second object overlap;
wherein detection by the object overlap function that the first object and the second object overlap triggers presenting a congratulatory prompt.

11. The activity assistance system of claim 10 wherein the output device comprises a display and the prompt asking the person to dispense the substance onto the specified object includes displaying an image or graphical representation of the interaction of the first object and the second object.

12. The activity assistance system of claim 1 wherein the video camera comprises a video camera of smart glasses or a webcam of a computer.

13. The activity assistance system of claim 1 wherein each event is detected by performing object detection on the video to detect one or more objects depicted in the video delineated by bounding boxes.

14. The activity assistance system of claim 1 wherein each event is detected by performing object detection comprising instance segmentation on the video to detect one or more objects depicted in the video delineated by pixel boundaries of the respective one or more objects.

15. The activity assistance system of claim 1 wherein the output device includes at least one of a display of a computer, a display of smart glasses, a loudspeaker of the computer, and/or a loudspeaker of the smart glasses.

16. An activity assistance system comprising:
a video camera arranged to acquire video of a person performing an activity;
an output device configured to output human-perceptible prompts; and
an electronic processor programmed to:
execute an activity script comprising a sequence of steps choreographing the activity wherein the execution of each step includes presenting a prompt via the output device and detecting an event or sequence of events subsequent to the presenting of the prompt;
track detected events indicating mistakes by the person in performing the activity; and
upon completion of the execution of the activity script, presenting a performance report including metrics of the performance of the activity determined from the tracked events;

wherein each event is detected by performing object detection on the video to detect one or more objects depicted in the video and applying one or more object-oriented image analysis functions to detect a spatial or temporal arrangement of one or more of the detected objects; and
each event detection triggers an action comprising at least one of presenting a prompt via the output device and and/or going to another step of the activity script.

17. The activity assistance system of claim 16 wherein the electronic processor is further programmed to generate a quantitative traumatic brain injury (TBI) diagnosis based on the metrics of the performance of the activity determined from the tracked events.

18. The activity assistance system of claim 16 wherein the electronic processor is further programmed to:
calculate a reaction time for an event as a time between presenting a prompt to engage an obje

Assignees

• Battelle Memorial Institute

Inventors

• Colachis Samuel
• Sharma Gaurav
• Dunlap Collin

Classifications

• G16H20/70 Primary

  relating to mental therapies, e.g. psychological therapy or autogenous training · CPC title

• A61B5/1124 Primary

  Determining motor skills · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V20/52

  Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) · CPC title

• G06V20/00 Primary

  Scenes; Scene-specific elements (control of digital cameras H04N23/60) · CPC title