System and method for providing guidance or feedback to a user

Patent metadata

Field	Value
Publication number	US-10521669-B2
Application number	US-201615351321-A
Country	US
Kind code	B2
Filing date	Nov 14, 2016
Priority date	Nov 14, 2016
Publication date	Dec 31, 2019
Grant date	Dec 31, 2019

Abstract

A device for providing guidance or feedback to a user. The device includes a camera configured to detect image data indicating a user performance of an activity. The system includes a guidance unit connected to the camera. The guidance unit is configured to identify the activity based on image processing of the image data or an identification of the activity from the user. The guidance unit is also configured to determine a criteria associated with the activity. The guidance unit is also configured to determine a user performance of the activity based on the image data. The guidance unit is also configured to determine feedback based on a comparison of the criteria and the user performance of the activity, the feedback indicating an improvement or suggestion for the user. The system also includes an output unit connected to the guidance unit, the output unit configured to output the feedback.

First claim

What is claimed is:

1. A wearable neck device for providing guidance or feedback to a user, the wearable neck device comprising:
a body having a neck portion configured to rest against a back of a neck of the user, a first side portion connected to the neck portion, and a second side portion connected to the neck portion, the first side portion configured to extend across a first shoulder of the user and rest on a front body portion of the user and the second side portion configured to extend across a second shoulder of the user and rest on the front body portion of the user;
a camera located on the first side portion or the second side portion and configured to detect image data of a first angle of a user performance of an activity;
a GPS unit located within the body and configured to detect location data;
a memory located within the body and configured to store a learned model associated with the activity;
an activity detection unit located in the body, connected to the camera and the GPS unit, and configured to automatically analyze the image data to identify a presence of one or more objects associated with the activity, compare the one or more objects associated with the activity to the learned model and identify the activity based on the location data and the comparison of the one or more objects associated with the activity to the learned model;
a guidance unit located in the body, connected to the camera and the IMU, and configured to:
obtain, from a second camera of another device, additional image data of a second angle of the user performance of the activity,
determine a series of instructions associated with the activity and including a plurality of steps to be performed by the user based on the learned model,
determine an action being performed within the detected image data,
determine a current step of the plurality of steps of the series of instructions associated with the activity that is being performed based on the action,
determine that the current step of the activity has been completed based on the image data of the first angle of the user performance of the activity and the additional image data of the second angle of the user performance of the activity, and
determine an instruction associated with a next step within the series of instructions in response to determining that the current step has been completed; and

an output unit located on the body, connected to the guidance unit, and configured to output the instruction to the user.

2. The wearable neck device of claim 1 , further comprising an input unit configured to detect input data from the user identifying the activity.

3. The wearable neck device of claim 1 , wherein the learned model stored in the memory is periodically updated.

4. The wearable neck device of claim 1 , wherein the output unit includes a speaker configured to provide an audio output or a vibration unit configured to provide a tactile output.

5. The wearable neck device of claim 1 , wherein the one or more objects identified by the activity detection unit are one or more stationary objects unconnected to the user.

6. A wearable neck device for providing guidance or feedback to a user, the wearable neck device comprising:
a body having a neck portion configured to rest against a back of a neck of the user, a first side portion connected to the neck portion, and a second side portion connected to the neck portion, the first side portion and the second side portion configured to extend across a shoulder of the user and rest on a front body portion of the user;
a camera located on the first side portion or the second side portion and configured to detect image data of a first angle of a user performance of an activity;
an inertial measurement unit (IMU) located within the body and configured to detect movement data associated with the user during the user performance of the activity;
a GPS unit located within the body and configured to detect location data;
an activity detection unit connected to the camera and the GPS unit, and configured to:
automatically analyze the image data to identify the activity based on a presence of one or more objects associated with the activity, and
identify the activity based on the presence of the one or more objects associated with the activity and the location data;

a guidance unit connected to the camera and the IMU, and configured to:
obtain, from a second camera of another device, additional image data of a second angle of the user performance of the activity,
determine a series of instructions associated with the activity and including a plurality of steps to be performed by the user,
determine an action being performed within the image data,
determine a current step of the plurality of steps of the series of instructions associated with the activity that is being performed based on the image data, the additional image data and the movement data,
determine that the current step of the activity has been completed based on the image data of the first angle of the user performance of the activity and the additional image data of the second angle of the user performance of the activity, and
determine a instruction associated with a next step within the series of instructions; and

an output unit connected to the guidance unit, the output unit configured to output the instruction.

7. The wearable neck device of claim 6 , further comprising an input unit configured to detect input data from the user indicating the activity.

8. The wearable neck device of claim 6 , further comprising:
a memory configured to store a learned model;
wherein the guidance unit is configured to determine that the current step of the activity has been completed by comparing the image data to the learned model stored in the memory.

9. The wearable neck device of claim 6 , wherein the output unit includes a speaker configured to provide an audio output or a vibration unit configured to provide a tactile output.

10. A method of providing guidance or feedback to a user of a wearable neck device, the method comprising:
providing, by the wearable neck device, a body having a neck portion configured to rest against a back of a neck of the user, a first side portion connected to the neck portion, and a second side portion connected to the neck portion, the first side portion configured to extend across a first shoulder of the user and rest on a front body portion of the user and the second side portion configured to extend across a second shoulder of the user and rest on the front body portion of the user;
detecting, by a camera located on the first side portion or the second side portion of the body of the wearable neck device, image data of a first angle of a user performance of an activity;
detecting, by a GPS unit, location data associated with the wearable neck device;
storing, by a memory located within the body, a learned model associated with the activity;
automatically analyzing, by an activity detection unit, the image data to identify a presence of one or more objects;
identifying, by the activity detection unit, the presence of the one or more objects associated with the activity;
comparing, by the activity detection unit, the one or more objects associated with the activity to the learned model;
identifying, by the activity detection unit, the activity based on the location data and the comparison of the one or more objects associated with the activity to the learned model;
obtaining, by a guidance unit and from a second camera of another device, additional image data of a second angle of the user performance of the activity;
determining, by the guidance unit, a series of instructions associated with the activity

Assignees

• Toyota Eng & Mfg North America

Inventors

• Dayal Rajiv
• Chen Tiffany L
• Ching Patrick K
• Northcutt Brandon D

Classifications

• G09B19/0069

  Engineering, e.g. mechanical, electrical design (computer engineering G09B19/0053) · CPC title

• G06V10/70

  using pattern recognition or machine learning (optical pattern recognition or electronic computations therefor G06V10/88) · CPC title

• G06V20/35

  Categorising the entire scene, e.g. birthday party or wedding scene · CPC title

• G06F3/016 Primary

  Input arrangements with force or tactile feedback as computer generated output to the user · CPC title

• A63B24/0003

  Analysing the course of a movement or motion sequences during an exercise or trainings sequence, e.g. swing for golf or tennis · CPC title