Estimating posture using head-mounted cameras

Patent metadata

Field	Value
Publication number	US-10165949-B2
Application number	US-201615182592-A
Country	US
Kind code	B2
Filing date	Jun 14, 2016
Priority date	Jun 14, 2015
Publication date	Jan 1, 2019
Grant date	Jan 1, 2019

Abstract

System and method for estimating posture of a user based on a model and images captured by at least two head-mounted cameras that are worn by the user and are oriented downward such that portions of the user's torso are in the respective fields of view of the cameras when the user stands up straight.

First claim

What is claimed is:

1. A system comprising:
a frame configured to be worn on a user's head;
first and second cameras, physically coupled to the frame at locations that are to the right and to the left of a symmetry axis that divides the user's face to its right and left sides, respectively, and less than 15 cm away from the user's head;
wherein the first and second cameras are oriented downward such that portions of the user's torso are in respective fields of view of the first and second cameras when the user stands upright; and
a computer configured to estimate an angle between the user's head and the user's torso based on data received from the first and second cameras and orientation of the frame.

2. The system of claim 1 , wherein the computer is further configured to estimate posture of the user based on a model and the data received from the first and second cameras, and to train the model based on data received from the first and second cameras;
wherein the training of the model is performed under the assumption that the locations of the first and second cameras, relative to the user's head, do not change by more than 5° even when the user's head performs wide angular and lateral movements.

3. The system of claim 2 , wherein the first and second cameras are coupled to the frame at locations that are less than 5 cm away from the user's head, and further comprising an inertial measurement unit (IMU) coupled to the frame and configured to measure orientation of the frame;
wherein the computer is further configured to utilize the orientation for training the model.

4. The system of claim 2 , further comprising an external camera, which is not physically coupled to the frame, having a direct line of sight to the front side of the user's torso;
wherein the computer is further configured to utilize data from the external camera to train the model.

5. The system of claim 1 , wherein the first camera points at least 20° to the right of the symmetry axis, and the second camera points at least 20° to the left of the symmetry axis; and
wherein the first and second cameras are coupled to the frame at locations that are less than 5 cm away from the user's head.

6. The system of claim 1 , further comprising a third camera, physically coupled to the frame at a location behind an ear of the user;
wherein the third camera is oriented downwards such that a portion of the back of the user's torso is in field of view of the third camera when the user stands upright; and
wherein the computer is configured to estimate posture of the user based on data received from the first, second and third cameras.

7. The system of claim 6 , wherein the third camera is mounted on an extender that helps to prevent concealment of the portion of the user's torso by at least one of the user's head and hair.

8. The system of claim 1 , further comprising third and fourth cameras, physically coupled to the right and left sides of the frame, respectively, at locations that enable them to capture the user's hands when stretched horizontally to the sides;
wherein the computer is configured to estimate posture of the user based on data received from the first, second, third and fourth cameras.

9. The system of claim 1 , further comprising a third camera, physically coupled to the frame and oriented such that at least a portion of at least one of the user's eyebrows is in the field of view of the third camera;
wherein the computer is further configured to estimate a facial expression of the user based on a model of a human face parameterized by expression and data extracted from images captured by the third camera.

10. The system of claim 9 , wherein the computer is further configured to estimate posture of the user based on a model and data received from the first and second cameras, and to render an avatar of the user based on the posture and the facial expression.

11. The system of claim 10 , wherein the frame belongs to an augmented reality device or a virtual reality device, and the computer is further configured to show the user his/her own avatar;
whereby seeing his/her own avatar helps the user to understand how he/she looks to others.

12. The system of claim 10 , wherein the frame belongs to an augmented reality device or a virtual reality device, and the computer is further configured to show the user at least two of his/her own avatars who imitate the user by doing what the user does.

13. The system of claim 9 , wherein the computer is further configured to estimate posture of the user based on a model and data received from the first and second cameras, the frame belongs to an augmented reality device, and further comprising a communication module configured to send a representation of the posture and the facial expression to a second computer configured to render an avatar of the user;
wherein the second computer is further configured to receive additional measurements of the user from a second augmented reality device that is worn by a second user located nearby the user; and
the second computer is further configured to render, for the second user, an avatar of the user based on: the additional measurements, the posture, and the facial expression.

14. The system of claim 1 , wherein the first and second cameras are light field cameras.

15. A method comprising: receiving images from first and second head mounted cameras that are located to the right and to the left of a symmetry axis that divides a user's face to its right and left sides, respectively;
wherein the first and second head mounted cameras physically coupled to a frame worn on the user's head, are located less than 15 cm away from the user's head, and are oriented downward such that portions of the user's torso are in respective fields of view of the first and second head mounted cameras when the user stands upright; and
estimating an angle between the user's head and the user's torso based on the data received from the first and second cameras and orientation of the frame.

16. The method of claim 15 , further comprising training a model based on the data received from the first and second head mounted cameras;
wherein the training of the model is performed under the assumption that the locations of the first and second head mounted cameras, relative to the user's head, do not change by more than 5° even when the user's head performs wide angular and lateral movements.

17. The method of claim 15 , further comprising a third head mounted camera oriented such that at least a portion of at least one of the user's eyebrows is in the field of view of the third head mounted camera; and further comprising
estimating a facial expression of the user based on a model of a human face parameterized by expression and data extracted from images captured by the third head mounted camera.

18. The method of claim 17 , further comprising estimating posture of the user based on a model and data received from the first and second cameras, and rendering an avatar of the user based on the posture and the facial expression.

19. The method of claim 15 , wherein the first and second head mounted cameras belong to an augmented reality device or a virtual reality device, and further comprising showing the user his/her own avatar from the side;
whereby seeing his/her own avatar helps the user to understand how he/she looks to others.

Assignees

• Facense Ltd

Inventors

• Tzvieli Arie
• Frank Ari M
• Thieberger Gil

Classifications

• A61B2576/00

  Medical imaging apparatus involving image processing or analysis (A61B1/00009, A61B6/52 and A61B8/52 take precedence) · CPC title

• A61B5/163

  by tracking eye movement, gaze, or pupil change · CPC title

• A61B5/6803

  Head-worn items, e.g. helmets, masks, headphones or goggles · CPC title

• A61B5/0077

  Devices for viewing the surface of the body, e.g. camera, magnifying lens · CPC title

• A61B5/165

  Evaluating the state of mind, e.g. depression, anxiety · CPC title