Method and system for reconstructing obstructed face portions for virtual reality environment

Patent metadata

Field	Value
Publication number	US-10445863-B2
Application number	US-201515501160-A
Country	US
Kind code	B2
Filing date	Aug 4, 2015
Priority date	Aug 4, 2014
Publication date	Oct 15, 2019
Grant date	Oct 15, 2019

Abstract

A method and a system for reconstructing obstructed face portions are provided herein. The method may include the following steps: obtaining off-line 3D data, being 3D data of a head of a person not wearing a face-obstructing object, being an object which obstructs a portion of the face of the person; obtaining in real time, real-time 3D data, being 3D data of said head, wherein said person wears said face-obstructing object; applying a 3D transformation to at least a portion of the off-line 3D data, based on the real-time 3D data, to yield reconstructed real time 3D data, being real-time 3D data related to the obstructed face portions; and merging the reconstructed real time 3D data into the real-time 3D data. The system may implement the aforementioned steps over a computer processor.

First claim

The invention claimed is:

1. A method comprising:
capturing an image of a scene containing at least one person wearing a face-obstructing object;
obtaining, based on sensor data captured by a sensor on the face-obstructing object, data indicative of face portions obstructed by the face-obstructing object;
reconstructing a 3D face image of the obstructed face portions based on the obtained data; and
merging the reconstructed 3D face image into respective 3D image of the person in the captured scene.

2. The method according to claim 1 , further comprising presenting the merged image to the specified user via a display.

3. The method according to claim 1 , wherein the obtaining data indicative of face portions obstructed by the face-obstructing object is carried out by sensing the obstructed face portions from an inner side of said face-obstructing object.

4. The method according to claim 3 , wherein said sensing comprises image capturing and wherein said obtained data is an image of portions of the obstructed face.

5. The method according to claim 3 , wherein the said sensing is of facial gestures, and wherein the reconstructing a face 3D image is carried out by modeling sensed gestures to change the face relative to a base image of the obstructed face portions captured previously.

6. The method according to claim 1 , wherein the obtaining of the data indicative of face portions obstructed by the face-obstructing object is carried out by deriving an expression of the affecting the eyes.

7. The method according to claim 1 , wherein the obtaining of the data indicative of face portions obstructed by the face-obstructing object is carried out by deriving a context of the person in communication with another person.

8. The method according to claim 1 , wherein the obtaining data indicative of face portions obstructed by the face-obstructing object comprises obtaining a base image of the obstructed face portions, which has been previously captured.

9. The method according to claim 8 , wherein the reconstructing the 3D face image of the obstructed face portions is based on the obtained data carried out by monitoring at least one of: position, orientation, and profile; and applying the obtained data onto a three dimensional model of the face of the at least one person, to yield a modified face image.

10. The method according to claim 8 , wherein the reconstructing the 3D face image of the obstructed face portions is based on the obtained data carried out by monitoring vital signs of the at least one person wearing the face-obstructing object, and applying the obtained data onto a three dimensional model of the face of the at least one person, to yield a modified face image.

11. A system comprising:
an image capturing device configured to capture an image of a scene containing at least one person wearing a face-obstructing object; and
a computer processor configured to:
obtain, from a sensor on the face-obstructing object, data indicative of face portions obstructed by the face-obstructing object;
reconstruct a 3D face image of the obstructed face portions based on the obtained data; and
merge the reconstructed 3D face image into respective 3D image of the person in the captured scene.

12. The system according to claim 11 , further comprising a display configured to present the merged image.

13. The system according to claim 11 , wherein the obtaining data indicative of face portions obstructed by the face-obstructing object is carried out by sensors configured to sense the obstructed face portions from an inner side of said face-obstructing object.

14. The system according to claim 13 , wherein said sensing comprises image capturing and wherein said obtained data is an image of portions of the obstructed face.

15. The system according to claim 13 , wherein the said sensing is of facial gestures, and wherein the reconstructing a face 3D image is carried out by modeling sensed gestures to change the face relative to a base image of the obstructed face portions captured previously.

16. The system according to claim 11 , wherein the obtaining of the data indicative of face portions obstructed by the face-obstructing object is carried out by deriving an expression of the affecting the eyes.

17. The system according to claim 11 , wherein the obtaining of the data indicative of face portions obstructed by the face-obstructing object is carried out by deriving a context of the person in communication with another person.

18. The system according to claim 11 , wherein the obtaining data indicative of face portions obstructed by the face-obstructing object comprises obtaining a base image of the obstructed face portions, which has been previously captured.

19. The system according to claim 11 , wherein the reconstructing the 3D face image of the obstructed face portions is based on the obtained data carried out by monitoring at least one of: position, orientation, and profile; and applying the obtained data onto a three dimensional model of the face of the at least one person, to yield a modified face image.

20. The system according to claim 11 , wherein the reconstructing the 3D face image of the obstructed face portions is based on the obtained data carried out by monitoring vital signs of the at least one person wearing the face-obstructing object, and applying the obtained data onto a three dimensional model of the face of the at least one person, to yield a modified face image.

Assignees

• Facebook Tech Llc

Inventors

• Grossinger Nadav
• Alon Emil

Classifications

• G16B45/00

  ICT specially adapted for bioinformatics-related data visualisation, e.g. displaying of maps or networks · CPC title

• G06T2207/30201

  Face · CPC title

• G06T2200/04

  involving 3D image data · CPC title

• G06T7/30

  Determination of transform parameters for the alignment of images, i.e. image registration · CPC title

• G02B27/017

  Head mounted · CPC title