Depth data adjustment based on non-visual pose data

Patent metadata

Field	Value
Publication number	US-10992916-B2
Application number	US-201715659971-A
Country	US
Kind code	B2
Filing date	Jul 26, 2017
Priority date	Jul 26, 2017
Publication date	Apr 27, 2021
Grant date	Apr 27, 2021

Abstract

An HMD adjusts adjusting depth information based on detected motion of the system. The HMD includes a depth camera that collects depth data for objects in the local environment of the HMD. The HMD further includes an inertial measurement unit (IMU) including non-visual motion sensors such as one or more accelerometers, gyroscopes, and the like. The HMD adjusts the received depth information based on motion data provided by the IMU, thereby improving the accuracy of the depth information, and in turn reducing visual artifacts that can result from inaccuracies in the depth information.

First claim

What is claimed is:

1. A method comprising:
identifying a first motion of a head mounted display (HMD);
adjusting a first raw image received from a depth camera of the HMD based on the first motion to generate a first adjusted image;
generating a first ray based on a center of projection and a focal length of the depth camera;
transforming the first ray based on the first motion to generate a second ray;
generating adjusted depth data based on a plurality of images including the first adjusted image, wherein the adjusted depth data is generated using a point-to-pixel representation of the second ray, the point-to-pixel representation being based on a first index representation of the second ray, a second index representation of the second ray, and a third index representation of the second ray;
determining a first contour of an object in a local environment based on the adjusted depth data;
generating virtual reality (VR) content that includes a virtual object having a second contour, the second contour matching the first contour; and
providing the VR content at the HMD based on the adjusted depth data.

2. The method of claim 1 , wherein identifying the first motion comprises:
identifying the first motion based on non-visual sensor data received from sensors of the HMD.

3. The method of claim 2 , wherein the sensors of the HMD comprise an inertial measurement unit (IMU) of the HMD.

4. The method of claim 1 , wherein adjusting the first raw image comprises:
adjusting the first raw image based on the center of projection and the focal length.

5. The method of claim 4 , wherein adjusting the first raw image further comprises:
adjusting the first raw image based on the second ray.

6. The method of claim 5 , wherein adjusting the first raw image further comprises:
generating a set of adjusted pixel values based on the second ray; and
storing the set of adjusted pixel values as a frame of the first adjusted image.

7. The method of claim 1 , wherein adjusting the first raw image comprises:
adjusting a first pixel of a frame of the first raw image by a first amount; and
adjusting a second pixel of the frame of the first raw image by a second amount different from the first amount.

8. A head mounted display (HMD) comprising:
a motion detection module to identify a first motion of the HMD;
a depth camera to generate a first raw image;
a depth adjustment module to adjust the first raw image based on the first motion to generate a first adjusted image and to generate first adjusted depth data based on the first adjusted image by:
generating a first ray based on a center of projection and a focal length of the depth camera;
transforming the first ray based on the first motion to generate a second ray, wherein the first adjusted depth data is generated using a point-to-pixel representation of a second ray, the point-to-pixel representation being based on a first index representation of the second ray, a second index representation of the second ray, and a third index representation of the second ray; and

a virtual reality (VR) content generator to determine a first contour of an object in a local environment based on the first adjusted depth data, generate VR content that includes a virtual object having a second contour, the second contour matching the first contour, and generate the VR content based on the first adjusted depth data.

9. The HMD of claim 8 , wherein the motion detection module indicates the first motion based on non-visual sensor data.

10. The HMD of claim 9 , wherein the motion detection module an inertial measurement unit (IMU).

11. The HMD of claim 8 , wherein the depth adjustment module adjusts the first raw image by:
adjusting the first raw image based on the center of projection and the focal length.

12. The HMD of claim 11 , wherein the depth adjustment module adjusts the first raw image by:
adjusting the first raw image based on the second ray.

13. The HMD of claim 12 , wherein the depth adjustment module adjusts the first raw image by:
generating a set of adjusted pixel values based on the second ray; and
storing the set of adjusted pixel values as a frame of the adjusted depth data.

14. The HMD of claim 8 , wherein the depth adjustment module adjusts the first raw image by:
adjusting a first pixel of a frame of the first raw image by a first amount; and
adjusting a second pixel of the frame of the first raw image by a second amount different from the first amount.

15. A non-transitory computer readable medium embodying instructions to manipulate a processor to execute a method comprising:
identifying a first motion of a head mounted display (HMD);
adjusting a first raw image received from a depth camera of the HMD based on the first motion to generate a first adjusted image;
generating a first ray based on a center of projection and a focal length of the depth camera;
transforming the first ray based on the first motion to generate a second ray;
generating adjusted depth data based on the first adjusted image, wherein the adjusted depth data is generated using a point-to-pixel representation of a second ray, the point-to-pixel representation being based on a first index representation of the second ray, a second index representation of the second ray, and a third index representation of the second ray;
determining a first contour of an object in a local environment based on the adjusted depth data;
generating virtual reality (VR) content that includes a virtual object having a second contour, the second contour matching the first contour; and
providing the VR content at the HMD based on the adjusted depth data.

16. The non-transitory computer readable medium of claim 15 , wherein identifying the first motion comprises:
identifying the first motion based on non-visual sensor data received from sensors of the HMD.

17. The non-transitory computer readable medium of claim 16 , wherein the sensors of the HMD comprise an inertial measurement unit (IMU) of the HMD.

18. The non-transitory computer readable medium of claim 17 , wherein adjusting the first raw image comprises:
adjusting the first raw image based on the center of projection and the focal length.

19. The non-transitory computer readable medium of claim 18 , wherein adjusting the first raw image further comprises:
adjusting the first raw image based on the second ray.

Assignees

• Google Llc

Inventors

• Schmidt Mirko
• Dryanovski Ivan

Classifications

• H04N23/689

  Motion occurring during a rolling shutter mode · CPC title

• H04N23/683

  performed by a processor, e.g. controlling the readout of an image memory · CPC title

• H04N23/6812

  based on additional sensors, e.g. acceleration sensors · CPC title

• G01S17/86

  Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders · CPC title

• H04N13/344

  with head-mounted left-right displays · CPC title