Use of surface reconstruction data to identify real world floor

Patent metadata

Field	Value
Publication number	US-9645397-B2
Application number	US-201514697227-A
Country	US
Kind code	B2
Filing date	Apr 27, 2015
Priority date	Jul 25, 2014
Publication date	May 9, 2017
Grant date	May 9, 2017

Abstract

In a virtual reality or mixed reality environment, an HMD device is configured to use surface reconstruction data points obtained with a sensor package to identify a location of a floor of a real world environment in which the device operates by sorting the data points by height into respective buckets where each bucket holds a different range of heights. A bucket having the greatest number of data points that are below the height of a user of the HMD device is used to identify the height of the real world floor, for example, by calculating an average of height values of data points in that bucket. A floor for the virtual reality environment may then be aligned to the identified height of the real world floor.

First claim

What is claimed is:

1. A method performed by a head mounted display (HMD) device to identify a height of a real world floor, the HMD device supporting rendering of a virtual or mixed reality environment, the method comprising:
obtaining surface reconstruction data associated with a real world environment adjoining a user of the HMD device;
classifying surface reconstruction data points by height;
sorting the classified data points into respective buckets;
selecting a bucket having a greatest number of data points that are below a height of the user; and
identifying a height of the real world floor relative to the user based on the data points in the selected bucket.

2. The method of claim 1 in which the surface reconstruction data includes depth data and further including generating the surface reconstruction data using a depth sensor and applying surface reconstruction techniques to reconstruct the real world environment geometry.

3. The method of claim 1 further including generating depth data using one or more depth-from-stereo imaging analyses.

4. The method of claim 1 further including aligning a height of a virtual world floor with the height of the real world floor.

5. The method of claim 1 further including aligning a virtual world object to the real world floor.

6. The method of claim 1 in which the surface reconstruction data is captured using a sensor package in the HMD device, the sensor package including a depth sensor or depth camera.

7. The method of claim 1 further including configuring the HMD device to expose a user interface (UI) to the user and prompting the user, through the UI, to stand and look down.

8. The method of claim 1 further including determining the height of the user utilizing surface reconstruction data that is captured when the user looks down.

9. The method of claim 1 further including selecting a bucket that is below the user's height by some predetermined threshold.

10. The method of claim 1 further including calculating an average of height values of data points in the selected bucket to determine the height of the real world floor.

11. A head mounted display (HMD) device operable by a user in a physical environment having a real world floor, comprising:
one or more processors;
a sensor package;
a display for rendering a virtual reality environment to the user; and
one or more memory devices storing computer-readable instructions which, when executed by the one or more processors, perform a method comprising the steps of:
i) generating surface reconstruction data for the physical environment using the sensor package,
ii) using the surface reconstruction data to identify a location of the real world floor relative to the HMD device by
a) sorting surface reconstruction data points into a group among a plurality of groups according to vertical distance from the HMD device, each group in the plurality of groups covering a different range of vertical distances,
b) selecting a group having a greatest number of data points that exceed a threshold vertical distance from a current position of the HMD device,
c) identifying a location of the real world floor relative to the HMD device based on the data points in the selected group, and

iii) operating the HMD device to render the virtual reality environment on the display so that the virtual world floor is aligned with the identified location of the real world floor.

12. The HMD device of claim 11 further including calculating an average of values of data points in the selected group to identify the location of the real world floor relative to the HMD device.

13. The HMD device of claim 11 further including modeling the physical environment using a surface reconstruction data pipeline that implements a volumetric method creating multiple overlapping surfaces that are integrated.

14. The HMD device of claim 11 further including a sealed protective visor that substantially encases the display.

15. The HMD device of claim 11 further including dynamically identifying the location of the real world floor relative to the HMD device as the user traverses the physical environment.

16. The HMD device of claim 15 further including realigning the virtual world floor with the real world floor responsively to the dynamic identifying.

17. One or more computer readable memories not consisting of propagated signals storing computer-executable instructions for rendering a virtual or mixed reality environment having variable elevations of virtual ground on a head mounted display (HMD) device located in a physical environment having a floor, the method comprising the steps of:
obtaining a point cloud of surface reconstruction data;
generating a mesh of the physical environment from the point cloud, the mesh including a plurality of vertices and faces;
selecting a vertex in the mesh having a height that is lowest relative to the HMD device; and
establishing a location of the floor of the physical environment at the height of the selected vertex.

18. The one or more computer readable memories of claim 17 further including performing an adjustment of the virtual ground to align with the location of the floor of the physical environment.

19. The one or more computer readable memories of claim 18 in which the adjustment is performed dynamically as a vertical adjustment.

20. The one or more computer-readable memories of claim 19 further including filtering or smoothing the dynamic vertical adjustment.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Da Veiga Alexandre
• Sugden Ben
• Salter Tom

Classifications

• H04N13/204

  using stereoscopic image cameras (stereoscopic photography G03B35/00) · CPC title

• G06T7/73

  using feature-based methods · CPC title

• H04N13/279

  the virtual viewpoint locations being selected by the viewers or determined by tracking · CPC title

• G02B2027/0187

  slaved to motion of at least a part of the body of the user, e.g. head, eye · CPC title

• G02B2027/0167

  Emergency system, e.g. to prevent injuries · CPC title