System and method for generating a mixed reality environment

Patent metadata

Field	Value
Publication number	US-2017193710-A1
Application number	US-201715465530-A
Country	US
Kind code	A1
Filing date	Mar 21, 2017
Priority date	Oct 27, 2008
Publication date	Jul 6, 2017
Grant date	—

Abstract

A system and method for generating a mixed-reality environment is provided. The system and method provides a user-worn sub-system communicatively connected to a synthetic object computer module. The user-worn sub-system may utilize a plurality of user-worn sensors to capture and process data regarding a user's pose and location. The synthetic object computer module may generate and provide to the user-worn sub-system synthetic objects based information defining a user's real world life scene or environment indicating a user's pose and location. The synthetic objects may then be rendered on a user-worn display, thereby inserting the synthetic objects into a user's field of view. Rendering the synthetic objects on the user-worn display creates the virtual effect for the user that the synthetic objects are present in the real world.

First claim

1 . A method of rendering a synthetic object onto a user-worn display showing a user's view of a real world scene, the method comprising:
capturing real world scene information using one or more user-worn sensors; producing a first pose estimation data set and a first depth data set based on at least a portion of the captured real world scene information, wherein the first pose estimation data set includes predicted inter-frame pose information for at least one video sensor of the one or more user-worn sensors to compensate for a data capture time lag between the at least one video sensor and at least one other user-worn sensor of the one or more user-worn sensors; receiving the synthetic object generated in accordance with the first pose estimation data set and the first depth data set; and rendering the synthetic object onto the user-worn display in accordance with the first pose estimation data set and the first depth data set to integrate the synthetic object into the user's view of the real world scene.

2 . The method of claim 1 , wherein the real world scene information is captured only by the one or more user-worn sensors.

3 . The method of claim 1 , wherein the at least one video sensor of the one or more user-worn sensors comprises a first video sensor and a second video sensor in stereo configuration.

4 . The method of claim 1 , wherein one of the one or more user-worn sensors further comprises a LIDAR sensor.

5 . The method of claim 1 , wherein one of the one or more user-worn sensors further comprises at least one of a global positioning system sensor or an inertial measurement unit.

6 . The method of claim 1 , wherein one of the one or more user-worn sensors further comprises an inertial measurement unit, and wherein the method further includes processing a portion of the first pose estimation data set based on at least a portion of the real world scene information captured by the inertial measurement unit and the at least one video sensor.

7 . The method of claim 6 , further comprising:
receiving a first video frame from the at least one video sensor; receiving an inertial measurement unit data set from the inertial measurement unit; and predicting the portion of the first pose estimation data set based on the inertial measurement unit data set and the first video frame.

8 . The method of claim 1 , further comprising:
calculating a user-controlled device pose as part of the first pose estimation data set.

9 . The method of claim 8 , wherein the user-controlled device includes a marker, wherein the at least one video sensor of the one or more user-worn sensors comprises a first video sensor and a second video sensor in stereo configuration, and wherein calculating the user-controlled device pose comprises:
receiving a first video frame of the marker from the first video sensor; receiving a second video frame of the marker from the second video sensor; and utilizing the first video frame and the second video frame to triangulate a location of the marker.

10 . The method of claim 8 , wherein calculating the user-controlled device pose comprises:
receiving a user-controlled device data set from one of the one or more user-worn sensors located on a user-controlled device; and calculating the user-controlled device pose based on the user-controlled device data set and data received from a communicatively connected landmark database.

11 . The method of claim 8 , wherein calculating the user-controlled device pose comprises:
estimating a relative orientation between the user and the user-controlled device through use of at least one user-worn sensor; and calculating the user-controlled device pose based on the relative orientation and information received from a landmark database.

12 . A system for rendering a synthetic object onto a user-worn display within a user's field of view as the user views real world scene, comprising:
a user-worn computer module configured to:
capture real world scene information using one or more user-worn sensors;
produce a first pose estimation data set and a first depth data set based on at least a portion of the captured real world scene information, wherein the first pose estimation data set includes predicted inter-frame pose information for at least one video sensor of the one or more user-worn sensors to compensate for a data capture time lag between the at least one video sensor and at least one other user-worn sensor of the one or more user-worn sensors;
receive the synthetic object generated in accordance with the first pose estimation data set and the first depth data set; and
render the synthetic object onto the user-worn display, within the user's field of view as the user views the real world scene, in accordance with the first pose estimation data set and the first depth data set to integrate the synthetic object into the user's view of the real world scene.

13 . The system of claim 12 , further comprising:
a synthetic object computer module configured to:
retrieve the synthetic object from a database in accordance with the first pose estimation data set and the first depth data set; and
transmit the synthetic object to the user-worn computer module.

14 . The system of claim 12 , wherein the real world information is captured only by the one or more user-worn sensors.

15 . The system of claim 12 , wherein the at least one video sensor of the one or more user-worn sensors comprises a first video sensor and a second video sensor in stereo configuration.

16 . The system of claim 12 , wherein one of the one or more user-worn sensors further comprises at least one of a LIDAR sensor, an inertial measurement unit, or a global positioning system sensor.

17 . The system of claim 12 , wherein the one or more user-worn sensors comprises an inertial measurement unit, and wherein the user-worn computer module is further configured to produce a portion of the first pose estimation data set based on at least a portion of the real world scene information captured by the inertial measurement unit and the at least one video sensor.

18 . The system of claim 12 , further comprising:
a user-controlled device which includes a marker, wherein the user-worn computer module is further configured to calculate a user-controlled device pose by:
receiving a first video frame of the marker from a first video sensor and, receiving a second video frame of the marker from a second video sensor; and
utilizing the first video frame and the second video frame to triangulate a location of the marker.

19 . The system of claim 12 , wherein the user-worn computer module is further configured to:
receive a user-controlled device data set from one of the one or more user-worn sensors located on a user-controlled device, and calculate a user-controlled device pose based on the user-controlled device data set and data received from a communicatively connected landmark database.

20 . The system of claim 12 , wherein the user-worn module is further configured to:
prior to rendering the synthetic object, predict the first pose estimation data set with latency less than a predetermined threshold using sensors with data capture interval rates greater than the predetermined threshold.

Assignees

• Stanford Res Inst Int

Inventors

• Kumar Rakesh
• Oskiper Targay
• Naroditsky Oleg
• Samarasekera Supun
• Zhu Zhiwei
• Kim Janet

Classifications

• G06F3/012 Primary

  Head tracking input arrangements · CPC title

• G06F3/0346

  with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title