System and method for generating a mixed reality environment

Patent metadata

Field	Value
Publication number	US-9600067-B2
Application number	US-60658109-A
Country	US
Kind code	B2
Filing date	Oct 27, 2009
Priority date	Oct 27, 2008
Publication date	Mar 21, 2017
Grant date	Mar 21, 2017

Abstract

A system and method for generating a mixed-reality environment is provided. The system and method provides a user-worn sub-system communicatively connected to a synthetic object computer module. The user-worn sub-system may utilize a plurality of user-worn sensors to capture and process data regarding a user's pose and location. The synthetic object computer module may generate and provide to the user-worn sub-system synthetic objects based information defining a user's real world life scene or environment indicating a user's pose and location. The synthetic objects may then be rendered on a user-worn display, thereby inserting the synthetic objects into a user's field of view. Rendering the synthetic objects on the user-worn display creates the virtual effect for the user that the synthetic objects are present in the real world.

First claim

We claim:

1. A method of rendering a synthetic object onto a user-worn display showing a user's view of a real world scene, the method comprising the steps of:
capturing real world scene information using one or more user-worn sensors;
producing a first pose estimation data set and a first depth data set based on at least a portion of the captured real world scene information, wherein the first pose estimation data set includes predicted inter-frame pose information for at least one camera of the one or more user-worn sensors to compensate for a data capture time lad between the at least one camera and at least one other user-worn sensor of the one or more user-worn sensors;
receiving the synthetic object generated in accordance with the first pose estimation data set and the first depth data set;
rendering the synthetic object onto the user-worn display in accordance with the first pose estimation data set and the first depth data set, thereby integrating the synthetic object into the user's view of the real world scene;
producing a second pose estimation data set and second depth data set for a user-controlled device; and
determining, based at least in part on the first pose estimation data set and the second pose estimation data set, whether a user-controlled device has interacted with the synthetic object and causing the synthetic object to react to the interaction based at least in part on the first pose estimation data and the second pose estimation data set.

2. The method of claim 1 , wherein the real world scene information is captured only by the one or more user-worn sensors.

3. The method of claim 1 , wherein the one or more user-worn sensors comprises a first video camera and a second video camera in stereo configuration.

4. The method of claim 3 , wherein the user-controlled device includes a marker and producing the second pose estimation data set for the user-controlled device comprises:
receiving a first video frame of the marker from a first video camera;
receiving a second video frame of the marker from a second video camera; and
utilizing the first video frame and the second video frame to triangulate a location of the marker, thereby producing the second pose estimation data set for the user-controlled device.

5. The method of claim 1 , wherein one of the one or more user-worn sensors comprises a LIDAR sensor.

6. The method of claim 1 , wherein one of the one or more user-worn sensors comprises an inertial measurement unit.

7. The method of claim 1 , wherein one of the one or more user-worn sensors comprises a global positioning system sensor and an inertial measurement unit.

8. The method of claim 1 , wherein the one or more sensors comprises an inertial measurement unit and a video sensor.

9. The method of claim 8 , further comprising:
processing a portion of the first pose estimation data set based on at least a portion of the real world scene information captured by the inertial measurement unit and the video sensor.

10. The method of claim 9 , further comprising:
receiving a first video frame from the video sensor;
receiving an inertial measurement unit data set from the inertial measurement unit; and
predicting the portion of the first pose estimation data set based on the inertial measurement unit data set and the first video frame.

11. The method of claim 1 , wherein producing the second pose estimation data set for the user-controlled device comprises:
receiving a user-controlled device data set from one of the one or more user-worn sensors located on a user-controlled device; and
producing the second pose estimation data set for the user-controlled device based on the user-controlled device data set and data received from a communicatively connected landmark database.

12. The method of claim 1 , wherein producing the second pose estimation data set for the user-controlled device comprises:
estimating a relative orientation between the user and the user-controlled device through the use of at least one user-worn sensor; and
producing the second pose estimation data set for the user-controlled device based on the relative orientation and information received from a landmark database.

13. The method of claim 1 , wherein the step of producing the pose estimation data set utilizes a landmark database and the real world scene information.

14. The method of claim 1 , wherein the captured real world scene information is used to update a landmark database.

15. The method of claim 1 , wherein the inter-frame pose information for the at least one camera is predicted by assuming that a movement of the camera occurs at a constant translational velocity and a constant rotational velocity.

16. The method of claim 1 , wherein the inter-frame pose information for the at least one camera is predicted by integrating data points to determine an incremental rotational motion between frames captured by the at least one camera.

17. A method of rendering a synthetic object onto a first user-worn display showing a first user's view of a real world scene and the synthetic object onto a second user-worn display showing the second user's view of the real world scene, the method comprising the steps of:
capturing a first set of real world scene information using one or more sensors worn by the first user;
capturing a second set of real world scene information using one or more sensors worn by the second user;
producing a first pose estimation data set and a first depth data set based on at least a portion of the first set of real world scene information, wherein the first pose estimation data set includes predicted inter-frame pose information for at least one camera of the one or more user-worn sensors to compensate for a data capture time lag between the at least one camera and at least one other user-worn sensor of the one or more user-worn sensors;
producing a second pose estimation data set and a second depth data set based on at least a portion of the second set of real world scene information;
receiving the synthetic object generated in accordance with the first pose estimation data set and the first depth data set;
receiving the synthetic object generated in accordance with the second pose estimation data set and the second depth data set;
rendering the synthetic object onto the first user-worn display in accordance with the first pose estimation data set and the first depth data set, thereby integrating the synthetic object into the first user's perception of the real world scene;
rendering the synthetic object onto the second user-worn display in accordance with the second pose estimation data set and the second depth data set, thereby integrating the synthetic object into the second user's perception of the real world scene, wherein the synthetic object appears consistent within the first user's perception of the real world scene and the second user's perception of the real world scene.

18. A system for rendering a synthetic object onto a user-worn display within the user's field of view as the user views real world scene, comprising:
a user-worn computer module configured to:
capture real world scene information using one or more user-worn sensors;
produce a first pose estimation data set and first depth data set based on at least a portion of the real world scene information, wherein the first pose estimation data set includes predicted inter-frame pose information for at least one camera of the one or more user-worn sensors to compensate for a data capture time lag between the at least one camera and at least o

Assignees

• Kumar Rakesh
• Oskiper Targay
• Naroditsky Oleg
• Samarasekera Supun
• Zhu Zhiwei
• Kim Janet
• Stanford Res Inst Int

Inventors

• Kumar Rakesh
• Oskiper Targay
• Naroditsky Oleg
• Samarasekera Supun
• Zhu Zhiwei
• Kim Janet

Classifications

• G06F3/012 Primary

  Head tracking input arrangements · CPC title

• G06F3/0346

  with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title