Rendering virtual objects in 3D environments

Patent metadata

Field	Value
Publication number	US-10140773-B2
Application number	US-201715422407-A
Country	US
Kind code	B2
Filing date	Feb 1, 2017
Priority date	Feb 1, 2017
Publication date	Nov 27, 2018
Grant date	Nov 27, 2018

Abstract

Systems, methods, devices, and other techniques for placing and rendering virtual objects in three-dimensional environments. The techniques include providing, by a device, a view of an environment of a first user. A first computing system associated with the first user receives an instruction to display, within the view of the environment of the first user, a virtual marker at a specified position of the environment of the first user, the specified position derived from a second user's interaction with a three-dimensional (3D) model of at least a portion of the environment of the first user. The device displays, within the view of the environment of the first user, the virtual marker at the specified position of the environment of the first user.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
providing, by a device, a view of an environment of a first user;
receiving, by the device, an instruction to display a virtual marker within the view of the environment of the first user,
wherein the instruction comprises a coordinate value that specifies a first position of the virtual marker in a three-dimensional (3D) space of an environment of a second user that is different from the environment of the first user, and the first position was determined based on the second user's interaction with a 3D model of at least a portion of the environment of the first user that includes a 3D model of a first physical object located in the environment of the first user;

mapping the coordinate value from the 3D space of the environment of the second user to a 3D space of the environment of the first user to determine a second position of the virtual marker in the 3D space of the environment of the first user that corresponds to the first position of the virtual marker in the 3D space of the environment of the second user;
displaying, by the device and within the view of the environment of the first user, the virtual marker at the second position of the environment of the first user, wherein the virtual marker prompts the first user to manipulate the first physical object that is located in the environment of the first user, wherein the first physical object at least partially appears within the view of the environment of the first user;
detecting, by the device, an action taken by the first user to manipulate the first physical object according to the prompt indicated by the virtual marker, including determining coordinates of a second physical object that the first user brought into proximity of or contact with the first physical object in taking the action to manipulate the first physical object
transmitting, to a computing system of the second user, information that describes the coordinates of the second physical object that the first user brought into proximity of or contact with the first physical object in taking the action to manipulate the first physical object
receiving, by the device and from the computing system of the second user, an indication of whether the action taken by the first user to manipulate the first physical object satisfies a criterion; and
displaying, by the device and within the view of the environment of the first user, the indication of whether the action taken by the first user to manipulate the first physical object satisfies the criterion.

2. The computer-implemented method of claim 1 , wherein:
the coordinate value specifies the first position of the virtual marker in the 3D space of the environment of the second user using a first 3D offset from one or more anchor points of the environment of the second user; and
mapping the coordinate value from the 3D space of the environment of the second user to the 3D space of the environment of the first user comprises:
identifying one or more anchor points of the environment of the first user; and
applying, using the coordinate value, a second 3D offset from the one or more anchor points of the environment of the first user to identify the corresponding position of the virtual marker in the environment of the first user.

3. The computer-implemented method of claim 1 , wherein displaying the virtual marker within the view of the environment of the first user comprises superimposing the virtual marker over a portion of the view of the environment of the first user so that the virtual marker visually appears as if it is located a distance from the first user at the second position of the environment of the first user.

4. The computer-implemented method of claim 1 , wherein the second user's interaction with the 3D model of the at least the portion of the environment of the first user occurs while the second user is remotely located from the environment of the first user.

5. The computer-implemented method of claim 1 , wherein:
the second user's interaction with the 3D model of the at least the portion of the environment comprises placing the virtual marker at a particular location on a surface of the 3D model of the first physical object; and
displaying the virtual marker within the view of the environment of the first user comprises displaying the virtual marker at a corresponding location on a surface of the first physical object within the view of the environment of the first user.

6. The computer-implemented method of claim 1 , further comprising:
obtaining, from one or more spatial sensors, spatial sensing data that indicates spatial characteristics of the environment of the first user; and
generating, based on the spatial sensing data, the 3D model of the at least the portion of the environment of the first user.

7. The computer-implemented method of claim 1 , wherein the device is a head-mounted display (HMD) device, wherein the HIVID device displays the virtual marker within the view of the environment of the first user while the first user is wearing the HMD device.

8. The computer-implemented method of claim 1 , wherein providing the view of the environment of the first user comprises generating an augmented view of the environment of the first user that shows one or more virtual objects at fixed positions in the 3D space of the environment of the first user.

9. The computer-implemented method of claim 1 , wherein the second physical object is a physical implement other than a body of the first user.

10. The computer-implemented method of claim 1 , further comprising:
determining an orientation of the second physical object at a particular time when the first user brought the second physical object into proximity of or contact with the first physical object in taking the action to manipulate the first physical object; and
transmitting, to the computing system of the second user along with the information that describes the coordinates of the second physical object, information that describes the orientation of the second physical object at the particular time.

11. A computer-implemented method, comprising:
receiving, by a computing system, data that represents a 3D model of at least a portion of an environment of a first user, wherein the 3D model includes a virtual object that is a 3D representation of a physical object in the environment of the first user;
rendering, by the computing system for display to a second user, the 3D model of the at least the portion of the environment of the first user;
identifying, by the computing system, that a user input placed a first virtual marker at a first position of a virtual environment that includes the 3D model of the at least the portion of the environment of the first user, including identifying a location on a surface of the virtual object where the user input placed the first virtual marker;
transmitting, by the computing system, an instruction for the first virtual marker to be displayed within a view of the environment of the first user at a first position of the environment of the first user that corresponds to the first position of the virtual environment where the user input placed the first virtual marker;
after transmitting the instruction for the first virtual marker to be displayed within the view of the environment of the first user:
receiving, by the computing system, data that identifies a second position in the environment of the first user where a physical implement was identified as having been brought into contact with or proximity of the physical object in the environment of the first user; and
rendering, by the computing system and for display to the second user withi

Assignees

• Accenture Global Solutions Ltd

Inventors

• Short Matthew Thomas
• Webb Sunny
• Opel Joshua
• Christensen Theo E

Classifications

• G06T7/73

  using feature-based methods · CPC title

• G02B2027/0141

  characterised by the informative content of the display · CPC title

• G06T2219/024

  Multi-user, collaborative environment · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G02B2027/014

  comprising information/image processing systems · CPC title