Virtualizing objects using object models and object position data

Patent metadata

Field	Value
Publication number	US-11062139-B2
Application number	US-201816222333-A
Country	US
Kind code	B2
Filing date	Dec 17, 2018
Priority date	Dec 13, 2017
Publication date	Jul 13, 2021
Grant date	Jul 13, 2021

Abstract

Described herein are a system and methods for generating a record of objects, as well as respective positions for those objects, with respect to a user. In some embodiments, a user may use a user device to scan an area that includes one or more objects. The one or more objects may be identified from image information obtained from the user device. Positional information for each of the one or more objects may be determined from depth information obtained from a depth sensor installed upon the user device. In some embodiments, the one or more objects may be mapped to object models stored in an object model database. The image information displayed on the user device may be augmented so that it depicts the object models associated with the one or more objects instead of the actual objects.

First claim

What is claimed is:

1. A method of facilitating installation or servicing of an object, comprising:
receiving, from a first user device, a request for a virtualized object corresponding to a real object;
receiving, from the first user device, position information for the virtualized object, the position information corresponding to a position within a physical space;
receiving a request to install or service the real object, wherein the request to install or service the real object is a request for a physical installation or servicing of the real object in the physical space at the position within the physical space;
identifying an installation or servicing entity to install or service the real object based on the request, wherein the installation or servicing entity is identified as an entity that provides the physical installation or servicing; and
transmitting the virtualized object to the first user device or transmitting the virtualized object and the position information to a second user device associated with the installation or servicing entity; and
displaying, on at least one of a display of the first user device or a display of the second user device, the virtualized object and a representation of at least a portion of the physical space that includes the position, the virtualized object being displayed in a position with respect to the representation of the at least a portion of the physical space such that the virtualized object appears to be located in the position.

2. The method of claim 1 , wherein displaying of the virtualized object and the at least a portion of the physical space is performed using augmented reality and a camera of the first or second user device to capture at least one image of the at least a portion of the physical space and to show the virtualized object superimposed on the at least one image of the at least a portion of the physical space.

3. The method of claim 1 , wherein the virtualized object is a 3D model of the object.

4. The method of claim 1 , wherein the first user device includes a camera and at least one depth sensor adapted to detect distances from the depth sensor to objects within at least one portion of the physical space, and wherein the first user device is adapted to generate, based on the detected distances and image information captured by the camera, the representation of the at least a portion of the physical space as a 3D model of the at least a portion of the physical space.

5. The method of claim 1 , further comprising transmitting, to the second user device associated with the installation or servicing entity, routing information to guide the installation or servicing entity to a location in which the request to install or service the real object is to be completed.

6. The method of claim 1 , wherein the position information includes both location information and orientation information.

7. The method of claim 1 , further comprising:
providing, to the first user device, location information for the real object and information that facilitates movement of the real object from a location indicated in the location information to the position.

8. A method of facilitating servicing for an object, comprising:
transmitting, from a first user device, a request for a virtualized object corresponding to a real object;
transmitting, from the first user device, position information for the virtualized object, the position information corresponding to a position within a physical space;
transmitting a request to install or service the real object, wherein the request to install or service the real object is a request for a physical installation or servicing of the real object in the physical space at the position within the physical space;
receiving a response from an installation or servicing entity offering to install or service the real object based on the request, wherein the installation or servicing entity is an entity that provides the physical installation or servicing; and
receiving the virtualized object by the first user device or receiving the virtualized object and the position information by a second user device associated with the installation or servicing entity; and
displaying, on at least one of a display of the first user device or a display of the second user device, the virtualized object and a representation of at least a portion of the physical space that includes the position, the virtualized object being displayed in a position with respect to the representation of the at least a portion of the physical space such that the virtualized object appears to be located in the position.

9. The method of claim 8 , wherein displaying of the virtualized object and the at least a portion of the physical space is performed using augmented reality and a camera of the first or second user device to capture at least one image of the at least a portion of the physical space and to show the virtualized object superimposed on the at least one image of the at least a portion of the physical space.

10. The method of claim 8 , wherein the virtualized object is a 3D model of the object.

11. The method of claim 8 , wherein the first user device includes a camera and at least one depth sensor adapted to detect distances from the depth sensor to objects within at least one portion of the physical space, and wherein the first user device is adapted to generate, based on the detected distances and image information captured by the camera, the representation of the at least a portion of the physical space as a 3D model of the at least a portion of the physical space.

12. The method of claim 8 , wherein the real object is associated with a user and wherein the first user device is associated with the user.

13. The method of claim 8 , wherein the position information includes both location information and orientation information.

14. A system comprising:
a processor; and
a memory including instructions that, when executed with the processor, cause the system to, at least:
receive a servicing request for a real object, wherein the servicing request is a request for physical servicing of the real object in a physical space at a position within the physical space;
retrieve a virtualized object corresponding to the real object and position information associated with the servicing request for the real object, the position information corresponding to the position within the physical space;
determine, based on the servicing request, a user device associated with completion of the servicing request wherein the user device is associated with an entity that provides the physical servicing; and
provide, to the determined user device, the virtualized object and the position information such that the virtualized object is caused to be displayed within a representation of at least a portion of the physical space that includes the position, the virtualized object being displayed in a position with respect to the representation of the at least a portion of the physical space such that the virtualized object appears to be located in the position.

15. The system of claim 14 , wherein the servicing request for the real object is a request to place the real object in the position.

16. The system of claim 15 , wherein the instructions, when executed with the processor, further cause the system to provide, to the user device, location information for the real object and information that facilitates movement of the real object from a location indicated in the location information to the position.

17. The system of claim 14 , wherein the in

Assignees

• Lowes Companies Inc
• Lowes Conpanies Inc

Inventors

• Sheffield Mason E
• Shabtai Josh

Classifications

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06V20/20 Primary

  in augmented reality scenes · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06F16/9537

  Spatial or temporal dependent retrieval, e.g. spatiotemporal queries · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title