Virtualizing objects using object models and object position data

Patent metadata

Field	Value
Publication number	US-10192115-B1
Application number	US-201715840567-A
Country	US
Kind code	B1
Filing date	Dec 13, 2017
Priority date	Dec 13, 2017
Publication date	Jan 29, 2019
Grant date	Jan 29, 2019

Abstract

Described herein are a system and methods for generating a record of objects, as well as respective positions for those objects, with respect to a user. In some embodiments, a user may use a user device to scan an area that includes one or more objects. The one or more objects may be identified from image information obtained from the user device. Positional information for each of the one or more objects may be determined from depth information obtained from a depth sensor installed upon the user device. In some embodiments, the one or more objects may be mapped to object models stored in an object model database. The image information displayed on the user device may be augmented so that it depicts the object models associated with the one or more objects instead of the actual objects.

First claim

What is claimed is:

1. A method of generating a catalog of positioned objects, comprising:
receiving sensor output obtained using one or more input sensors of a user device;
identifying, within the sensor output, one or more objects;
determining, based on the sensor output and based on a location of the user device, a position of the one or more objects;
mapping the one or more objects to one or more corresponding object models within an object model database;
identifying that the one or more objects are to be associated with the determined position;
storing an indication of the one or more object models and the determined position of the one or more objects;
generating an extrapolated image of an area where the one or more objects are located that depicts the area without the one or more objects; and
causing the extrapolated image to be displayed on a display screen such that the one or more objects appear to be absent from the sensor output.

2. The method of claim 1 , wherein the sensor output includes at least image information and depth information associated with the one or more objects.

3. The method of claim 1 , further comprising:
receiving, from a second user device, a second location associated with the second user device;
determining, based on the second location and the determined position of the one or more objects, that the second user device is within a vicinity of the one or more objects; and
providing, to the second user device, the one or more object models.

4. The method of claim 3 , further comprising causing the second user device to augment an image displayed on a display screen of the second user device such that the one or more object model appears to be positioned in the determined position of the one or more objects.

5. The method of claim 1 , further comprising:
determining, based on the identification of the one or more objects, whether any one or more other objects not present in the image might be of interest to a person using the method; and
recommending at least one of the one or more other objects to the person.

6. The method of claim 1 , further comprising:
identifying that the one or more objects are to be associated with a user based on the user being associated with the user device; and
storing an indication of the user with respect to the one or more object models and the determined position of the one or more objects.

7. The method of claim 1 , further comprising:
detecting an event which impacts a geographic area;
determining that the one or more objects are located within the geographic area based on the determined position of the one or more objects; and
assessing an impact of the event based on one or more attributes of the one or more objects.

8. A system comprising:
one or more camera devices;
a processor; and
a memory including instructions that, when executed with the processor, cause the system to, at least:
receive sensor data from the one or more camera devices;
identify, within the sensor data, at least one object, the object being distinct from an area in which the object is located;
determine, based on the sensor data and a location of the one or more camera devices, a position of the at least one object;
match the at least one object with at least one object model stored in an object model database;
store an indication of the at least one object model and the determined position of the at least one object;
generate an extrapolated image of an area where the at least one object is located that depicts the area without the at least one object; and
cause the extrapolated image to be displayed on a display screen such that the at least one object appears to be absent from the sensor output.

9. The system of claim 8 , wherein the determined position includes a location and orientation of the at least one object.

10. The system of claim 8 , wherein the location of the one or more camera devices comprises GPS coordinates of a user device on which the one or more camera devices was installed.

11. The system of claim 8 , wherein one of the one or more camera devices comprises a depth sensor.

12. The system of claim 8 , wherein the instructions, when executed with the processor, cause the system to match the at least one object to the at least one object model using one or more machine learning techniques.

13. The system of claim 8 , wherein the instructions, when executed with the processor, cause the system to identify the at least one object using one or more object recognition techniques.

14. The system of claim 8 , wherein the memory further includes instructions that, when executed with the processor, cause the system to, at least:
determine, based on the identity of the one or more objects, whether any one or more other objects might be of interest to a person using the method; and
recommend at least one of the one or more other objects to the person.

15. An apparatus comprising:
a camera device configured to capture image information;
a depth sensor device configured to capture depth information;
a display screen; and
a mobile application stored in a computer-readable medium that, when executed, causes the apparatus to, at least:
identify, within image information captured by the camera device, one or more objects;
determine a position of the one or more objects from depth information captured by the depth sensor;
provide, to a server in communication with the apparatus, an indication of the one or more objects and the position of the one or more objects to cause the server to generate an extrapolated image of an area where the one or more objects are located that depicts the area without the one or more objects; and
display, on the display screen, the extrapolated image such that the one or more objects appear to be absent from the sensor output.

16. The apparatus of claim 15 , wherein the mobile application, when executed, further causes the apparatus to receive, from the server, one or more object models associated with the one or more objects.

17. The apparatus of claim 16 , wherein the mobile application, when executed, further causes the one or more object models to be displayed upon the display screen in the area where the one or more objects are located.

18. The apparatus of claim 17 , wherein the mobile application, when executed, causes the at least one object to be removed from the image information displayed on the display and replaced by the one or more object models.

19. The apparatus of claim 16 , wherein the mobile application, when executed, causes the one or more object models to be received upon determining that the apparatus is within a vicinity of the determined position.

20. The apparatus of claim 15 , wherein the mobile application, when executed, causes the apparatus to identify a category associated with the one or more objects.

21. The apparatus of claim 15 , wherein the mobile application, when executed, causes the apparatus to identify a type or model of the one or more objects.

22. The apparatus of claim 15 , wherein the mobile application, when executed, further causes the apparatus to, at least:
determine, based on the identity of the one or more objects, whether any one or more other objects might be of interest to a person using the apparatus; and
recommend at least one of the one or more other objects to the person.

23. A method of generating a catalog of positioned objects, comprising:
receiving sensor output

Assignees

• Lowes Companies Inc

Inventors

• Sheffield Mason E
• Shabtai Josh

Classifications

• G06F16/5854

  using shape and object relationship · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06N20/00

  Machine learning · CPC title

• G06T7/75

  involving models · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title