Systems and methods for rendering a portion of a 3D display

Patent metadata

Field	Value
Publication number	US-11727657-B2
Application number	US-202117227006-A
Country	US
Kind code	B2
Filing date	Apr 9, 2021
Priority date	Apr 9, 2021
Publication date	Aug 15, 2023
Grant date	Aug 15, 2023

Abstract

Systems and methods including one or more processors and one or more non-transitory storage devices storing computing instructions configured to run on the one or more processors and perform: generating a mask of an object using one or more images; generating a 3D model of the object using the mask of the object; simulating an artificial 3D capture environment; generating an artificial surface for the object in the artificial 3D capture environment; transferring the artificial surface for the object to the one or more images; and blending the artificial surface for the object with a real-world surface in the one or more images. Other embodiments are disclosed herein.

First claim

What is claimed is:

1. A system comprising:
one or more processors; and
one or more non-transitory computer-readable storage devices storing computing instructions configured to run on the one or more processors causing the one or more processors to perform operations comprising:
generating a mask of an object using one or more images;
generating a 3D model of the object using the mask of the object;
simulating an artificial 3D capture environment for the 3D model of the object;
generating an artificial surface for the 3D model of the object in the artificial 3D capture environment;
transferring the artificial surface for the 3D model of the object to the one or more images; and
blending the artificial surface for the 3D model of the object with a real-world surface in the one or more images to create a blended surface that (1) comprises both the artificial surface and the real-world surface and (2) has a higher cleanliness level than the real world surface when displayed on an electronic device.

2. The system of claim 1 , wherein:
the simulating the artificial 3D capture environment comprises:
creating a 3D image map of a real-world capture environment where the one or more images were taken; and
performing path tracing in the artificial 3D capture environment using the 3D image map to generate artificial light for the 3D environment; and

the generating the artificial surface for the object comprises:
generating the artificial surface in the 3D capture environment using the artificial light for the 3D environment.

3. The system of claim 1 , wherein the generating the mask of the object using the one or more images comprises:
training a machine learning algorithm on one or more training images;
identifying, using the machine learning algorithm, as trained, (1) the object in the one or more images and (2) objects other than the object in the one or more images;
removing the objects from the one or more images; and
after removing the objects from the one or more images, generating the mask from only the object in the one or more images.

4. The system of claim 3 ,
wherein the machine learning algorithm, as trained, comprises a neural network configured identify an automobile;
wherein the one or more training images comprise one or more voxel models; and
wherein at least a portion of the one or more training images are labeled with:
a make of the automobile;
a model of the automobile; and
a body type of the automobile.

5. The system of claim 1 ,
wherein the mask comprises at least a portion of the one or more images, and
wherein the one or more images comprise images taken radially around the object; and
the generating the 3D model of the object using the mask of the object comprises performing volume carving using at least the portion of the one or more images to create a voxelized model of the object.

6. The system of claim 1 , wherein the generating the artificial surface for the object in the artificial 3D capture environment comprises:
simulating reflections and shadows for the object in the artificial 3D capture environment by reflecting artificial light for the artificial 3D capture environment off of the 3D model of the object;
rendering the artificial surface for the object using path trace rendering at a lower resolution than a display resolution; and
supersampling the reflections and shadows at the display resolution; and
wherein the artificial light is generated using a 3D image map of a real-world capture environment where the one or more images were taken.

7. The system of claim 1 , further comprising:
facilitating displaying a 3D display of the object using the one or more images, the 3D display comprising the artificial surface for the object and the real-world surface for the object.

8. The system of claim 7 , further comprising:
receiving a zoom selection point selected by a user on the 3D display of the object;
facilitating zooming the 3D display of the object into a zoomed 3D display of the object; and
facilitating rotating the zoomed 3D display of the object around the zoom selection point.

9. The system of claim 1 , wherein the artificial 3D capture environment comprises a 3D rendering of a real-world capture environment where the one or more images were taken.

10. The system of claim 1 , wherein the generating the mask of the object comprises:
receiving an image of the object;
determining a respective pixel intensity for each respective pixel in the image; and
when the respective pixel intensity of a respective pixel of each respective pixel is above a predetermined threshold, labeling the respective pixel as not the object.

11. A method comprising:
generating a mask of an object using one or more images;
generating a 3D model of the object using the mask of the object;
simulating an artificial 3D capture environment for the 3D model of the object;
generating an artificial surface for the 3D model of the object in the artificial 3D capture environment;
transferring the artificial surface for the 3D model of the object to the one or more images; and
blending the artificial surface for the 3D model of the object with a real-world surface in the one or more images to create a blended surface that (1) comprises both the artificial surface and the real-world surface and (2) has a higher cleanliness level than the real world surface when displayed on an electronic device.

12. The method of claim 11 , wherein:
the simulating the artificial 3D capture environment comprises:
creating a 3D image map of a real-world capture environment where the one or more images were taken; and
performing path tracing in the artificial 3D capture environment using the 3D image map to generate artificial light for the 3D environment; and

the generating the artificial surface for the object comprises:
generating the artificial surface in the 3D capture environment using the artificial light for the 3D environment.

13. The method of claim 11 , wherein the generating the mask of the object using the one or more images comprises:
training a machine learning algorithm on one or more training images;
identifying, using the machine learning algorithm, as trained, (1) the object in the one or more images and (2) objects other than the object in the one or more images;
removing the objects from the one or more images; and
after removing the objects from the one or more images, generating the mask from only the object in the one or more images.

14. The method of claim 13 ,
wherein the machine learning algorithm, as trained, comprises a neural network configured identify an automobile;
wherein the one or more training images comprise one or more voxel models; and
wherein at least a portion of the one or more training images are labeled with:
a make of the automobile;
a model of the automobile; and
a body type of the automobile.

15. The method of claim 11 ,
wherein the mask comprises at least a portion of the one or more images, and
wherein the one or more images comprise images taken radially around the object; and
the generating the 3D model of the object using the mask of the object comprises performing volume carving using at least the portion of the one or more images to create a voxelized model of the object.

16. The method of claim 11 , wherein the generating the artificial surface for the object in the artificial 3D capture environment comprises:
simulating reflections and shadows for the object in the artificial 3D capture environment by reflecting artificial light for the artificial 3D capture e

Assignees

• Carvana Llc

Inventors

• Melling Alan Richard
• Velez Salas Pedro Damian
• Schindler Grant Evan
• Francois Bruno Jean
• Cilia Remy Tristan

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06T19/20 Primary

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title

• G06F3/04845

  for image manipulation, e.g. dragging, rotation, expansion or change of colour · CPC title

• G06N20/00

  Machine learning · CPC title