Systems and methods for labeling 3D models using virtual reality and augmented reality

Patent metadata

Field	Value
Publication number	US-11983817-B2
Application number	US-202117543105-A
Country	US
Kind code	B2
Filing date	Dec 6, 2021
Priority date	Jun 14, 2019
Publication date	May 14, 2024
Grant date	May 14, 2024

Abstract

A computer-implemented method for labeling a three-dimensional (3D) model using virtual reality (VR) techniques implemented by a computer system including a processor is provided herein. The method may include (i) receiving a 3D model including an environmental feature that is unlabeled, (ii) displaying, through a VR device in communication with the processor, a VR environment to a user representing the 3D model, (iii) prompting a user to input labeling data for the environmental feature displayed within the VR environment of the VR device by prompting the user to select the environmental feature through user interaction with the VR device, and input labeling data for the environmental feature, wherein the labeling data identifies the environmental feature, and (iv) generating a labeled 3D model by embedding the labeling data associated with the selected environmental feature into the 3D model.

First claim

We claim:

1. A computer-implemented method for labeling a three-dimensional (3D) model using virtual reality (VR) techniques, the method implemented by a computer system including at least one processor, the method comprising:
receiving, by the processor, a 3D model including at least one environmental feature that is unlabeled;
displaying, through a VR device in communication with the processor, a VR environment to a user representing the 3D model, the VR environment modifiable by the user;
displaying, in the VR environment, a prompt requesting the user to input labeling data for the at least one environmental feature displayed within the VR environment and one or more rules associated with the at least one environmental feature, the labeling data inputted by selecting the at least one environmental feature through gesture icons interacting with the VR environment displayed on the VR device, the labeling data identifying the at least one environmental feature, the one or more rules indicating one or more predicted outcomes associated with the at least one environmental feature; and
generating, by the processor, a labeled 3D model by embedding the labeling data associated with the selected at least one environmental feature into the 3D model along with the one or more rules inputted for the selected at least one environmental feature.

2. The computer-implemented method of claim 1 , the method further comprising:
extracting the at least one environmental feature and associated labeling data from the labeled 3D model; and
generating, by the processor, training data based upon the at least one extracted environmental feature of the labeled 3D model.

3. The computer-implemented method of claim 2 , the method further comprising:
training a machine learning model using the training data; and
training a machine learning model using the labeled 3D model.

4. The computer-implemented method of claim 1 , wherein the 3D model is a point cloud, and wherein the point cloud is generated based upon aerial photos of a real-world location.

5. The computer-implemented method of claim 1 , wherein the 3D model is processed using object recognition and includes processing the 3D model using a segmentation technique, and wherein the segmentation technique is a semantic segmentation technique.

6. The computer-implemented method of claim 1 , wherein generating the labeled 3D model further comprises updating meta-data of data points representing the at least one environmental feature.

7. The computer-implemented method of claim 1 , wherein generating the labeled 3D model further comprises:
generating data points representing the at least one environmental feature; and
embedding the data points representing the at least one environmental feature into the 3D model, wherein the data points representing the at least one environmental feature are a surface mesh.

8. The computer-implemented method of claim 7 , wherein displaying the VR environment further comprises altering an appearance of the surface mesh, and wherein generating the labeled 3D model by embedding the labeling data into the 3D model further comprises updating meta-data associated with the surface mesh.

9. The computer-implemented method of claim 1 , wherein displaying the VR environment further comprises altering an appearance of the at least one environmental feature.

10. The computer-implemented method of claim 9 , wherein altering the appearance of the at least one environmental feature includes shading the at least one environmental feature.

11. The computer-implemented method of claim 9 , wherein altering the appearance of the at least one environmental feature includes outlining the at least one environmental feature.

12. The computer-implemented method of claim 1 , wherein the user inputs data into the VR environment using hand gestures made within the VR environment.

13. The computer-implemented method of claim 1 , wherein the user inputs data into the VR environment using eye movement detected by the processor.

14. The computer-implemented method of claim 1 , wherein the user inputs data into the VR environment using spoken commands.

15. The computer-implemented method of claim 1 , wherein generating the labeled 3D model by embedding the labeling data into the 3D model further comprises updating meta-data associated with the 3D model.

16. The computer-implemented method of claim 1 , wherein generating the labeled 3D model by embedding the labeling data into the 3D model further comprises generating a surface mesh, embedding the surface mesh in the 3D model, and updating meta-data associated with the surface mesh.

17. The computer-implemented method of claim 1 , further comprising generating training data, wherein generating the training data further comprises:
receiving a second labeled 3D model;
aggregating the labeled 3D model and the second labeled 3D model as the training data;
capturing a first and second element of the labeled 3D model; and
aggregating the first and second elements of the labeled 3D model as the training data.

18. The computer-implemented method of claim 17 , wherein generating the training data includes translating the labeled 3D model from a first file format to a second file format.

19. A virtual reality (VR) labeling computer system for labeling a three dimensional (3D) model using VR techniques, the VR labeling computer system including at least one processor in communication with at least one memory device, wherein the at least one processor is configured to:
receive a 3D model including at least one environmental feature that is unlabeled;
display, through a VR device in communication with the processor, a VR environment to a user representing the 3D model, the VR environment modifiable by the user;
display, in the VR environment, a prompt requesting the user to input labeling data for the at least one environmental feature displayed within the VR environment and one or more rules associated with the at least one environmental feature, the labeling data inputted by selecting the at least one environmental feature through gesture icons interacting with the VR environment displayed on the VR device, the labeling data identifying the at least one environmental feature, the one or more rules indicating one or more predicted outcomes associated with the at least one environmental feature; and
generate a labeled 3D model by embedding the labeling data associated with the selected at least one environmental feature into the 3D model along with the one or more rules inputted for the selected at least one environmental feature.

20. At least one non-transitory computer-readable storage medium having computer-executable instructions embodied thereon for labeling a three dimensional (3D) model using a virtual reality (VR) technique, wherein when executed by at least one processor, the computer-executable instructions cause the at least one processor to:
receive a 3D model including at least one environmental feature that is unlabeled;
display, through a VR device in communication with the processor, a VR environment to a user representing the 3D model, the VR environment modifiable by the user;
display, in the VR environment, a prompt requesting the user to input labeling data for the at least one environmental feature displayed within the VR environment and one or more rules associated with the at least one environmental feature, the labeling data inputted by selecting the at least one environmental feature through gesture icons interac

Assignees

• State Farm Mutual Automobile Insurance Co

Inventors

• Nussbaum Bryan
• Carnahan Jeremy
• Knuffman Ryan

Classifications

• G06T17/205 Primary

  Re-meshing · CPC title

• G06F16/5866

  using information manually generated, e.g. tags, keywords, comments, manually generated location and time information · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06V20/20 Primary

  in augmented reality scenes · CPC title

• H04W4/021

  Services related to particular areas, e.g. point of interest [POI] services, venue services or geofences · CPC title