Experience learning in virtual world

Patent metadata

Field	Value
Publication number	US-11568109-B2
Application number	US-202016868265-A
Country	US
Kind code	B2
Filing date	May 6, 2020
Priority date	May 6, 2019
Publication date	Jan 31, 2023
Grant date	Jan 31, 2023

Abstract

A computer-implemented method of machine-learning is described that includes obtaining a dataset of virtual scenes. The dataset of virtual scenes belongs to a first domain. The method further includes obtaining a test dataset of real scenes. The test dataset belongs to a second domain. The method further includes determining a third domain. The third domain is closer to the second domain than the first domain in terms of data distributions. The method further includes learning a domain-adaptive neural network based on the third domain. The domain-adaptive neural network is a neural network configured for inference of spatially reconfigurable objects in a real scene. Such a method constitutes an improved method of machine learning with a dataset of scenes including spatially reconfigurable objects.

First claim

The invention claimed is:

1. A computer-implemented method of machine-learning, comprising:
obtaining:
a dataset of virtual scenes, the dataset of virtual scenes belonging to a first domain, and
a test dataset of digital images of real scenes belonging to a second domain;

determining a third domain, the third domain being closer to the second domain than the first domain in terms of data distributions, closeness between domains being quantified by a distance, the distance between the third domain and the second domain being smaller than the distance between the second domain and the first domain; and
learning a domain-adaptive neural network based on the third domain, the domain-adaptive neural network being a neural network configured for inference of spatially reconfigurable objects in a real scene in that the domain-adaptive neural network is configured to take as input a digital image of a real-scene and to detect one or more spatially reconfigurable objects in the real scene.

2. The method of claim 1 , wherein the determining of the third domain includes, for each scene of the dataset of virtual scenes:
extracting one or more spatially reconfigurable objects from the scene; and
for each extracted object, transforming the extracted object into an object closer to the second domain than the extracted object in terms of data distributions.

3. The method of claim 2 , wherein the determining of the third domain further comprises:
for each transformed extracted object, placing the transformed extracted object in one or more scenes each closer to the second domain than the first domain in terms of data distributions, the third domain including each scene on which the transformed extracted object is placed.

4. The method of claim 3 , wherein the placing of the transformed extracted object in the one or more scenes is carried out randomly.

5. The method of claim 3 , wherein the determining of the third domain including placing one or more distractors in each respective one of one or more scenes included in the third domain, a distractor being an object which is not a spatially reconfigurable object.

6. The method of claim 1 , wherein the learning of the domain-adaptive neural network further comprises:
obtaining a teacher extractor, the teacher extractor being a machine-learned neural network configured for outputting image representations of real scenes; and
training a student extractor, the student extractor being a neural network configured for outputting image representations of scenes belonging to the third domain, the training of the student extractor including minimizing a loss which, for each scene of one or more real scenes, penalizes a disparity between a result of applying the teacher extractor to the scene and a result of applying the student extractor to the scene.

7. The method of claim 6 , wherein the result of applying the teacher extractor to the scene is a first Gram matrix and the result of applying the student extractor to the scene is a second Gram matrix.

8. The method of claim 7 , wherein:
the first Gram matrix is computed on several layers of neurons of the teacher extractor, said layers of neurons including at least the last layer of neurons, and
the second Gram matrix is computed on several layers of neurons of the student extractor, said layers of neurons including at least the last layer of neurons.

9. The method of claim 6 , wherein the disparity is a Euclidean distance between the result of applying the teacher extractor to the scene and the result of applying the student extractor to the scene.

10. The method of claim 1 , wherein each virtual scene of the dataset of virtual scenes is a virtual manufacturing scene includes one or more spatially reconfigurable manufacturing tools, the domain-adaptive neural network being configured for inference of spatially reconfigurable manufacturing tools in a real manufacturing scene.

11. The method of claim 1 , wherein each real scene of the test dataset is a real manufacturing scene includes one or more spatially reconfigurable manufacturing tools, the domain-adaptive neural network being configured for inference of spatially reconfigurable manufacturing tools in a real manufacturing scene.

12. A non-transitory data storage medium having recorded thereon a domain-adaptive neural network learnable according to a computer-implemented method of machine-learning, the method comprising:
obtaining:
a dataset of virtual scenes, the dataset of virtual scenes belonging to a first domain, and
a test dataset of digital images of real scenes belonging to a second domain;

determining a third domain, the third domain being closer to the second domain than the first domain in terms of data distributions, closeness between domains being quantified by a distance, the distance between the third domain and the second domain being smaller than the distance between the second domain and the first domain; and
learning a domain-adaptive neural network based on the third domain, the domain-adaptive neural network being a neural network configured for inference of spatially reconfigurable objects in a real scene in that the domain-adaptive neural network is configured to take as input a digital image of a real-scene and to detect one or more spatially reconfigurable objects in the real scene.

13. The non-transitory data storage medium of claim 12 , wherein the determining of the third domain includes, for each scene of the dataset of virtual scenes:
extracting one or more spatially reconfigurable objects from the scene; and
for each extracted object, transforming the extracted object into an object closer to the second domain than the extracted object in terms of data distributions.

14. The non-transitory data storage medium of claim 13 , wherein the determining of the third domain further comprises:
for each transformed extracted object, placing the transformed extracted object in one or more scenes each closer to the second domain than the first domain in terms of data distributions, the third domain including each scene on which the transformed extracted object is placed.

15. The non-transitory data storage medium of claim 14 , wherein the placing of the transformed extracted object in the one or more scenes is carried out randomly.

16. A device comprising:
a non-transitory data storage medium having recorded thereon a computer program having instructions for machine-learning that when executed a processor causes the processor to be configured to:
obtain:
a dataset of digital images of virtual scenes, the dataset of virtual scenes belonging to a first domain, and
a test dataset of real scenes belonging to a second domain,

determine a third domain, the third domain being closer to the second domain than the first domain in tennis of data distributions, closeness between domains being quantified by a distance, the distance between the third domain and the second domain being smaller than the distance between the second domain and the first domain, and
learn a domain-adaptive neural network based on the third domain, the domain-adaptive neural network being a neural network configured for inference of spatially reconfigurable objects in a real scene in that the domain-adaptive neural network is configured to take as input a digital image of a real-scene and to detect one or more spatially reconfigurable objects in the real scene.

17. The device of claim 16 , wherein the processing is further configured to determine the third domain by being configured to, for each scene of the dataset of virtual scenes:

Assignees

• Dassault Systemes

Inventors

• Rejeb Sfar Asma
• Mezghanni Mariem
• Boulkenafed Malika

Classifications

• G06F30/17

  Mechanical parametric or variational design · CPC title

• G06N3/0454

  Physics · mapped topic

• G06F30/12 Primary

  characterised by design entry means specially adapted for CAD, e.g. graphical user interfaces [GUI] specially adapted for CAD · CPC title

• G06N5/04

  Inference or reasoning models · CPC title

• G06N3/0472

  Physics · mapped topic