Spatial localization design service

Patent metadata

Field	Value
Publication number	US-11087176-B2
Application number	US-201815974665-A
Country	US
Kind code	B2
Filing date	May 8, 2018
Priority date	May 8, 2018
Publication date	Aug 10, 2021
Grant date	Aug 10, 2021

Abstract

A synthetic world interface may be used to model digital environments, sensors, and motions for the evaluation, development, and improvement of localization algorithms. A synthetic data cloud service with a library of sensor primitives, motion generators, and environments with procedural and game-like capabilities, facilitates engineering design for a manufactural solution that has localization capabilities. In some embodiments, a sensor platform simulator operates with a motion orchestrator, an environment orchestrator, an experiment generator, and an experiment runner to test various candidate hardware configurations and localization algorithms in a virtual environment, advantageously speeding development and reducing cost. Thus, examples disclosed herein may relate to virtual reality (VR) or mixed reality (MR) implementations.

First claim

What is claimed is:

1. A system, comprising:
memory embodied with executable instructions for simulating a first hardware configuration and a second hardware configuration each comprising two or more virtual cameras; and
at least one processor programmed for:
simulating one or more virtual environments;
simulating motion of the one or more simulated hardware configurations within the one or more virtual environments;
generating synthetic experiment data for the one or more simulated hardware configurations having the simulated motion within the one or more virtual environments;
generating the synthetic experiment data for the first hardware configuration and the second hardware configuration with the simulated motion in the one or more virtual environments;
simulating movement of the one or more hardware configurations with the simulated motion in the one or more virtual environments;
applying one or more localization algorithms to the simulated movement of the one or more hardware configurations with the simulated motion in the one or more virtual environments to determine performance of the one or more hardware configurations as a localization solution;
determining disparity data of the simulated hardware configuration compared with a ground truth for the simulated hardware configuration;
determining the disparity data exceeds a variance threshold from the ground truth; and
based on the disparity data exceeding the variance threshold, directing an artificial intelligence (AI) application to run subsequent testing of the one or more localization algorithms against a different synthetic scene, motion, or hardware configuration.

2. The system of claim 1 wherein a first hardware configuration comprises a first lens and a first sensor, wherein a second hardware configuration comprises a second lens and a second sensor, wherein the first lens differs from the second lens and the first sensor differs from the second sensor.

3. The system of claim 1 wherein the at least one processor is further programmed for comparing localization algorithm output data with the ground truth data.

4. The system of claim 1 wherein a first hardware configuration is a first camera and a second hardware configuration is a second camera that is different than the first camera.

5. The system of claim 1 wherein the synthetic experiment data comprises inertial measurement unit (IMU) data.

6. The system of claim 5 wherein the IMU data comprises accelerometer data.

7. The system of claim 5 wherein the IMU data comprises gyroscope data.

8. A method comprising:
importing, into a simulation, one or more hardware configurations for a sensor platform comprising one or more virtual cameras;
generating an environment simulation for one or more virtual environments;
generating a motion profile simulating motion of the one or more hardware configurations within the one or more virtual environments;
generating synthetic experiment data for the one or more hardware configurations having the simulated motion within the one or more virtual environments;
iterating the generation of synthetic experiment data for the one or more hardware configurations, the virtual environment and the motion;
simulating movement of the one or more hardware configurations with the simulated motion in the one or more virtual environments;
applying one or more localization algorithms to the simulated movement of the one or more hardware configurations with the simulated motion in the one or more virtual environments to determine performance of the one two or more hardware configurations as a localization solution;
determining disparity data of the simulated hardware configuration compared with a ground truth for the simulated hardware configuration;
determining the disparity data exceeds a variance threshold from the ground truth; and
based on the disparity data exceeding the variance threshold, directing an artificial intelligence (AI) application to run subsequent testing of the one or more localization algorithms against a different synthetic scene, motion, or hardware configuration.

9. The method of claim 8 wherein generating synthetic experiment data comprises simulating, for the one or more hardware configurations, sensor data that can be supplied to a localization algorithm.

10. The method of claim 9 further comprising:
evaluating the localization solution by comparing a localization algorithm output data with ground truth data.

11. The method of claim 8 wherein the synthetic experiment data comprises synthetic images.

12. The method of claim 8 wherein the synthetic experiment data comprises inertial measurement unit (IMU) data.

13. The method of claim 12 wherein the IMU data comprises accelerometer data.

14. The method of claim 12 wherein the IMU data comprises gyroscope data.

15. One or more computer storage devices having computer-executable instructions stored thereon for developing a localization solution, which, on execution by a computer, cause the computer to perform operations comprising:
simulating one or more hardware configurations comprising one or more virtual cameras;
simulating one or more virtual environments;
simulating motion of the one or more simulated hardware configurations within the one or more virtual environments;
generating synthetic experiment data for a plurality of candidate localization solutions having differing hardware configurations or localization algorithm parameters;
iterating the experiment generator to generate the synthetic experiment data for the one or more hardware configurations, the virtual environment, and motion;
simulating movement of the one or more hardware configurations with the simulated motion in the one or more virtual environments to determine performance of the one or more hardware configurations as a localization solution;
applying one or more localization algorithms to the simulated movement of the one or more hardware configurations with the simulated motion in the one or more virtual environments to determine performance of the one or more hardware configurations as a localization solution;
determining disparity data of the simulated hardware configuration compared with a ground truth for the simulated hardware configuration;
determining the disparity data exceeds a variance threshold from the ground truth; and
based on the disparity data exceeding the variance threshold, directing an artificial intelligence (AI) application to run subsequent testing of the one or more localization algorithms against a different synthetic scene, motion, or hardware configuration.

16. The one or more computer storage devices of claim 15 wherein the operations further comprise:
comparing localization algorithm output data with ground truth data.

17. The one or more computer storage devices of claim 15 wherein the synthetic experiment data comprises synthetic images.

18. The one or more computer storage devices of claim 15 wherein the synthetic experiment data comprises inertial measurement unit (IMU) data.

19. The one or more computer storage devices of claim 18 wherein the IMU data comprises accelerometer data.

20. The one or more computer storage devices of claim 18 wherein the IMU data comprises gyroscope data.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Ebstyne Michael
• Escos Pedro Urbina
• Shalev Emanuel
• Kipman Alex
• Pekelny Yuri
• Chan Jonathan Chi Hang

Classifications

• G06V10/774 Primary

  Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06F18/217

  Validation; Performance evaluation; Active pattern learning techniques · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06F30/20

  Design optimisation, verification or simulation (optimisation, verification or simulation of circuit designs G06F30/30) · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title