Computer vision and speech algorithm design service

Patent metadata

Field	Value
Publication number	US-2022261516-A1
Application number	US-202217737911-A
Country	US
Kind code	A1
Filing date	May 5, 2022
Priority date	May 8, 2018
Publication date	Aug 18, 2022
Grant date	—

Abstract

A synthetic world interface may be used to model digital environments, sensors, and motions for the evaluation, development, and improvement of computer vision and speech algorithms. A synthetic data cloud service with a library of sensor primitives, motion generators, and environments with procedural and game-like capabilities, facilitates engineering design for a manufactural solution that has computer vision and speech capabilities. In some embodiments, a sensor platform simulator operates with a motion orchestrator, an environment orchestrator, an experiment generator, and an experiment runner to test various candidate hardware configurations and computer vision and speech algorithms in a virtual environment, advantageously speeding development and reducing cost. Thus, examples disclosed herein may relate to virtual reality (VR) or mixed reality (MR) implementations.

First claim

What is claimed is:

1 . A system for selecting a hardware configuration in a simulation of one or more hardware configurations for a sensor platform, the sensor platform comprising one or more virtual sensors and having an environment simulation generated for one or more virtual environments, comprising:
memory embodied with executable instructions for simulating the one or more hardware configurations comprising one or more virtual sensors; and at least one processor programmed for:
generating a motion profile simulating motion of the one or more hardware configurations within the one or more virtual environments;
generating synthetic experiment data for the one or more hardware configurations having the simulated motion within the one or more virtual environments, wherein the synthetic experiment data comprises inertial measurement unit (IMU) data;
simulating movement of the one or more hardware configurations with the simulated motion in the one or more virtual environments;
applying an object tracking service to the simulated movement of the one or more hardware configurations with the simulated motion in the one or more virtual environments to determine performance of object tracking by the one or more hardware configurations;
determining disparity data of a simulated hardware configuration compared with a ground truth for the simulated hardware configuration;
determining the disparity data exceeds a variance threshold from the ground truth; and
based on the disparity data exceeding the variance threshold, directing an artificial intelligence (AI) application to run subsequent testing of the object tracking service against a different synthetic scene, motion, or hardware configuration.

2 . The system of claim 1 wherein a first hardware configuration is a first camera and a second hardware configuration is a second camera that is different than the first camera.

3 . The system of claim 1 wherein the at least one processor is further programmed for comparing a computer vision algorithm with ground truth data.

4 . The system of claim 1 wherein a first hardware configuration is a first mobile phone and a second hardware configuration is a second mobile phone that is different than the first mobile phone.

5 . The system of claim 1 wherein the synthetic experiment data comprises inertial measurement unit (IMU) data.

6 . The system of claim 5 wherein the IMU data comprises accelerometer data.

7 . The system of claim 5 wherein the IMU data comprises gyroscope data.

8 . The system of claim 1 wherein the variance threshold being a certain percentage.

9 . A method for selecting a hardware configuration in a simulation of one or more hardware configurations for a sensor platform, the sensor platform comprising one or more virtual sensors and having an environment simulation generated for one or more virtual environments, the method comprising:
generating a motion profile simulating motion of the one or more hardware configurations within the one or more virtual environments; generating synthetic experiment data for the one or more hardware configurations having the simulated motion within the one or more virtual environments, wherein the synthetic experiment data comprises inertial measurement unit (IMU) data; simulating movement of the one or more hardware configurations with the simulated motion in the one or more virtual environments; applying an object tracking service to the simulated movement of the one or more hardware configurations with the simulated motion in the one or more virtual environments to determine performance of object tracking by the one or more hardware configurations; determining disparity data of a simulated hardware configuration compared with a ground truth for the simulated hardware configuration; determining the disparity data exceeds a variance threshold from the ground truth; and based on the disparity data exceeding the variance threshold, directing an artificial intelligence (AI) application to run subsequent testing of the object tracking service against a different synthetic scene, motion, or hardware configuration.

10 . The method of claim 9 wherein generating synthetic experiment data comprises simulating, for the one or more hardware configurations, sensor data that can be supplied to a computer vision or speech algorithm.

11 . The method of claim 10 further comprising:
evaluating the computer vision or speech algorithm by comparing a computer vision or speech algorithm output data with ground truth data.

12 . The method of claim 9 wherein the IMU data comprises accelerometer data.

13 . The method of claim 9 wherein the IMU data comprises gyroscope data.

14 . One or more computer storage devices having computer-executable instructions stored thereon for developing a computer speech solution, which, on execution by a computer, cause the computer to perform operations, comprising:
simulating one or more hardware configurations comprising one or more virtual sensors; simulating one or more virtual environments; simulating motion of the one or more simulated hardware configurations within the one or more virtual environments; generating synthetic experiment data for the one or more hardware configurations having the simulated motion within the one or more virtual environments, wherein the synthetic experiment data comprises inertial measurement unit (IMU) data; simulating movement of the one or more hardware configurations with the simulated motion in the one or more virtual environments; applying an object tracking service to the simulated movement of the one or more hardware configurations with the simulated motion in the one or more virtual environments to determine performance of the one or more hardware configurations as a computer vision solution; determining disparity data of a simulated hardware configuration compared with a ground truth for the simulated hardware configuration; determining the disparity data exceeds a variance threshold from the ground truth; and based on the disparity data exceeding the variance threshold, directing an artificial intelligence (AI) application to run subsequent testing of the object tracking service against a different synthetic scene, motion, or hardware configuration.

15 . The one or more computer storage devices of claim 14 wherein the operations further comprise:
modifying the computer speech solution based on disparity data between the performance of the two or more hardware configurations and ground truth data.

16 . The one or more computer storage devices of claim 14 wherein the synthetic experiment data comprises inertial measurement unit (IMU) data.

17 . The one or more computer storage devices of claim 16 wherein the IMU data comprises accelerometer data.

18 . The one or more computer storage devices of claim 16 , wherein the IMU data comprises gyroscope data.

19 . The one or more computer storage devices of claim 14 , wherein generating synthetic experiment data comprises simulating, for the one or more hardware configurations, sensor data that can be supplied to a computer vision or speech algorithm.

20 . The one or more computer storage devices of claim 14 , wherein a first hardware configuration is a first camera and a second hardware configuration is a second camera that is different than the first camera.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Ebstyne Michael
• Escos Pedro Urbina
• Pekelny Yuri
• Chan Jonathan Chi Hang
• Shalev Emanuel
• Kipman Alex
• Flick Mark

Classifications

• G06V10/774 Primary

  Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06F30/20 Primary

  Design optimisation, verification or simulation (optimisation, verification or simulation of circuit designs G06F30/30) · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06F18/217

  Validation; Performance evaluation; Active pattern learning techniques · CPC title

• G06F30/27

  using machine learning, e.g. artificial intelligence, neural networks, support vector machines [SVM] or training a model · CPC title