Sensor data segmentation

Patent metadata

Field	Value
Publication number	US-10535138-B2
Application number	US-201715820245-A
Country	US
Kind code	B2
Filing date	Nov 21, 2017
Priority date	Nov 21, 2017
Publication date	Jan 14, 2020
Grant date	Jan 14, 2020

Abstract

A system may include one or more processors configured to receive a plurality of images representing an environment. The images may include image data generated by an image capture device. The processors may also be configured to transmit the image data to an image segmentation network configured to segment the images. The processors may also be configured to receive sensor data associated with the environment including sensor data generated by a sensor of a type different than an image capture device. The processors may be configured to associate the sensor data with segmented images to create a training dataset. The processors may be configured to transmit the training dataset to a machine learning network configured to run a sensor data segmentation model, and train the sensor data segmentation model using the training dataset, such that the sensor data segmentation model is configured to segment sensor data.

First claim

What is claimed is:

1. A system comprising:
one or more processors; and
one or more computer readable storage media communicatively coupled to the one or more processors and storing instructions executable by the one or more processors to:
receive a plurality of images representing an environment, the plurality of images comprising image data generated by one or more image capture devices;
transmit the image data to an image segmentation network configured to segment the plurality of images and generate segmented images;
receive first sensor data associated with the environment, the first sensor data received from a light detection and ranging (LIDAR) sensor;
associate the first sensor data with the segmented images to create a training dataset;
transmit the training dataset to a machine learning network configured to run a sensor data segmentation model; and
train the sensor data segmentation model using the training dataset such that, upon receiving input comprising additional LIDAR data and excluding additional image data, the sensor data segmentation model is configured to output segmented additional LIDAR sensor data.

2. The system of claim 1 , wherein the instructions are further executable by the one or more processors to:
project at least a first portion of the first sensor data onto a first segmented image of the segmented images; and
project at least a second portion of the first sensor data onto a second segmented image of the segmented images.

3. The system of claim 1 , wherein:
a first image of the plurality of images is captured at a first image time;
a second image of the plurality of images is captured at a second image time;
a first portion of the first sensor data is associated with a first sensor time;
a second portion of the first sensor data is associated with a second sensor time; and
the instructions are further executable by the one or more processors to:
determine a first time difference between the first image time and the first sensor time;
determine a second time difference between the second image time and the first sensor time;
determine that the first time difference is less than the second time difference; and
associate the first image with the first portion of the first sensor data based at least in part on the first time difference being less than the second time difference.

4. The system of claim 1 , wherein the instructions are further executable by the one or more processors to receive second sensor data from the LIDAR sensor and segment second sensor data based at least in part on the sensor data segmentation model.

5. The system of claim 1 , wherein the sensor data segmentation model comprises one or more kernels, the one or more kernels associated with an asymmetric stride, and
wherein training the sensor data segmentation model comprises computing a loss function, the loss function based at least in part on an output of the sensor data segmentation model and the training dataset, the loss function comprising one or more of a cross-entropy softmax loss, a focal loss, or a logistic regression loss.

6. The system of claim 1 , wherein the instructions are further executable by the one or more processors to:
identify discontinuities in the first sensor data; and
delete sensor data associated with the discontinuities.

7. The system of claim 1 , wherein the instructions are further executable by the one or more processors to:
receive sensor data from a LIDAR sensor; and
segment the sensor data received from the LIDAR sensor using the sensor data segmentation model and generate segmented sensor data.

8. The system of claim 7 , wherein the instructions are further executable by the one or more processors to generate a trajectory for an autonomous vehicle based at least in part on the segmented sensor data.

9. A method comprising:
receiving a plurality of images representing an environment, the plurality of images comprising image data generated by an image capture device;
transmitting the image data to an image segmentation network configured to segment the plurality of images and generate segmented images;
receiving first sensor data generated by a first sensor comprising a light detection and ranging (LIDAR) sensor, the first sensor data comprising data representative of the environment;
segmenting the image data to generate segmented images;
associating the first sensor data with the segmented images to create a training dataset;
transmitting the training dataset to a machine learning network configured to run a sensor data segmentation model; and
training the sensor data segmentation model using the training dataset such that, upon receiving input comprising additional LIDAR data and excluding additional image data, the sensor data segmentation model is configured to output segmented additional LIDAR data.

10. The method of claim 9 , wherein receiving the first sensor data generated by the first sensor comprises receiving sensor data generated by a light detection and ranging (LIDAR) sensor, and training the sensor data segmentation model comprises training the sensor data segmentation model.

11. The method of claim 9 , further comprising:
projecting at least a first portion of the first sensor data onto a first segmented image of the segmented images; and
projecting at least a second portion of the first sensor data onto a second segmented image of the segmented images.

12. The method of claim 9 , wherein:
a first image of the plurality of images is captured at a first image time;
a second image of the plurality of images is captured at a second image time;
a first portion of the first sensor data is associated with the first image time;
a second portion of the first sensor data is associated with a second sensor time; and
the method further comprises:
determining a first time difference between the first image time and the first sensor time;
determining a second time difference between the second image time and the first sensor time;
determining that the first time difference is less than the second time difference; and
associating the first image with the first portion of the first sensor data based at least in part on the first time difference being less than the second time difference.

13. The method of claim 9 , further comprising receiving second sensor data from the first sensor and segmenting the second sensor data based at least in part on the sensor data segmentation model.

14. The method of claim 9 , wherein the sensor data segmentation model comprises one or more kernels, the one or more kernels associated with an asymmetric stride, and
wherein training the sensor data segmentation model comprises computing a loss function, the loss function based at least in part on an output of the sensor data segmentation model and the training dataset, the loss function comprising one or more of a cross-entropy softmax loss, a focal loss, or a logistic regression loss.

15. The method of claim 9 , further comprising:
identifying discontinuities in the first sensor data; and
deleting first sensor data associated with the discontinuities.

16. The method of claim 9 , further comprising:
receiving second sensor data from a sensor; and
segmenting the second sensor data received from the sensor using the sensor data segmentation model.

17. The method of claim 16 , further comprising generating a trajectory for an autonomous vehicle based at least in part on the segmented second sensor data.

18. A computer-readable storage medium having computer-executab

Assignees

• Zoox Inc

Inventors

• Pfeiffer David

Classifications

• G05D1/622

  Obstacle avoidance (predicting or avoiding probable or impending collision of road vehicles B60W30/08) · CPC title

• G05D2101/15

  using machine learning, e.g. neural networks · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G01S17/02

  Systems using the reflection of electromagnetic waves other than radio waves (G01S17/66 takes precedence) · CPC title

• G06T2207/30252

  Vehicle exterior; Vicinity of vehicle · CPC title