Automatically generating machine-learning training data

Patent metadata

Field	Value
Publication number	US-12288402-B2
Application number	US-202217850241-A
Country	US
Kind code	B2
Filing date	Jun 27, 2022
Priority date	Jun 27, 2022
Publication date	Apr 29, 2025
Grant date	Apr 29, 2025

Abstract

A computer includes a processor and a memory, and the memory stores instructions executable by the processor to receive first environmental data recorded by an environmental sensor on board a vehicle, receive nonenvironmental data recorded on board the vehicle independently of the first environmental data, add a plurality of annotations derived from the nonenvironmental data to the environmental data, and train a machine-learning program to process second environmental data by using the first environmental data as training data and the annotations as ground truth for the first environmental data.

First claim

The invention claimed is:

1. A computer comprising a processor and a memory, the memory storing instructions executable by the processor to:
receive second environmental data of an area outside a second vehicle;
execute a machine-learning program with the second environmental data as input; and
actuate a component of the second vehicle based on the execution of the machine-learning program;
wherein the machine-learning program is trained by:
receiving first environmental data of an area outside a first vehicle recorded by an environmental sensor on board the first vehicle;
receiving nonenvironmental data recorded on board the first vehicle about an inside of the first vehicle, the nonenvironmental data including at least one of audio data or motion data, the motion data indicating motion of the first vehicle, the nonenvironmental data being time-synchronized with the first environmental data;
receiving speed data that is time-synchronized with the first environmental data;
determining a plurality of annotation times for a plurality of respective annotations based on the speed data, the annotations being derived from the nonenvironmental data at a plurality of respective record times, wherein determining the annotation times includes applying offsets to the record times, resulting in offsetted times, and applying the annotations at the offsetted times instead of the record times, the offsets dependent on the speed data;
adding the annotations derived from the at least one of the audio data or the motion data to the first environmental data, adding the annotations being based on the speed data, the speed data indicating a speed at which the first vehicle is traveling, wherein adding the annotations includes adding the annotations at the respective annotation times; and
training the machine-learning program to process the second environmental data by using the first environmental data as training data and the annotations as ground truth for the first environmental data.

2. The computer of claim 1 , wherein the component includes at least one of a propulsion system, a brake system, or a steering system.

3. The computer of claim 1 , wherein the environmental sensor generates the first environmental data at a constant data rate, and determining the annotation times is based on the data rate.

4. The computer of claim 1 , wherein the nonenvironmental data includes the audio data, and the annotations are derived from the audio data.

5. The computer of claim 4 , wherein the machine-learning program is further trained by converting the audio data to text, and the annotations are derived from the text.

6. The computer of claim 4 , wherein the audio data is recorded from a passenger cabin of the first vehicle.

7. The computer of claim 1 , wherein the nonenvironmental data includes data transmitted over a vehicle network of the first vehicle.

8. The computer of claim 7 , wherein the nonenvironmental data is generated by components of the first vehicle.

9. The computer of claim 7 , wherein the nonenvironmental data is automatically generated on board the first vehicle.

10. The computer of claim 7 , wherein the vehicle network includes a controller area network (CAN) bus.

11. The computer of claim 1 , wherein the first environmental data is based on electromagnetic radiation.

12. The computer of claim 1 , wherein the first environmental data is first video data, the environmental sensor is a camera, and the second environmental data is second video data.

13. The computer of claim 12 , wherein the machine-learning program is trained to perform image recognition.

14. The computer of claim 1 , wherein, when used to train the machine-learning program, the first environmental data lacks manually added annotations.

15. The computer of claim 1 , wherein the nonenvironmental data includes the motion data indicating the motion of the first vehicle, and the annotations are derived from the motion data.

16. A method comprising:
receiving first environmental data of an area outside a first vehicle recorded by an environmental sensor on board the first vehicle;
receiving nonenvironmental data recorded on board the first vehicle about an inside of the first vehicle, the nonenvironmental data including at least one of audio data or motion data, the motion data indicating motion of the first vehicle, the nonenvironmental data being time-synchronized with the first environmental data;
receiving speed data that is time-synchronized with the first environmental data;
determining a plurality of annotation times for a plurality of respective annotations based on the speed data, the annotations being derived from the nonenvironmental data at a plurality of respective record times, wherein determining the annotation times includes applying offsets to the record times, resulting in offsetted times, and applying the annotations at the offsetted times instead of the record times, the offsets dependent on the speed data;
adding the annotations derived from the at least one of the audio data or the motion data to the first environmental data, adding the annotations being based on the speed data, the speed data indicating a speed at which the first vehicle is traveling, wherein adding the annotations includes adding the annotations at the respective annotation times;
training a machine-learning program to process second environmental data by using the first environmental data as training data and the annotations as ground truth for the first environmental data;
installing the machine-learning program on a vehicle computer of a second vehicle; and
actuating a component of the second vehicle based on the machine-learning program.

Assignees

• Ford Global Tech Llc

Inventors

• Parenti Robert
• Siddiqui Adil Nizam
• Neubecker Cynthia M

Classifications

• B60W2420/408

  Radar; Laser, e.g. lidar · CPC title

• B60W2420/403

  Image sensing, e.g. optical camera · CPC title

• B60W40/08

  related to drivers or passengers · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V20/59

  inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions · CPC title