Audio logging for model training and onboard validation utilizing autonomous driving vehicle

Patent metadata

Field	Value
Publication number	US-11735205-B2
Application number	US-202117248172-A
Country	US
Kind code	B2
Filing date	Jan 12, 2021
Priority date	Jan 12, 2021
Publication date	Aug 22, 2023
Grant date	Aug 22, 2023

Abstract

Systems and methods for generating labelled audio data and onboard validation of the labelled audio data utilizing an autonomous driving vehicle (ADV) while the ADV is operating within a driving environment are disclosed. The method includes recording a sound emitted by an object within the driving environment of the ADV, and converting the recorded sound into audio samples. The method further includes labelling the audio samples, and refining the labelled audio samples to produce refined labelled audio data. The refined labelled audio data is utilized to subsequently train a machine learning algorithm to recognize a sound source during autonomous driving of the ADV. The method further includes generating a performance profile of the refined labelled audio data based at least on the audio samples, a position of the object, and a relative direction of the object. The position of the object and the relative direction of the object are determined by a perception system of the ADV.

First claim

What is claimed is:

1. A method of generating labelled audio data and onboard validation of the labelled audio data utilizing an autonomous driving vehicle (ADV) while the ADV is operating within a driving environment, the method comprising:
recording a sound emitted by an object within the driving environment of the ADV, and converting the recorded sound into audio samples;
labelling the audio samples, and refining the labelled audio samples to produce refined labelled audio data, wherein the refined labelled audio data is utilized to subsequently train a machine learning algorithm to recognize a sound source during autonomous driving of the ADV; and
generating a performance profile of the refined labelled audio data based at least on the audio samples, a position of the object, and a relative direction of the object, wherein the position of the object and the relative direction of the object are determined by a perception system of the ADV, wherein using the audio samples, the position of the object, and the relative direction of the object to generate the performance profile comprises profiling the refined labelled audio data against the audio samples, the position of the object, and the relative direction of the object.

2. The method of claim 1 , wherein generating the performance profile of the refined labelled audio data further comprises determining the position of the object and the relative direction of the object based on sensors data provided by visual sensors of the ADV, wherein the visual sensors are coupled to the perception system.

3. The method of claim 1 , wherein labelling the audio samples comprises tagging the audio samples with an audio sample identifier (ID), one or more positions associated with the audio samples, and a direction associated with the audio samples.

4. The method of claim 1 , wherein the object is an emergency vehicle and the emitted sound is a siren sound.

5. The method of claim 1 , wherein the audio samples are manually labelled by a user of the ADV.

6. The method claim 1 , wherein the performance profile is stored locally in a persistent storage device in the ADV.

7. A computer-implemented method for onboard validation of labelled audio data utilizing an autonomous driving vehicle (ADV) while the ADV is operating within a driving environment, the method comprising:
recording a sound emitted by an obstacle within the driving environment of the ADV to create audio samples;
determining a position of the obstacle and a relative direction of the obstacle based on sensors data provided by visual sensors of the ADV; and
using the audio samples, the position of the obstacle, and the relative direction of the obstacle to generate a performance profile of refined labelled audio data, wherein the refined labelled audio data is generated by labelling the audio samples and refining the labelled audio samples, and is utilized to subsequently train a machine learning algorithm to recognize a sound source during autonomous driving of the ADV, wherein using the audio samples, the position of the obstacle, and the relative direction of the obstacle to generate the performance profile comprises profiling the refined labelled audio data against the audio samples, the position of the obstacle, and the relative direction of the obstacle.

8. The method of claim 7 , wherein the labelled audio samples comprise the audio samples, an audio sample identifier (ID), one or more positions associated with the audio samples, and a direction associated with the audio samples.

9. The method of claim 7 , wherein the obstacle is an emergency vehicle and the emitted sound is a siren sound.

10. The method of claim 7 , wherein the audio samples are manually labelled by a user of the ADV.

11. The method of claim 7 , wherein the performance profile is stored locally in a persistent storage device in the ADV.

12. A system for onboard validation of labelled audio data, comprising:
a processor; and
a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations, the operations including
recording a sound emitted by an obstacle within a driving environment of an autonomous driving vehicle (ADV) to create audio samples;
determining a position of the obstacle and a relative direction of the obstacle based on sensors data provided by visual sensors of the ADV; and
using the audio samples, the position of the obstacle, and the relative direction of the obstacle to generate a performance profile of refined labelled audio data, wherein the refined labelled audio data is generated by labelling the audio samples and refining the labelled audio samples, and is utilized to subsequently train a machine learning algorithm to recognize a sound source during autonomous driving of the ADV, wherein using the audio samples, the position of the obstacle, and the relative direction of the obstacle to generate the performance profile comprises profiling the refined labelled audio data against the audio samples, the position of the obstacle, and the relative direction of the obstacle.

13. The system of claim 12 , wherein the labelled audio samples comprise the audio samples, an audio sample identifier (ID), one or more positions associated with the audio samples, and a direction associated with the audio samples.

14. The system of claim 12 , wherein the obstacle is an emergency vehicle and the emitted sound is a siren sound.

15. The system of claim 12 , wherein the audio samples are manually labelled by a user of the ADV.

16. The system of claim 12 , wherein the performance profile is stored locally in a persistent storage device in the ADV.

17. A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor of an autonomous driving vehicle (ADV), cause the ADV to perform operations, the operations comprising:
recording a sound emitted by an obstacle within the ADV to create audio samples;
determining a position of the obstacle and a relative direction of the obstacle based on sensors data provided by visual sensors of the ADV; and
using the audio samples, the position of the obstacle, and the relative direction of the obstacle to generate a performance profile of refined labelled audio data, wherein the refined labelled audio data is generated by labelling the audio samples and refining the labelled audio samples, and is utilized to subsequently train a machine learning algorithm to recognize a sound source during autonomous driving of the ADV, wherein using the audio samples, the position of the obstacle, and the relative direction of the obstacle to generate the performance profile comprises profiling the refined labelled audio data against the audio samples, the position of the obstacle, and the relative direction of the obstacle.

18. The non-transitory machine-readable medium of claim 17 , wherein the labelled audio samples comprise the audio samples, an audio sample identifier (ID), one or more positions associated with the audio samples, and a direction associated with the audio samples.

19. The non-transitory machine-readable medium of claim 17 , wherein the obstacle is an emergency vehicle and the emitted sound is a siren sound.

20. The non-transitory machine-readable medium of claim 17 , wherein the audio samples are manually labelled by a user of the ADV.

21. The non-transitory machine-readable medium of claim 17 , wherein the performance profile is stored locally in a persistent storage de

Assignees

• Baidu Usa Llc

Inventors

• Luo Qi
• Xu Kecheng
• Sun Hongyi
• Reynolds Wesley
• Lin Zejun
• Wang Wei
• Cao Yu
• Lin Weiman

Classifications

• G05D1/0221

  involving a learning process · CPC title

• G05D1/0246

  using a video camera in combination with image processing means · CPC title

• G05D1/0255

  using acoustic signals, e.g. ultra-sonic singals (sonar systems designed for anti-collision purposes G01S15/93) · CPC title

• G10L25/51 Primary

  for comparison or discrimination · CPC title

• G06N20/00

  Machine learning · CPC title