Vehicle data relation device and methods therefor

Patent metadata

Field	Value
Publication number	US-12198049-B2
Application number	US-202117211930-A
Country	US
Kind code	B2
Filing date	Mar 25, 2021
Priority date	Mar 25, 2021
Publication date	Jan 14, 2025
Grant date	Jan 14, 2025

Abstract

A vehicle data relation device includes an internal audio/image data analyzer, configured to receive first data representing at least one of audio from within the vehicle or an image from within the vehicle; identify within the first data second data representing an audio indicator or an image indicator, wherein the audio indicator is human speech associated with a significance of an object external to the vehicle, and wherein the image indicator is an action of a human within the vehicle associated with a significance of an object external to the vehicle; an external image analyzer, configured to receive third data representing an image of a vicinity external to the vehicle; identify within the third data an object corresponding to at least one of the audio indicator or the video indicator; and an object data generator, configured to generate data corresponding to the object.

First claim

What is claimed is:

1. A vehicle data relation device, comprising:
an internal audio/image data analyzer, configured to
identify first data representing audio data from within the vehicle or an image from within the vehicle,
identify, from the first data, second data representing an audio indicator or an image indicator,
wherein the audio indicator is human speech, and wherein the image indicator represents an action of a human within the vehicle;

an external image analyzer, configured to:
identify, within third data representing an image of a vicinity external to the vehicle, an object corresponding to at least one of the audio indicator or the image indicator; and

an object data generator, configured to generate object data to classify the object.

2. The vehicle data device of claim 1 , wherein the object data generator is configured to classify the object based on the third data's relevance in training of a trainable model.

3. The vehicle data relation device of claim 1 , wherein the object data comprise at least one of an identity of the object, an action of the object, or a priority of the object.

4. The vehicle data relation device of claim 1 , wherein the internal audio/image data analyzer configured to identify second data comprises the internal audio/image data analyzer configured to identify one or more keywords within the audio.

5. The vehicle data relation device of claim 4 , wherein the internal audio/image data analyzer is configured to send data representing the one or more keywords to the external image analyzer, and the external image analyzer is configured to identify the object based on a relationship between the object and the one or more keywords.

6. The vehicle data relation device of claim 4 , wherein the external image analyzer is configured to iteratively identify the object using at least two keywords.

7. The vehicle data relation device of claim 1 , wherein the internal audio/image data analyzer being configured to identify second data comprises the internal audio/image data analyzer being configured to identify a human gesture within the image; wherein the human gesture is at least one of pointing in a direction, making an attention gesture, making a negating gesture, making a stop gesture, or any combination thereof; and
wherein the external image analyzer is configured to identify the object within the third data based on the human gesture.

8. The vehicle data relation device of claim 7 , wherein the internal audio/image data analyzer is configured to send data representing the human gesture to the external image analyzer and the external image analyzer is configured to map a pointing action to the third data and to identify the object based on a mapped relationship between the pointing action and the object.

9. The vehicle data relation device of claim 1 , wherein the internal audio/image data analyzer being configured to identify second data comprises the internal audio/image data analyzer being configured to identify a human gaze direction within the image; wherein the external image analyzer is configured to identify the object within the third data based on the human gaze direction.

10. The vehicle data relation device of claim 9 , wherein the internal audio/image data analyzer is configured to send data representing the gaze direction to the external image analyzer and the external image analyzer is configured to identify the object at least in part based on a relationship between the object and the gaze direction.

11. The vehicle data relation device of claim 9 , wherein the internal audio/image data analyzer is configured to send data representing the gaze direction to the external image analyzer and the external image analyzer is configured to map the gaze direction to the third data and to identify the object based on a mapped relationship between the gaze direction and the object.

12. The vehicle data relation device of claim 1 , wherein the external image analyzer is configured to identify the object based on at least two of: one or more keywords, one or more gestures, or a gaze direction.

13. The vehicle data relation device of claim 1 , further comprising a vehicle sensor data analyzer configured to receive vehicle sensor data from a vehicle sensor, and wherein the external image analyzer is further configured to generate object data based on the vehicle sensor data, wherein the vehicle sensor comprises a steering sensor, an accelerometer, a braking sensor, a speedometer, or any combination thereof.

14. The vehicle data relation device of claim 1 , further comprising a vehicle actuator data analyzer, configured to receive actuator data, and wherein the external image analyzer is further configured to generate object data based on the vehicle actuator data, wherein the vehicle actuator data comprise data representing a steering wheel position, a brake position, a brake depression, a braking force, speed, velocity, acceleration, or any combination thereof.

15. The vehicle data relation device of claim 14 , wherein the sensor data analyzer and/or the vehicle actuator data analyzer is configured to determine from the sensor data and/or vehicle actuator data an action of the vehicle relative to an object represented by the object data.

16. The vehicle data relation device of claim 1 , wherein the object data generator is configured to determine a priority of the object based on the audio indicator or the image indicator, wherein the priority is based on an importance of avoiding a collision with the object, risk of a collision with the object, an estimated damage associated with a collision with the object, or any combination thereof.

17. The vehicle data relation device of claim 1 , wherein the object data comprise a label of one or more detected objects.

18. The vehicle data relation device of claim 1 , wherein the object data generator is further configured to generate a sensor data label, wherein the sensor data label is a label representing at least one of the identity of the object, the action of the object, or the priority of the object.

19. A non-transitory computer readable medium, comprising instructions which, if executed, cause one or more processors to:
identify first data representing audio from within a vehicle or an image from within the vehicle;
identify, from the first data, second data representing an audio indicator or an image indicator, wherein the audio indicator is human speech from within the vehicle, and wherein the image indicator represents an action of a human within the vehicle;
identify within third data representing an image of a vicinity external to the vehicle, an object corresponding to at least one of the audio indicator or the video indicator; and
generate object data to classify the object.

20. The non-transitory computer readable medium of claim 19 , wherein the identity of the object comprises one or more coordinates defining a boundary of the object.

21. The non-transitory computer readable medium of claim 19 , wherein identifying second data comprises identifying one or more keywords within the audio.

22. The non-transitory computer readable medium of claim 21 , wherein instructions are further configured to cause the one or more processors to send data representing the one or more keywords to the external image analyzer and the external image analyzer is configured to identify the object based on a relationship between the object and the one or more keywords.

23. A means for

Assignees

• Intel Corp

Inventors

• Graefe Ralf
• Kose Cihangir Neslihan

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• B60K35/29

  Instruments characterised by the way in which information is handled, e.g. showing information on plural displays or prioritising information according to driving conditions · CPC title

• B60K35/28

  characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver · CPC title

• B60K35/85

  Arrangements for transferring vehicle- or driver-related data · CPC title