Method, apparatus, and system for generating synthetic image data for machine learning

Patent metadata

Field	Value
Publication number	US-11475677-B2
Application number	US-202016927625-A
Country	US
Kind code	B2
Filing date	Jul 13, 2020
Priority date	Dec 29, 2017
Publication date	Oct 18, 2022
Grant date	Oct 18, 2022

Abstract

An approach is provided for generating synthetic image data for machine learning. The approach, for instance, involves determining, by a processor, a set of parameters for indicating an action by one or more objects. The action is a dynamic movement of the one or more objects through a geographic space over a period of time. The approach also involves processing the set of parameters to generate synthetic image data. The synthetic image data includes a computer-generated image sequence of the one or more objects performing the action in the geographic space over the period of time. The approach further involves automatically labeling the synthetic image data with at least one label representing the action, the set of parameters, or a combination thereof. The approach further involves providing the labeled synthetic image data for training or evaluating a machine learning model to detect the action.

First claim

What is claimed is:

1. A computer-implemented method comprising:
processing, using a machine learning model, image data depicting a scene associated with a vehicle, wherein the machine learning model is trained using synthetic image data to detect at least one action that can result in a collision, a near miss, a potential collision, a hazardous driving behavior, or a combination thereof, and wherein the synthetic image data is generated based on a set of parameters indicating the at least one action, and the synthetic image data includes a computer-generated image sequence of the one or more objects performing the at least one action;
determining a prediction that the at least one action is occurring based on the processing; and
providing data for controlling an autonomous operation of the vehicle, for presenting an alert message, or a combination thereof based on the prediction.

2. The method of claim 1 , wherein the image data is captured by a camera sensor of the vehicle, a street camera, or a combination thereof.

3. The method of claim 1 , wherein the data for controlling the autonomous operation of the vehicle is used to modify the vehicle's operation to avoid the at least one action, the collision, the near miss, the potential collision, the hazardous driving behavior, or a combination thereof.

4. The method of claim 1 , wherein the controlling of the autonomous operation includes changing a direction of the vehicle, slowing down the vehicle, honking, or a combination thereof.

5. The method of claim 1 , wherein the at least one action is detected to involve a pedestrian, and wherein the alert message is provided to alert the pedestrian.

6. The method of claim 1 , wherein the synthetic image data is labeled with at least one label representing the at least one action, the set of parameters, or a combination thereof.

7. The method of claim 1 , wherein the set of parameters includes an action parameter describing a type of the at least one action.

8. The method of claim 6 , wherein the type of the at least one action includes failing to drive at a safe distance, driving above the speed limit, failing to yield, running a red light, driving in the wrong direction, driving while impaired, an imminent collision, an accident, a pedestrian or animal about to cross, an unsafe or dangerous overtaking, or a combination thereof.

9. A computer-implemented method comprising:
determining, by a processor, a set of parameters for indicating at least one action by one or more objects, wherein the at least one action can result in a collision, a near miss, a potential collision, a hazardous driving behavior, or a combination thereof between the one or more objects within a time threshold;
processing the set of parameters to generate synthetic image data, wherein the synthetic image data includes a computer-generated image sequence of the one or more objects performing the at least one action in the geographic space over the period of time;
automatically labeling the synthetic image data with at least one label representing the at least one action, the set of parameters, or a combination thereof; and
providing the labeled synthetic image data for training or evaluating a machine learning model to detect the at least one action.

10. The method of claim 9 , wherein the at least one action is a dynamic movement of the one or more objects through a geographic space over a period of time.

11. The method of claim 9 , wherein the set of parameters includes an action parameter describing a type of the at least one action.

12. The method of claim 11 , wherein the type of the at least one action includes failing to drive at a safe distance, driving above the speed limit, failing to yield, running a red light, driving in the wrong direction, driving while impaired, an imminent collision, an accident, a pedestrian or animal about to cross, an unsafe or dangerous overtaking, or a combination thereof.

13. The method of claim 9 , wherein the labeling of the synthetic image data comprises labeling a span of frames of the computer-generated image sequence as a positive case or a negative case of the at least one action.

14. The method of claim 9 , wherein the set of parameters includes a perspective, a frame size, a frame rate, a resolution, an image sequence length, a format or codec, a delivery option, or a combination thereof for generating the synthetic image data.

15. An apparatus comprising:
at least one processor; and
at least one memory including computer program code for one or more programs, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to perform at least the following,
process, using a machine learning model, image data depicting a scene associated with a vehicle, wherein the machine learning model is trained using synthetic image data to detect at least one action that can result in a collision, a near miss, a potential collision, a hazardous driving behavior, or a combination thereof, and wherein the synthetic image data is generated based on a set of parameters indicating the at least one action, and the synthetic image data includes a computer-generated image sequence of the one or more objects performing the at least one action;
determine a prediction that the at least one action is occurring based on the processing; and
provide data for controlling an autonomous operation of the vehicle, for presenting an alert message, or a combination thereof based on the prediction.

16. The apparatus of claim 15 , wherein the image data is captured by a camera sensor of the vehicle, a street camera, or a combination thereof.

17. The apparatus of claim 15 , wherein the data for controlling the autonomous operation of the vehicle is used to modify the vehicle's operation to avoid the at least one action, the collision, the near miss, the potential collision, the hazardous driving behavior, or a combination thereof.

18. The apparatus of claim 15 , wherein the controlling of the autonomous operation includes changing a direction of the vehicle, slowing down the vehicle, honking, or a combination thereof.

19. The apparatus of claim 15 , wherein the at least one action is detected to involve a pedestrian, and wherein the alert message is provided to alert the pedestrian.

20. The apparatus of claim 15 , wherein the synthetic image data is labeled with at least one label representing the at least one action, the set of parameters, or a combination thereof.

Assignees

• Here Global Bv

Inventors

• Avidan Avi
• Luk-Zilberman Evgeny

Classifications

• G08G1/166 Primary

  for active traffic, e.g. moving vehicles, pedestrians, bikes · CPC title

• G06V10/772

  Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V20/58 Primary

  Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads · CPC title

• G06F18/2155

  characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling · CPC title