Method, apparatus, and system for generating synthetic image data for machine learning

Patent metadata

Field	Value
Publication number	US-10755115-B2
Application number	US-201715858772-A
Country	US
Kind code	B2
Filing date	Dec 29, 2017
Priority date	Dec 29, 2017
Publication date	Aug 25, 2020
Grant date	Aug 25, 2020

Abstract

An approach is provided for generating synthetic image data for machine learning. The approach, for instance, involves determining, by a processor, a set of parameters for indicating an action by one or more objects. The action is a dynamic movement of the one or more objects through a geographic space over a period of time. The approach also involves processing the set of parameters to generate synthetic image data. The synthetic image data includes a computer-generated image sequence of the one or more objects performing the action in the geographic space over the period of time. The approach further involves automatically labeling the synthetic image data with at least one label representing the action, the set of parameters, or a combination thereof. The approach further involves providing the labeled synthetic image data for training or evaluating a machine learning model to detect the action.

First claim

What is claimed is:

1. A computer-implemented method comprising:
determining, by a processor, a set of parameters for indicating at least one action by one or more objects, wherein the at least one action is a dynamic movement of the one or more objects through a geographic space over a period of time, and the set of parameters is user defined, wherein the set of parameters includes an action parameter describing a type of the at least one action, and wherein the type of the at least one action includes failing to drive at a safe distance, driving above the speed limit, failing to yield, running a red light, driving in the wrong direction, driving while impaired, an imminent collision, an accident, a pedestrian or animal about to cross, unsafe or dangerous overtaking, dangerous driving behavior, or a combination thereof;
processing the set of parameters to generate synthetic image data, wherein the synthetic image data includes a computer-generated image sequence of the one or more objects performing the at least one action in the geographic space over the period of time;
automatically labeling the synthetic image data with at least one label representing the at least one action, the set of parameters, or a combination thereof; and
providing the labeled synthetic image data for training or evaluating a machine learning model to detect the at least one action.

2. The method of claim 1 , wherein the at least one action is related to an automotive scenario, and the one or more objects include a vehicle, a pedestrian, a cyclist, an animal, a road, road debris, a road object, or a combination thereof.

3. A computer-implemented method comprising:
determining, by a processor, a set of parameters for indicating at least one action by one or more objects, wherein the at least one action is a dynamic movement of the one or more objects through a geographic space over a period of time, and the set of parameters is user defined;
processing the set of parameters to generate synthetic image data, wherein the synthetic image data includes a computer-generated image sequence of the one or more objects performing the at least one action in the geographic space over the period of time;
automatically labeling the synthetic image data with at least one label representing the at least one action, the set of parameters, or a combination thereof;
providing the labeled synthetic image data for training or evaluating a machine learning model to detect the at least one action;
determining that the at least one action will result in a collision, a near miss, a potential collision, a hazardous driving behavior, or a combination thereof between the one or more objects within a time threshold; and
labeling the synthetic image data to further indicate the collision, the near miss, the potential collision, the hazardous driving behavior, or a combination thereof.

4. The method of claim 1 , wherein the labeling of the synthetic image data comprises labeling a span of frames of the computer-generated image sequence as a positive case or a negative case of the at least one action, or labeling the span of frames with a variable parameter value associated with the at least one action.

5. The method of claim 1 , wherein the labeled synthetic image data is provided as a download, as a data stream, via a physical media, or a combination thereof continuously or by a batch process.

6. The method of claim 1 , further comprising:
determining one or more randomizable rendering variables associated with the computer-generated image sequence; and
randomizing the one or more randomizable rendering variables to generate the synthetic image data.

7. The method of claim 1 , wherein the set of parameters includes an environmental parameter describing the geographic space, an object parameter describing a type of the one or more objects, or a combination thereof.

8. The method of claim 1 , further comprising:
querying a geographic database for map data associated with the geographic space; and
rendering the geographic space in the computer-generated image sequence based on the map data.

9. The method of claim 1 , further comprising:
selecting the geographic space based on an area of interest, an origin-destination pair, a navigation route, a road geometry, an accident rate, or a combination thereof.

10. The method of claim 1 , wherein the set of parameters includes a dataset parameter for describing a technical property of the synthetic image data, and wherein the technical property includes a perspective, a frame size, a frame rate, a resolution, an image sequence length, a format or codec, a delivery option, or a combination thereof of the labeled synthetic image data.

11. The method of claim 1 , further comprising:
varying a level of detail or an abstractness of a rendering of the one or more objects, the geographic space, or other objects in the computer-generated image sequence based on a target generalizability of the machine learning model.

12. The method of claim 1 , further comprising:
determining one or more background objects to render in the computer-generated image sequence based on the geographic space.

13. An apparatus comprising:
at least one processor; and
at least one memory including computer program code for one or more programs,
the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to perform at least the following,
process a set of parameters indicating at least one action by one or more objects to generate synthetic image data, wherein the synthetic image data includes a computer-generated image sequence of the one or more objects performing the at least one action in a geographic space over a period of time, and the set of parameters is user defined, wherein the set of parameters includes an action parameter describing a type of the at least one action, and wherein the type of the at least one action includes failing to drive at a safe distance, driving above the speed limit, failing to yield, running a red light, driving in the wrong direction, driving while impaired, an imminent collision, an accident, a pedestrian or animal about to cross, unsafe or dangerous overtaking, dangerous driving behavior, or a combination thereof;
automatically label the synthetic image data with at least one label representing the at least one action, the set of parameters, or a combination thereof; and
provide the labeled synthetic image data for training or evaluating a machine learning model to detect the at least one action.

14. The apparatus of claim 13 , wherein the at least one action is related to an automotive scenario, and the one or more objects include a vehicle, a pedestrian, a cyclist, an animal, a road, road debris, a road object, or a combination thereof.

15. The apparatus of claim 13 , wherein the apparatus is further caused to:
determine that the at least one action will result in a collision between the one or more objects within a time threshold; and
label the synthetic image data to further indicate the collision.

16. The apparatus of claim 13 , wherein the labeling of the synthetic image data causes the apparatus to label each frame of the computer-generated generated image sequence as a positive case or a negative case of the at least one action, or to label said each frame with a variable parameter value associated with the at least one action.

17. A non-transitory computer-readable storage medium, carrying one or more sequences of one or more instructions which, when executed by one or more processors, cau

Assignees

• Here Global Bv

Inventors

• Avidan Avi
• Luk-Zilberman Evgeny

Classifications

• G08G1/166 Primary

  for active traffic, e.g. moving vehicles, pedestrians, bikes · CPC title

• G06V10/772

  Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V20/58

  Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads · CPC title

• G06F18/28

  Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries · CPC title