Training a machine learning algorithm to perform motion estimation of objects in a set of frames

Patent metadata

Field	Value
Publication number	US-12347119-B2
Application number	US-202418619844-A
Country	US
Kind code	B2
Filing date	Mar 28, 2024
Priority date	Feb 27, 2020
Publication date	Jul 1, 2025
Grant date	Jul 1, 2025

Abstract

A method of generating a training dataset suitable for training machine learning algorithms to estimate the motion of objects, and for training a machine learning algorithm to perform motion estimation. A plurality of pairs of synthetic images are generated from obtained objects and backgrounds, each pair have a first frame and a second frame. The first frame includes a selection of objects in first positions and first orientations superimposed on a selected background, and the second frame includes the selection of objects in second positions and second orientations superimposed on the selected background. Also provided are processing systems configured to carry out these methods.

First claim

What is claimed is:

1. A method of generating a training dataset for training a machine learning algorithm to perform motion estimation, the method comprising:
obtaining a plurality of images of objects;
obtaining a plurality of images of backgrounds; and
generating a plurality of pairs of synthetic images, each pair comprising a first frame and a second frame; wherein
the first frame comprises a selection of objects in first positions and first orientations superimposed on a selected background,
the second frame comprises the selection of objects in second positions and second orientations superimposed on the selected background, and
at least some of the second positions and second orientations are different from the first positions and first orientations;

the method further comprising
generating translational ground truth motion vectors, describing the difference between the first positions and the second positions; and
generating non-translational ground truth motion vectors, describing the difference between the first orientations and the second orientations.

2. The method of claim 1 , wherein, for some of the pairs of synthetic images, the images of the objects are used in the respective first and second frames directly as obtained.

3. The method of claim 1 , further comprising, for some of the pairs of synthetic images, modifying the images of the objects before superimposing them on the background.

4. The method of claim 3 , wherein modifying at least one of the images of the objects comprises applying to one object the appearance of another object.

5. The method of claim 1 , further comprising, before generating the plurality of pairs of synthetic images, rejecting some of the obtained plurality of images of objects.

6. The method of claim 5 , wherein the rejecting comprises one or more of:
rejecting images that contain more than a first predetermined number of faces;
rejecting images that contain fewer than a second predetermined number of faces; and
rejecting objects that comprise multiple disjoint parts.

7. The method of claim 1 , wherein the translational ground truth motion vectors include motion vectors meeting at least one of the following conditions:
a horizontal component of the motion vector is at least 20%, optionally at least 50%, or at least 70% of the width of the first frame; and
a vertical component of the motion vector is at least 20%, optionally at least 50%, or at least 70% of the height of the first frame.

8. The method of claim 1 , further comprising dividing the plurality of pairs of images into a training set, for training the machine learning algorithm and a test set, for testing the performance of the machine learning algorithm.

9. The method of claim 1 , wherein each first frame is generated by selecting objects at random and positioning them randomly in the first positions.

10. The method of claim 1 , wherein the differences between the first positions and the second positions are selected randomly.

11. The method of claim 1 , further comprising rendering at least one of:
a translational flow field, containing a flow field derived from the translational ground truth motion vectors; and
a combined flow field, containing a flow field derived from the translational ground truth motion vectors and the non-translational ground truth motion vectors.

12. A method of training a machine learning algorithm to perform motion estimation, the method comprising:
obtaining a training dataset generated by the method as set forth in claim 1 ; and
training the machine learning algorithm to perform motion estimation, wherein the training is performed using the obtained training dataset.

13. A non-transitory computer readable storage medium having stored thereon computer readable code configured to cause the method as set forth in claim 12 to be performed when the code is run.

14. A non-transitory computer readable storage medium having stored thereon computer readable code configured to cause the method as set forth in claim 1 to be performed when the code is run.

15. A processing system comprising a processor configured to generate a training dataset for training a machine learning algorithm to perform motion estimation, and a memory storing code that configures the processor to generate said dataset, wherein the processor is configured to:
obtain a plurality of images of objects;
obtain a plurality of images of backgrounds; and
generate a plurality of pairs of synthetic images, each pair comprising a first frame and a second frame, the first frame comprising a selection of objects in first positions and first orientations superimposed on a selected background, the second frame comprising the selection of objects in second positions and second orientations superimposed on the selected background, wherein at least some of the second positions and second orientations are different from the first positions and first orientations;
generate translational ground truth motion vectors, describing the difference between the first positions and the second positions; and
generate non-translational ground truth motion vectors, describing the difference between the first orientations and the second orientations.

16. The processing system of claim 15 , wherein the processing system is a graphics processing system or an artificial intelligence accelerator system.

17. A method of manufacturing, using an integrated circuit manufacturing system, a processing system as set forth in claim 15 , the method comprising:
processing, using a layout processing system, a computer readable description of the processing system so as to generate a circuit layout description of an integrated circuit embodying the processing system; and
manufacturing, using an integrated circuit generation system, the processing system according to the circuit layout description.

18. A non-transitory computer readable storage medium having stored thereon an integrated circuit definition dataset that, when processed in an integrated circuit manufacturing system, configures the integrated circuit manufacturing system to manufacture a processing system as set forth in claim 15 .

19. A non-transitory computer readable storage medium having stored thereon a computer readable dataset description of a processing system as set forth in claim 15 that, when processed in an integrated circuit manufacturing system, causes the integrated circuit manufacturing system to manufacture an integrated circuit embodying the processing system.

20. An integrated circuit manufacturing system comprising:
a non-transitory computer readable storage medium having stored thereon a computer readable dataset description of a processing system as set forth in claim 15 ;
a layout processing system configured to process the computer readable description so as to generate a circuit layout description of an integrated circuit embodying the processing system; and
an integrated circuit generation system configured to manufacture the processing system according to the circuit layout description.

Assignees

• Imagination Tech Ltd

Inventors

• Ahmadi Aria
• Walton David
• Dikici Cagatay

Classifications

• G06F18/23

  Clustering techniques · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T7/248

  involving reference images or patches · CPC title

• G06T7/207

  for motion estimation over a hierarchy of resolutions (multi-resolution motion estimation or hierarchical motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/53) · CPC title