Systems and methods for training models to predict dense correspondences in images using geodesic distances

Patent metadata

Field	Value
Publication number	US-11954899-B2
Application number	US-202118274371-A
Country	US
Kind code	B2
Filing date	Mar 11, 2021
Priority date	Mar 11, 2021
Publication date	Apr 9, 2024
Grant date	Apr 9, 2024

Abstract

Systems and methods for training models to predict dense correspondences across images such as human images. A model may be trained using synthetic training data created from one or more 3D computer models of a subject. In addition, one or more geodesic distances derived from the surfaces of one or more of the 3D models may be used to generate one or more loss values, which may in turn be used in modifying the model's parameters during training.

First claim

The invention claimed is:

1. A method of training a neural network to predict correspondences in images, the method comprising:
generating, by one or more processors of a processing system and using the neural network, a first feature map based on a first image of a subject, and a second feature map based on a second image of the subject, the first image and the second image being different and having been generated using a three-dimensional model of the subject;
determining, by the one or more processors, a first feature distance between a first point as represented in the first feature map and a second point as represented in the second feature map, the first point and the second point corresponding to the same feature on the three-dimensional model of the subject;
determining, by the one or more processors, a second feature distance between a third point as represented in the first feature map and a fourth point as represented in the first feature map;
determining, by the one or more processors, a first geodesic distance between the third point and the fourth point as represented in a first surface map, the first surface map corresponding to the first image and having been generated using the three-dimensional model of the subject;
determining, by the one or more processors, a third feature distance between the third point as represented in the first feature map and a fifth point as represented in the first feature map;
determining, by the one or more processors, a second geodesic distance between the third point and the fifth point as represented in the first surface map;
determining, by the one or more processors, a first loss value of a set of loss values, the first loss value being based on the first feature distance;
determining, by the one or more processors, a second loss value of the set of loss values, the second loss value being based on the second feature distance, the third feature distance, the first geodesic distance, and the second geodesic distance; and
modifying, by the one or more processors, one or more parameters of the neural network based at least in part on the set of loss values.

2. The method of claim 1 , wherein the first loss value is further based on a set of additional feature distances, each given feature distance of the set of additional feature distances being between a selected point as represented in the first feature map and a corresponding point as represented in the second feature map, the selected point and the corresponding point corresponding to the same feature on the three-dimensional model of the subject.

3. The method of claim 2 , wherein the first point and each selected point collectively represent all pixels in the first image.

4. The method of claim 1 , wherein the second loss value is further based on at least one additional pair of feature distances and at least one additional pair of geodesic distances,
each given additional pair of feature distances of the at least one additional pair of feature distances comprising two feature distances between a set of three selected points as represented in the first feature map, and
each given additional pair of geodesic distances of the at least one additional pair of geodesic distances comprising two geodesic distances between the set of three selected points as represented in the first surface map.

5. The method of claim 1 , further comprising:
determining, by the one or more processors, a set of fourth feature distances between a sixth point as represented in the first feature map and all other points of the first image as represented in the first feature map;
determining, by the one or more processors, a set of third geodesic distances between the sixth point as represented in the first surface map, and all other points of the first image as represented in the first surface map; and
determining, by the one or more processors, a third loss value of the set of loss values, the third loss value being based on the set of fourth feature distances and the set of third geodesic distances.

6. The method of claim 5 , wherein the third loss value is further based on at least one additional set of feature distances and at least one additional set of geodesic distances,
each given additional set of feature distances of the at least one additional set of feature distances being between a selected point as represented in the first feature map and all other points of the first image as represented in the first feature map, and
each given additional set of geodesic distances of the at least one additional set of geodesic distances being between the selected point as represented in the first surface map and all other points of the first image as represented in the first surface map.

7. The method of claim 1 , further comprising:
determining, by the one or more processors, a set of fourth feature distances between a sixth point as represented in the first feature map and all points of the second image as represented in the second feature map;
determining, by the one or more processors, a set of third geodesic distances between a first point as represented in a second surface map and all points of the second image as represented in the second surface map, the second surface map corresponding to the second image and having been generated using the three-dimensional model of the subject, and the first point in the second surface map and the sixth point in the first feature map corresponding to the same feature on the three-dimensional model of the subject; and
determining, by the one or more processors, a third loss value of the set of loss values, the third loss value being based on the set of fourth feature distances and the set of third geodesic distances.

8. The method of claim 7 , wherein the third loss value is further based on at least one additional set of feature distances and at least one additional set of geodesic distances,
each given additional set of feature distances of the at least one additional set of feature distances being between a selected point as represented in the first feature map and all points of the second image as represented in the second feature map, and
each given additional set of geodesic distances of the at least one additional set of geodesic distances being between a corresponding point as represented in a second surface map and all points of the second image as represented in the second surface map, the corresponding point in the second surface map and the selected point in the first feature map corresponding to the same feature on the three-dimensional model of the subject.

9. The method of claim 5 , further comprising:
determining, by the one or more processors, a set of fifth feature distances between a seventh point as represented in the first feature map and all points of the second image as represented in the second feature map;
determining, by the one or more processors, a set of fourth geodesic distances between a first point as represented in a second surface map and all points of the second image as represented in the second surface map, the second surface map corresponding to the second image and having been generated using the three-dimensional model of the subject, and the first point in the second surface map and the seventh point in the first feature map corresponding to the same feature on the three-dimensional model of the subject; and
determining, by the one or more processors, a fourth loss value of the set of loss values, the fourth loss value being based on the set of fifth feature distances and the set of fourth geodesic distances.

10. The method of claim 7 , wherein the first point as represented in the second surface map corresponds to a feature on the three

Assignees

• Google Llc

Inventors

• Zhang Yinda
• Tan Feitong
• Tang Danhang
• Dou Mingsong
• Guo Kaiwen
• Fanello Sean Ryan Francesco
• Bouaziz Sofien
• Keskin Cem
• Du Ruofei
• Pandey Rohit Kumar
• Sun Deqing

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/0895

  Weakly supervised learning, e.g. semi-supervised or self-supervised learning · CPC title

• G06V20/653

  by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces · CPC title

• G06V10/82

  using neural networks · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title