Systems and methods for refined object estimation from image data

Patent metadata

Field	Value
Publication number	US-11836221-B2
Application number	US-202117200643-A
Country	US
Kind code	B2
Filing date	Mar 12, 2021
Priority date	Mar 12, 2021
Publication date	Dec 5, 2023
Grant date	Dec 5, 2023

Abstract

Systems and methods are directed to a method for estimation of an object state from image data. The method can include obtaining two-dimensional image data depicting an object. The method can include processing, with an estimation portion of a machine-learned object state estimation model, the two-dimensional image data to obtain an initial estimated state of the object. The method can include, for each of one or more refinement iterations, obtaining a previous loss value associated with a previous estimated state for the object, processing the previous loss value to obtain a current estimated state of the object, and evaluating a loss function to determine a loss value associated with the current estimated state of the object. The method can include providing a final estimated state for the object.

First claim

What is claimed is:

1. A computer-implemented method for estimation of an object state from image data, the method comprising:
obtaining, by a computing system comprising one or more computing devices, two-dimensional image data depicting an object;
processing, by the computing system with an estimation portion of a machine-learned object state estimation model, the two-dimensional image data to obtain an initial estimated state of the object;
performing, by the computing system, a plurality of refinement iterations to generate a final estimated state for the object, wherein performing a refinement iteration comprises:
obtaining, by the computing system, a previous loss value associated with a previous estimated state for the object, wherein the previous estimated state for the object is generated using a previous refinement submodel of a plurality of refinement submodels of a refinement portion of the machine-learned object state estimation model;
processing, by the computing system with a current refinement submodel of the plurality of refinement submodels, a set of inputs to obtain a current estimated state of the object, wherein the set of inputs comprises:
the initial estimated state of the object;
the previous loss value; and
internal memory data from the previous refinement submodel, wherein the internal memory data is descriptive of an internal memory state of the previous refinement submodel after being used to generate the previous estimated state for the object; and

evaluating, by the computing system, a loss function to determine a loss value associated with the current estimated state of the object; and

providing, by the computing system based at least in part on a final refinement iteration of the one or more refinement iterations, a final estimated state for the object, wherein the final estimated state for the object comprises data descriptive of a three-dimensional representation of the object.

2. The computer-implemented method of claim 1 , wherein the method further comprises:
evaluating, by the computing system, a loss function that evaluates a difference between the final estimated state for the object and ground truth data associated with the two-dimensional image data; and
adjusting, by the computing system based at least in part on the loss function, one or more parameters of at least one of the estimation portion or the refinement portion of the machine-learned object state estimation model.

3. The computer-implemented method of claim 2 , wherein the ground truth data comprises one or more annotations of the two-dimensional image data.

4. The computer-implemented method of claim 2 , wherein the ground truth data is based at least in part on the initial estimated state of the object.

5. The computer-implemented method of claim 1 , wherein:
the plurality of refinement submodels are respectively associated with the plurality of refinement iterations.

6. The computer-implemented method of claim 1 , wherein the set of inputs further comprises context data.

7. The computer-implemented method of claim 6 , wherein, prior to the plurality of refinement iterations, the method comprises:
processing, by the computing system with the estimation portion of the machine-learned object state estimation model, the two-dimensional image data to obtain the context data.

8. The computer-implemented method of claim 1 , wherein each of the plurality of refinement submodels comprises a recurrent neural network.

9. The computer-implemented method of claim 1 , wherein:
the object comprises one or more object segments; and
the initial estimated state of the object comprises one or more respective initial estimated states of the one or more object segments;
the machine-learned object state estimation model comprises one or more segment refinement portions respectively associated with the one or more object segments.

10. The computer-implemented method of claim 9 , wherein:
for each of the one or more object segments:
for each of plurality of refinement iterations:
obtaining the previous loss value associated with the previous estimated state for the object comprises obtaining, by the computing system, a previous loss value associated with a previous estimated state of a respective object segment of the one or more object segments, wherein the previous estimated state for the object segment is generated using a previous refinement submodel of the plurality of refinement submodels of the refinement portion of the machine-learned object state estimation model;
processing the previous loss value to obtain the current estimated state of the object comprises processing, by the computing system with a respective segment refinement portion of one or more segment refinement portions, the previous loss value to obtain a current estimated state of the respective object segment; and
evaluating the loss function to determine the loss value comprises evaluating, by the computing system, a respective loss term of one or more loss terms of the loss function to determine a loss value associated with the current estimated state of the respective object segment, wherein the one or more loss terms are respectively associated with the one or more object segments.

11. The computer-implemented method of claim 10 , wherein prior to providing the final estimated state for the object, the method comprises fusing, by the computing system, one or more respective final estimated states for the one or more object segments to obtain the final estimated state for the object.

12. The computer-implemented method of claim 1 , wherein the final estimated state for the object comprises three-dimensional pose data.

13. The computer-implemented method of claim 1 , wherein the final estimated state for the object comprises a three-dimensional mesh comprising one or more pose variables and one or more state variables.

14. The computer-implemented method of claim 1 , wherein the estimation portion of the machine-learned object state estimation model comprises one or more convolutional neural networks.

15. The computer-implemented method of claim 1 , wherein the object comprises a human body.

16. A computing system for estimation of an object state from image data, comprising:
one or more processors;
one or more tangible, non-transitory computer readable media storing computer-readable instructions that when executed by the one or more processors cause the one or more processors to perform operations, the operations comprising:
processing, with an estimation portion of a machine-learned object state estimation model, two-dimensional image data depicting an object to obtain an initial estimated state of the object;
performing a plurality of refinement iterations to generate a final estimated state for the object, wherein performing a refinement iteration comprises:
obtaining a previous loss value associated with a previous estimated state for the object, wherein the previous estimated state for the object is generated using a previous refinement submodel of a plurality of refinement submodels of a refinement portion of the machine-learned object state estimation model;
processing, with a current refinement submodel of the plurality of refinement submodels, a set of inputs to obtain a current estimated state of the object, wherein the set of inputs comprises:
the initial estimated state of the object;
the previous loss value; and
internal memory data from the previous refinement submodel, wherein the internal memory data is descriptive of an internal memory state of the previous refinement submodel after be

Assignees

• Google Llc

Inventors

• Sminchisescu Cristian
• Zanfir Andrei
• Bazavan Eduard Gabriel
• Zanfir Mihai
• Freeman William Tafel
• Sukthankar Rahul

Classifications

• G06N3/0985

  Hyperparameter optimisation; Meta-learning; Learning-to-learn · CPC title

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/0895

  Weakly supervised learning, e.g. semi-supervised or self-supervised learning · CPC title

• G06N3/09

  Supervised learning · CPC title