Head position extrapolation based on a 3D model and image data

Patent metadata

Field	Value
Publication number	US-11386609-B2
Application number	US-202017138207-A
Country	US
Kind code	B2
Filing date	Dec 30, 2020
Priority date	Oct 27, 2020
Publication date	Jul 12, 2022
Grant date	Jul 12, 2022

Abstract

An approach using 3D algorithms to solve 2D head localization problems is disclosed. A system can extrapolate aspects of one part of an object, e.g., extract characteristics of a person's head, using a 2D input image of another part of the object, e.g., a 2D image of the person's face. The system then selects an appropriate 3D model by the use of facial features detected in an image of a person's face. Using the selected 3D model and the 3D rotation angles provided by a face detector, the system rotates the model and then projects the model to a 2D shape. The system then scales and translates, e.g., transforms, the 2D shape to match the 2D face bounding box. Then, using the transformed 2D shape, the system extracts a bounding box for the extracted portion of an object, e.g., the head of the person depicted in the 2D input image.

First claim

The invention claimed is:

1. A method, comprising:
receiving an input image depicting a person, wherein the input image includes a face of the person;
obtaining input data from a face detector defining a face bounding box that provides boundaries of the face of the person, the input data further defining orientation data indicating at least one of a yaw, a pitch, and a roll of the face of the person;
selecting a 3D model from a number of 3D models, the 3D model selection based on a selection of a default 3D model or the 3D model selection being based on the 3D model having at least one feature associated with the person depicted in the input image;
causing a rotation operation to the 3D model using at least one of the yaw, the pitch, or the roll that is obtained from the input data;
causing a projection operation to the 3D model to generate a 2D shape of the 3D model, wherein a 2D model comprises geometric shapes from a head segment of the 3D model, wherein the geometric shapes from a head segment of the 3D model are configured to be used to provide an extension of the at least one feature of the face of the person; and
causing a scaling operation and a translation operation to the 2D shape utilizing the face bounding box to generate a 2D output image that includes a graphical extrapolation of physical features of a head of the person, the physical features being based on the geometric shapes from the head segment of the 3D model, wherein the graphical extrapolation of physical features of the head is positioned outside of the face bounding box and within a head bounding box.

2. The method of claim 1 , wherein the at least one feature associated with the person indicates at least one of a hair style, a clothing item, or an accessory associated with the person depicted in the input image, wherein the selection of the 3D model is based on the 3D model having at least one of the hair style, the clothing item, or the accessory.

3. The method of claim 1 , wherein the projection operation to the 3D model to generate the 2D shape of the 3D model comprises removing a depth axis transform from the vertices and polygons of the 3D model to a two-dimensional format.

4. The method of claim 1 , wherein scaling comprises:
receiving an estimated face length obtained by an analysis of the input image with a face length of a face segment of the 3D model, wherein the analysis is used to determine a scaling factor to match the size of the face length of the 3D model with the estimated face length; and
adjusting the scale of the 3D model in two axes using the scaling factor, wherein the adjustment of the scale of the 3D model generates a set of coordinates for the physical features that model the head of the person.

5. The method of claim 1 , wherein the at least one physical feature includes an object worn by the person, wherein the 3D model selection is based on the 3D model having a virtual object that represents the object worn by the person, wherein the 2D shape includes a graphical extrapolation of the object relative to the head of the person.

6. The method of claim 1 , wherein the input data further defines at least one physical feature associated with the person.

7. A method, comprising:
receiving an input image depicting an object, wherein the input image includes a first portion of the object;
obtaining input data defining a first bounding box that provides graphical boundaries of the first portion of the object, the input data further defining orientation data indicating at least one of a yaw, a pitch, and a roll of the first portion of the object;
selecting a 3D model from a number of 3D models, the 3D model selection based on a default 3D model or the selection being based on the 3D model having at least one physical feature of the object depicted in the input image, wherein the 3D model has at least a first segment that is associated with the first portion of the object and a second segment that is associated with a second portion of the object;
causing a rotation of the 3D model using at least one of the yaw, the pitch, or the roll that is obtained from the input data;
applying a projection to the 3D model to generate a 2D shape of the 3D model, wherein a 2D model comprises geometric shapes from a first segment of the 3D model; and
causing at least one of a scaling operation and a translation operation to the 2D shape utilizing the first bounding box to generate a 2D output image that includes a graphical extrapolation of physical features of the second portion of the object, wherein a position of the graphical extrapolation of physical features is positioned outside of the first bounding box and within a second bounding box.

8. The method of claim 7 , wherein the projection operation to the 3D model to generate the 2D shape of the 3D model comprises removing a depth axis transform from the vertices and polygons of the 3D model to a two-dimensional format.

9. The method of claim 7 , wherein scaling comprises:
receiving an estimated length of the first portion of the object obtained by an analysis of the input image with a first segment length of the 3D model, wherein the analysis is used to determine a scaling factor to match the size of the first segment length of the 3D model with the estimated length of the first portion of the object; and
adjusting the scale of the 3D model in two axes using the scaling factor, wherein the adjustment of the scale of the 3D model generates a set of coordinates for the physical features that model the second portion of the object.

10. The method of claim 7 , wherein the input image further defines at least one physical feature associated with the object.

11. The method of claim 7 , wherein the at least one physical feature of the object depicted in the input image comprises a shape of a component of the object.

12. The method of claim 7 , wherein the at least one physical feature of the object depicted in the input image comprises a curvature of a surface of the object.

13. The method of claim 7 , wherein the at least one physical feature of the object depicted in the input image comprises a texture of a component of the object.

14. A computing device, the computing device comprising:
one or more processing units; and
a computer-readable storage medium having encoded thereon computer-executable instructions to cause the one or more processing units to:
receiving an input image depicting a person, wherein the input image includes a face of the person;
obtaining input data from a face detector defining a face bounding box that provides boundaries of the face of the person, the input data further defining orientation data indicating at least one of a yaw, a pitch, and a roll of the face of the person;
selecting a 3D model from a number of 3D models, the 3D model selection based on a selection of a default 3D model or the 3D model selection being based on the 3D model having at least one feature associated with the person depicted in the input image;
causing a rotation operation to the 3D model using at least one of the yaw, the pitch, or the roll that is obtained from the input data;
causing a projection operation to the 3D model to generate a 2D shape of the 3D model, wherein a 2D model comprises geometric shapes from a head segment of the 3D model, wherein the geometric shapes from a head segment of the 3D model are configured to be used to provide an extension of the at least one feature of the face of the person; and
causing a scaling and a translation to the 2D shape utilizing the face bounding box to generate a 2D output image that includes a graphical extrapolation of physical feature

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Serry Mattan
• Figov Zvi
• Ofer Irit

Classifications

• G06V40/161

  Detection; Localisation; Normalisation · CPC title

• G06V20/647

  by matching two-dimensional images to three-dimensional objects · CPC title

• G06T2210/12

  Bounding box · CPC title

• G06T7/60

  Analysis of geometric attributes · CPC title

• G06T2207/30201

  Face · CPC title