Artificial intelligence apparatus and method for synthesizing images

Patent metadata

Field	Value
Publication number	US-11276226-B2
Application number	US-202016807442-A
Country	US
Kind code	B2
Filing date	Mar 3, 2020
Priority date	Dec 19, 2019
Publication date	Mar 15, 2022
Grant date	Mar 15, 2022

Abstract

Disclosed herein is an artificial intelligence apparatus including a memory configured to store image data, and a processor configured to acquire a human object image and a inanimate object image from the image data, generate a three-dimensional human model that matches the human object image, extract coordinate information of the image data, acquire a street view data matching the coordinate information, extract a reference object image, in which a inanimate object of the inanimate object image is photographed, from the street-view data, and acquire placement information of the human object image and the inanimate object image and place the three-dimensional human model on a street view image of the street view data using the placement information and the reference object image.

First claim

What is claimed is:

1. An artificial intelligence apparatus comprising:
a memory configured to store image data; and
a processor configured to acquire a human object image and an inanimate object image from the image data,
generate a three-dimensional human model that matches the human object image,
extract coordinate information of the image data, acquire a street view data matching the coordinate information,
extract a reference object image, in which an inanimate object of the inanimate object image is photographed, from the street-view data,
acquire placement information of the human object image and the inanimate object image and place the three-dimensional human model on a street view image of the street view data using the placement information and the reference object image,
wherein for acquiring the placement information the processor is configured to: acquire placement proportion information based on an area of the human object image and an area of the inanimate object image, and acquire placement distance information of the human object image and the inanimate object image, and
wherein placing of the three-dimensional human model on the street view image uses the placement information including the placement proportion information and the placement distance information.

2. The artificial intelligence apparatus of claim 1 , wherein the processor is configured to acquire human information including at least one of age, gender, body size, and posture information of a human from the human object image, generate a three-dimensional human mesh model based on the human information and generate the three-dimensional human model based on the 3D human mesh model.

3. The artificial intelligence apparatus of claim 2 , wherein the memory includes a three-dimensional mesh model database configured to store at least one three-dimensional human mesh model classified by age, gender and body size, and
wherein the processor is configured to acquire a three-dimensional human mesh model matching the human information from the three-dimensional mesh model database and generate the three-dimensional human model based on the three-dimensional human mesh model.

4. The artificial intelligence apparatus of claim 2 , wherein the processor is configured to acquire texture mapping information to be mapped to the three-dimensional human mesh model based on the human object image and generate the three-dimensional human model by mapping a texture to the three-dimensional human mesh model based on the texture mapping information.

5. The artificial intelligence apparatus of claim 2 , wherein the processor is configured to modify the three-dimensional human model to have a posture of the human object image by using the posture information included in the human information.

6. The artificial intelligence apparatus of claim 1 , wherein the processor is configured to acquire, from the street view data, viewpoint information from which the reference object image is extracted.

7. The artificial intelligence apparatus of claim 6 , wherein the processor is configured to:
compare the inanimate object image with the reference object image, and
change the viewpoint information when a matching rate between the inanimate object image and the reference object image is less than or equal to a preset matching rate.

8. The artificial intelligence apparatus of claim 6 , wherein the processor is configured to:
determine whether the inanimate object image is a pre-registered landmark object image, and set the viewpoint information by using location information of a landmark when the inanimate object image is the pre-registered landmark object image.

9. The artificial intelligence apparatus of claim 1 , wherein the processor is configured to:
acquire rotation information for changing the inanimate object image and the reference object image into an image photographed in the same direction; and
rotate the three-dimensional human model based on the rotation information to place the three-dimensional human model on the street view image.

10. An image synthesis method comprising:
storing image data;
acquiring a human object image and an inanimate object image from the image data;
generating a three-dimensional human model that matches the human object image;
extracting coordinate information of the image data;
acquiring a street view data matching the coordinate information;
extracting a reference object image, in which an inanimate object of the inanimate object image is photographed, from the street-view data;
acquiring placement information of the human object image and the inanimate object image; and
placing the three-dimensional human model on a street view image of the street view data using the placement information and the reference object image,
wherein the acquiring of the placement information includes:
acquiring placement proportion information based on an area of the human object image and an area of the inanimate object image; and
acquiring placement distance information of the human object image and the inanimate object image, and
wherein the placing of the three-dimensional human model on the street view image uses the placement information including the placement proportion information and the placement distance information.

11. The image synthesis method of claim 10 , wherein the generating of the three-dimensional human model includes:
acquiring human information including at least one of age, gender, body size, and posture information of a human from the human object image;
generating a three-dimensional human mesh model based on the human information; and
generating the three-dimensional human model based on the 3D human mesh model.

12. The image synthesis method of claim 11 ,
wherein the generating of the three-dimensional human mesh model includes:
acquiring a three-dimensional human mesh model matching the human information from the three-dimensional mesh model database; and
generating the three-dimensional human model based on the three-dimensional human mesh model, and
wherein the three-dimensional mesh model database is configured to store at least one three-dimensional human mesh model classified by age, gender and body size.

13. The image synthesis method of claim 11 , wherein the generating of the three-dimensional human model based on the three-dimensional human mesh model includes:
acquiring texture mapping information to be mapped to the three-dimensional human mesh model based on the human object image; and
generating the three-dimensional human model by mapping a texture to the three-dimensional human mesh model based on the texture mapping information.

14. The image synthesis method of claim 11 , further comprising:
modifying the three-dimensional human model to have a posture of the human object image by using the posture information included in the human information.

15. The image synthesis method of claim 10 , wherein the extracting of the reference object image includes acquiring, from the street view data, viewpoint information from which the reference object image is extracted.

16. The image synthesis method of claim 15 , wherein the extracting of the reference object image includes:
comparing the inanimate object image with the reference object image; and
changing the viewpoint information when a matching rate between the inanimate object image and the reference object image is less than or equal to a preset matching rate.

17. The image synthesis method of claim 15 , wherein the extracting of the reference object image includ

Assignees

• Lg Electronics Inc

Inventors

• Lim Sekeun

Classifications

• G06T15/205 Primary

  Image-based rendering · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06V20/20

  in augmented reality scenes · CPC title

• G06V10/242

  by image rotation, e.g. by 90 degrees · CPC title

• G06V20/647

  by matching two-dimensional images to three-dimensional objects · CPC title