Method and system of generating customized three-dimensional images

Patent metadata

Field	Value
Publication number	US-2024386661-A1
Application number	US-202318319536-A
Country	US
Kind code	A1
Filing date	May 18, 2023
Priority date	May 18, 2023
Publication date	Nov 21, 2024
Grant date	—

Abstract

A data processing system includes: a processor; a memory storing executable instructions which, when executed by the processor, cause the processor, alone or in combination with other processors, to implement a client application with a user interface. The client application is configured to: receive an image depicting an object; generate a fine-tuning input to an image-generating Artificial Intelligence (AI) model to associate image data of the object with an identifier; with the fine-tuning input, fine-tune the AI model; structure a prompt for the AI model using the identifier; and obtain from the AI model a new customized image that depicts the object while preserving an appearance of the object.

First claim

What is claimed is:

1 . A data processing system comprising;
a processor; a memory storing executable instructions which, when executed by the processor, cause the processor, alone or in combination with other processors, to implement a client application with a user interface to: receive a two-dimensional image depicting an object; generate a fine-tuning input to an image-generating Artificial Intelligence (AI) model to associate image data of the object with an identifier; with the fine-tuning input, fine-tune the AI model; structure a prompt for the AI model using the identifier; and obtain from the AI model a new customized image that depicts the object while preserving an appearance of the object.

2 . The data processing system of claim 1 , the AI model being trained to output a three-dimensional (3D) image based on the prompt.

3 . The data processing system of claim 1 , wherein the user interface of the client application is configured to receive textual user input describing the customized image, the application to structure the prompt for the AI model based on the user input.

4 . The data processing system of claim 1 , wherein the client application is further to receive an image depicting a background in which the object is to be visualized in the customized image, wherein the fine-tuning input further comprises association of image data of the background with a second identifier.

5 . The data processing system of claim 4 , wherein the client application is further to structure the prompt for the AI model using the identifiers for both the image data of the object and background such that the customized image depicts the object and background.

6 . The data processing system of claim 4 , wherein the user interface of the client application is configured to receive textual user input describing a relationship between the object and background to be used in generating the customized image.

7 . The data processing system of claim 1 , wherein the client application is to call an instance segmentation service to produce the image depicting the object from a different image depicting the object along with other image content.

8 . The data processing system of claim 1 , wherein the client application is to obtain from the AI model a new customized image that depicts the object while preserving an appearance of the object, the customized image being a two-dimensional image, the client application to then call a service to convert the two-dimensional image into a three-dimensional image.

9 . A non-transitory computer-readable medium comprising instructions for a client application for execution by a processor, alone or in combination with other processors, the client application comprising;
a user interface to receive input images and input text; a fine-tuning tool to generate a fine-tuning input to an image-generating Artificial Intelligence (AI) model to associate image data of an object with an identifier using an image of the object input through the user interface; the fine-tuning tool to implant the fine-tuning input in an output domain of the AI model; and a prompt engine to structure a prompt for the AI model using the identifier and to obtain from the AI model a new customized image that depicts the object based on the prompt while preserving an appearance of the object.

10 . The medium of claim 9 , the AI model being trained to output a three-dimensional (3D) image based on the prompt.

11 . The medium of claim 9 , wherein the user interface of the client application is configured to receive textual user input describing the customized image, the prompt engine to structure the prompt for the AI model based on the user input.

12 . The medium of claim 9 , wherein the client application is further to receive an image depicting a background in which the object is to be visualized in the customized image, wherein the fine-tuning input further comprises association of image data of the background with a second identifier.

13 . The medium of claim 12 , wherein the fine-tuning tool of the client application is further to structure the prompt for the AI model using the identifiers for both the image data of the object and background such that the customized image depicts the object and background.

14 . The medium of claim 12 , wherein the user interface of the client application is configured to receive textual user input describing a relationship between the object and background to be used generating the customized image.

15 . The medium of claim 10 , wherein the user interface includes tools for instructing the prompt engine to generate a new prompt for a new 3D image after viewing a first 3D image, the new prompt revising lighting, background or other element of the first 3D image.

16 . The medium of claim 9 , wherein the client application is to obtain from the AI model a new customized image that depicts the object while preserving an appearance of the object, the customized image being a two-dimensional image, the client application to then call a service to convert the two-dimensional image into a three-dimensional image.

17 . A method of generating a customized three-dimensional image, the method comprising;
receiving an image depicting an object; generating a fine-tuning input to an image-generating Artificial Intelligence (AI) model to associate image data of the object with an identifier; with the fine-tuning input, fine-tuning the AI model; structuring a prompt for the AI model using the identifier; and obtaining from the AI model the customized three-dimensional image that depicts the object while preserving an appearance of the object.

18 . The method of claim 17 , the AI model being trained to output a three-dimensional (3D) image based on the prompt.

19 . The method of claim 17 , further comprising:
receiving textual user input describing the customized image; and structuring the prompt for the AI model based on the user input.

20 . The method of claim 17 , further comprising:
receiving an image depicting a background in which the object is to be visualized in the customized image; and with the fine-tuning input, specifying an association of image data of the background with a second identifier.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Daha Fatima Zohra
• Li Ji
• Zhang Dachuan
• Hu Han
• Peng Houwen
• Bao Jianmin
• Wang Ruizhe
• Chen Dong
• Zhao Hanqing

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T7/174

  involving the use of two or more images · CPC title

• G06F40/30

  Semantic analysis · CPC title

• G06F3/048

  Interaction techniques based on graphical user interfaces [GUI] · CPC title