System and method for extracting object information from digital images to evaluate for realism

Patent metadata

Field	Value
Publication number	US-12488566-B2
Application number	US-202318478265-A
Country	US
Kind code	B2
Filing date	Sep 29, 2023
Priority date	Jun 30, 2023
Publication date	Dec 2, 2025
Grant date	Dec 2, 2025

Abstract

Described herein are systems, methods, devices, and other techniques for comprehensive and automated evaluation of digital images generated from artificial intelligence (AI) models in order to promote accurate representations of real-world content. Prompts are received at the system that are then passed to both a search engine and a generative AI model. Synthesized digital images are obtained from the generative AI model. The top-matching image from the search engine is used as a verification of the ground truth of the synthesized digital images. A realism score is generated for each synthesized digital image that characterizes the accuracy of the synthesized digital image with reference to the verification image. The realism score can be used to assist and expedite the image selection process, as well as serve as input to fine-tune the performance of generative models.

First claim

We claim:

1 . A computer-implemented method for extracting object information from digital images, the method comprising:
receiving, at a realism assessment system, a user prompt involving real-world content; submitting, from the realism assessment system and to a web-based search engine, the user prompt; receiving, at the realism assessment system, a reference digital image retrieved by the search engine in response to the user prompt; submitting, from the realism assessment system and to a first generative artificial intelligence (AI) model, the user prompt; receiving, at the realism assessment system, a plurality of synthetic digital images including a first synthetic digital image and a second synthetic digital image, the plurality of synthetic digital images generated by the first generative AI model in response to the user prompt; automatically classifying, via a deep learning-based instance segmentation model of the realism assessment system, a first set of pixels in the reference digital image as corresponding to a first object, a second set of pixels in the first synthetic digital image as corresponding to a second object, and a third set of pixels in the second synthetic digital image as corresponding to a third object; extracting one or more Hu Moments for each of the first object, the second object, and the third object; generating, at the realism assessment system, a first realism score based on a comparison of characteristics of the first object with the second object by comparing the one or more Hu Moments for the first object and the second object; generating, at the realism assessment system, a second realism score based on a comparison of characteristics of the first object with the third object by comparing the one or more Hu Moments for the first object and the third object, the second realism score being greater than the first realism score; determining, at the realism assessment system, the second synthetic digital image has a greater likelihood of accurately representing the real-world content than the first synthetic digital image based on the second realism score being greater than the first realism score; and ranking each synthetic digital image of the plurality of synthetic digital images based on their computed realism score and presenting, at a computing device, the synthetic digital images in order of their ranking.

2 . The method of claim 1 , wherein the computed Hu Moments are used to characterize the shape of a main object in each of the reference digital image, the first image, and the second image, the main object being identified by a large language (LLM)-based classifier.

3 . The method of claim 1 , further comprising:
detecting, via the deep learning-based instance segmentation model, a first set of objects in the reference digital image including at least the first object and a fourth object; generating, via the deep learning-based instance segmentation model, a label for each object in the first set of objects; passing both the user prompt and the labels for the objects in the first set of objects to a large language model (LLM) classifier; and identifying, via the LLM classifier, a first main object from the first set of objects, wherein the first main object corresponds to the first object.

4 . The method of claim 3 , further comprising:
detecting, via the instance segmentation model, a second set of objects in the first synthetic digital image including at least the second object and a fifth object; generating, via the instance segmentation model, a label for each object in the second set of objects; passing both the user prompt and the labels for the objects in the second set of objects to the LLM classifier; and identifying, via the LLM classifier, a second main object from the second set of objects, wherein the second main object corresponds to the second object.

5 . The method of claim 4 , further comprising filtering out characteristics of the fourth object and the fifth object by the LLM classifier before generating the first realism score.

6 . The method of claim 1 , further comprising excluding from presentation at the computing device the synthetic digital images of the plurality of synthetic digital images that are assigned a realism score below a predesignated first threshold score.

7 . The method of claim 1 , further comprising:
receiving, at the realism assessment system, a first image and a second image both retrieved by the search engine in response to the user prompt; converting, via an image captioning AI generator, the first image to a first text description; converting, via the image captioning AI generator, the second image to a second text description; determining the first text description is more similar to the user prompt than the second text description; and selecting the first image as the reference digital image in response to the first text description being more similar to the user prompt than the second text description.

8 . A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to extract object information from digital images by:
receiving, at a realism assessment system, a user prompt involving real-world content; submitting, from the realism assessment system and to a web-based search engine, the user prompt; receiving, at the realism assessment system, a reference digital image retrieved by the search engine in response to the user prompt; submitting, from the realism assessment system and to a first generative AI model, the user prompt; receiving, at the realism assessment system, a plurality of synthetic digital images including a first synthetic digital image and a second synthetic digital image, the plurality of synthetic digital images generated by the first generative AI model in response to the user prompt; automatically classifying, via a deep learning-based instance segmentation model of the realism assessment system, a first set of pixels in the reference digital image as corresponding to a first object, a second set of pixels in the first synthetic digital image as corresponding to a second object, and a third set of pixels in the second synthetic digital image as corresponding to a third object; extracting one or more Hu Moments for each of the first object, the second object, and the third object; generating, at the realism assessment system, a first realism score based on a comparison of characteristics of the first object with the second object by comparing the one or more Hu Moments for the first object and the second object; generating, at the realism assessment system, a second realism score based on a comparison of characteristics of the first object with the third object by comparing the one or more Hu Moments for the first object and the third object, the second realism score being greater than the first realism score; determining, at the realism assessment system, the second synthetic digital image has a greater likelihood of accurately representing the real-world content than the first synthetic digital image based on the second realism score being greater than the first realism score; and ranking each synthetic digital image of the plurality of synthetic digital images based on their computed realism score and presenting, at a computing device, the synthetic digital images in order of their ranking.

9 . The non-transitory computer-readable medium of claim 8 , wherein the computed Hu Moments are used to characterize the shape of a main object in each of the reference digital image, the first image, and the second image, the main object being identified by a l

Assignees

• Accenture Global Solutions Ltd

Inventors

• Cha Sujeong
• Vadlamani Surya Raghavendra
• Kang Sukryool
• Tripathi Anupam Anurag
• Suhail Mohamed
• Smith Jr Peter Royer
• Zhang Bo
• Garrison Daniel
• Singh Jatinder
• Dang Neha Wadhwa

Classifications

• G06T5/20

  using local operators · CPC title

• G06V2201/07

  Target detection · CPC title

• G06V10/761

  Proximity, similarity or dissimilarity measures · CPC title

• G06V20/70 Primary

  Labelling scene content, e.g. deriving syntactic or semantic representations · CPC title

• G06T7/50

  Depth or shape recovery · CPC title