Systems and methods for image labeling using artificial intelligence

Patent metadata

Field	Value
Publication number	US-11080838-B1
Application number	US-201916535322-A
Country	US
Kind code	B1
Filing date	Aug 8, 2019
Priority date	Aug 13, 2018
Publication date	Aug 3, 2021
Grant date	Aug 3, 2021

Abstract

An image analysis (“IA”) computer system for analyzing images of hail damage includes at least one processor in communication with at least one memory device. The at least one processor is programmed to: (i) store a damage prediction model associated with a rooftop, wherein the damage prediction model utilizes an artificial intelligence algorithm; (ii) display, to a user, an image of a rooftop; (iii) receive, from the user, a request to analyze damage to the rooftop; (iv) apply, by the at least one processor, the damage prediction model to the image, the damage prediction model outputting a plurality of damage prediction locations of the rooftop in relation to the image; and/or (v) display, by the at least one processor, an overlay box at each of the plurality of damage prediction locations, the overlay box being a virtual object overlaid onto the image for labeling the damage prediction locations.

First claim

We claim:

1. An image analysis (“IA”) computer system for analyzing images of hail damage using artificial intelligence, the IA computer system including at least one processor in communication with at least one memory device, the at least one processor programmed to:
store, in the at least one memory device, a damage prediction model associated with a rooftop, wherein the damage prediction model utilizes an artificial intelligence algorithm, including machine learning techniques for analyzing rooftops and identifying damage thereto;
display, to a user on a user interface of a user computing device, an image of a rooftop;
receive, from the user via the user interface of the user computing device, a request to analyze damage to the rooftop displayed in the image;
in response to the request, apply, by the at least one processor, the image to the damage prediction model, the damage prediction model outputting a plurality of damage prediction locations of the rooftop in relation to the image;
display, by the at least one processor, on the user interface of the user computing device, an overlay box at each of the plurality of damage prediction locations, the overlay box being a virtual object overlaid onto the image displayed on the user interface;
receive, from the user computing device, a user selection of a location on the image displayed on the user interface;
when the user-selected location corresponds to one overlay box at one of the plurality of damage prediction locations:
display additional information associated with the corresponding damage prediction location; and
receive, from the user computing device, a user interaction verifying whether the corresponding damage prediction location correlates to an actual damage location; and

when the user-selected location does not correspond to any overlay box:
add, in response to the user selection of the location, a new overlay box including a new virtual overlay at the user-selected location; and
determine whether the new overlay box correlates to an actual damage location.

2. The IA computer system of claim 1 , wherein the at least one processor is further programmed to, when the user-selected location corresponds to one overlay box at one of the plurality of damage prediction locations, change a color of the corresponding overlay box from a first color to a second color in response to the user interaction.

3. The IA computer system of claim 1 , wherein the at least one processor is further programmed to, when the location corresponds to one overlay box at one of the plurality of damage prediction locations, display a magnified view of the damage prediction location associated with the corresponding overlay box.

4. The IA computer system of claim 1 , wherein actual damage locations represent hail damage to the rooftop displayed in the image.

5. The IA computer system of claim 1 , wherein the at least one processor is further programmed to:
automatically determine, by the at least one processor, that the new overlay box corresponds to an actual damage location associated with hail damage; and
store, by the at least one processor, image data associated with the determination in a database of the at least one memory device.

6. The IA computer system of claim 5 , wherein the at least one processor is further programmed to automatically update the damage prediction model using the stored determination.

7. The IA computer system of claim 5 , wherein the image data includes location coordinates for the actual damage location defined by the new virtual object.

8. The IA computer system of claim 1 , wherein the at least one processor is further programmed to receive user feedback for the image, wherein the user feedback includes comments as to a quality of the image and a damage classification for each of the plurality of damage prediction locations.

9. The IA computer system of claim 1 , wherein the at least one processor is further programmed to:
assign, by the at least one processor, a first color to the overlay boxes associated with the plurality of damage prediction locations;
when the user-selected location corresponds to one overlay box at one of the plurality of damage prediction locations, update, in response to receiving the user selection, the corresponding overlay box from the first color to a second color; and
display, by the at least one processor, the updated overlay box at the corresponding damage prediction location.

10. The IA computer system of claim 1 , wherein the at least one processor is further programmed to:
collect performance data associated with the user, wherein the performance data is associated with an ability to correctly identify actual damage locations;
determine, based upon the collected performance data, a performance level of the user; and
display, to the user, a subsequent rooftop image after the user completes verification of the plurality of damage prediction locations, wherein an image quality of the subsequent rooftop image corresponds to the determined performance level.

11. The IA computer system of claim 1 , wherein the at least one processor is programmed to apply the image to the damage prediction model by:
storing, in the at least one memory device, historical image data associated with rooftop damage; and
applying a plurality of damage parameters to the image, wherein the plurality of damage parameters are determined, by the at least one processor, based upon the historical image data.

12. A computer-implemented method for analyzing images of hail damage using artificial intelligence, the method implemented using an image analysis (“IA”) computer system, the IA computer system including at least one processor in communication with at least one memory device, the method comprising:
storing, in the at least one memory device, a damage prediction model associated with a rooftop, wherein the damage prediction model utilizes an artificial intelligence algorithm, including machine learning techniques for analyzing rooftops and identifying damage thereto;
displaying, to a user on a user interface of a user computing device, an image of a rooftop;
receiving, from the user via the user interface of the user computing device, a request to analyze damage to the rooftop displayed in the image;
in response to the request, applying, by the at least one processor, the image to the damage prediction model, the damage prediction model outputting a plurality of damage prediction locations of the rooftop in relation to the image;
displaying, by the at least one processor, on the user interface of the user computing device, an overlay box at each of the plurality of damage prediction locations, the overlay box being a virtual object overlaid onto the image displayed on the user interface;
receiving, from the user computing device, a user selection of a location on the image displayed on the user interface;
when the user-selected location corresponds to one overlay box at one of the plurality of damage prediction locations:
displaying additional information associated with the corresponding damage prediction location; and
receiving, from the user computing device, a user interaction verifying whether the corresponding damage prediction location correlates to an actual damage location; and

when the user-selected location does not correspond to any overlay box:
adding, in response to the user selection of the location, a new overlay box including a new virtual overlay at the user-selected location; and

determining whether the new overlay box correlates to an actual damage location.

13. The IA computer-implemented method of claim 12 , further comp

Assignees

• State Farm Mutual Automobile Insurance Co

Inventors

• Li Yuntao
• Isaksson Charlie
• Bokshi-Drotar Marigona
• Wan Jing

Classifications

• G06T7/0004 Primary

  Industrial image inspection · CPC title

• G06V20/17

  taken from planes or by drones · CPC title

• G06V10/7784

  based on feedback from supervisors · CPC title

• G06T7/0002 Primary

  Inspection of images, e.g. flaw detection · CPC title

• G06V20/176

  Urban or other man-made structures · CPC title