Automatic painting system with drone, user interface and computer vision

Patent metadata

Field	Value
Publication number	US-10023311-B2
Application number	US-201615066708-A
Country	US
Kind code	B2
Filing date	Mar 10, 2016
Priority date	Mar 10, 2016
Publication date	Jul 17, 2018
Grant date	Jul 17, 2018

Abstract

A system and method for painting a structure. The system comprises: a computer vision processing system configured to obtain images of a target structure and generate first instructions signals for real time communication to an unmanned aerial vehicle (UAV) having a paint fluid dispensing system provided thereon; a control device at the UAV responsive to received the first instruction signals for controlling real time navigating of the UAV to a location at the target structure; and the computer vision processing system configured to generate second instruction signals for real time communication to the UAV, wherein the control device at the UAV is configured to automatically actuate the paint fluid dispensing system to apply a paint fluid at the location on the target structure in response, the received second instruction signals configuring the UAV to render a desired visual image on the target structure.

First claim

What is claimed is:

1. A painting system comprising:
a mobile device having a processor configured to receive a digital image of a target structure, said received digital image of said target structure including an overlay of a visual image desired to be physically rendered on said target structure;
an unmanned aerial vehicle (UAV) having a paint fluid dispensing system provided thereon;
a controller device at said UAV configured to receive instruction signals for controlling real time navigating of said UAV to a target location at said target structure;
said processor of said mobile device obtaining real-time images of said UAV relative to the target structure for recognizing a location of said UAV relative to said target structure and for generating said instruction signals
to configure said UAV to navigate said target structure by traversing said structure along a flight path, and
said processor of said mobile device generating further instruction signals
to configure said UAV controller to automatically actuate said paint fluid dispensing system to apply paint fluid at a location on the target structure corresponding to said visual image overlay of said received digital image while navigating said target structure along said flight path, said processor of said mobile device using said obtained real-time images of said UAV relative to the target location for real-time refining said instruction signals for traversing said target structure and refining said further instruction signals for controlling said paint fluid dispensing system to apply said paint, wherein said UAV paints the desired visual image on said target structure.

2. The painting system as claimed in claim 1 , wherein said processor of said mobile device runs a computer vision processing system configured to obtain said real-time images of said UAV relative to the target location and for said generating and refining said instructions signals for real time navigating of said UAV and communicating said instructions signals to said UAV and for said generating and refining said further instruction signals for controlling said paint fluid dispensing system to apply said paint and communicating said further instruction signals to said UAV.

3. The painting system as claimed in claim 2 , wherein said computer vision processing system comprises an image capture device for initially obtaining said digital image of the target structure,
said mobile device comprising: a processor device configured to run an application to edit said obtained digital image of the target structure and receive, from a user, said overlay of the desired visual image onto said obtained digital image via a display of said mobile device.

4. The painting system as claimed in claim 3 , wherein said processor device at said mobile device is further configured to:
run an application to generate flight path information for said UAV to traverse said target structure according to said flight path, said generated flight path information including initial location coordinates and final location coordinates corresponding to a pixel coordinate system of said mobile device display.

5. The painting system as claimed in claim 4 , wherein to generate said flight path information, said controller device at said UAV is configured to:
receive a signal to locate said UAV to a first position, said first position corresponding to a first location of said UAV displayed via a mobile device display;
receive a further signal to command said UAV to fly a physical distance in a single direction for a pre-determined time duration from the first position to a second position, said second position corresponding to a second location of said UAV displayed via the mobile device display;
determine, via the processor device at said mobile device, a pixel distance between the first and second locations of the UAV displayed via said mobile device display corresponding to the physical difference between the first UAV position and second UAV position;
compare the pixel distance to a predetermined pixel distance threshold; and
program said UAV with flying directions for said UAV to traverse said target structure at said pre-determined time duration increments if said pixel distance is less than the predetermined pixel distance threshold.

6. The painting system as claimed in claim 3 , wherein said image capture device is configured to:
capture an image of the target structure having said painted desired visual image thereon, said computer vision processing at said mobile device further configured to:
compare the captured image of the target structure having the painted visual image thereon with the received digital image of the target structure having the corresponding desired visual image overlayed thereon to determine need for additional painting by said UAV on the target structure.

7. The painting system as claimed in claim 6 , wherein responsive to determining a need for additional painting by said UAV, said processor device of said mobile device configured to:
communicate wireless instruction signals for receipt at said UAV to navigate said UAV to a particular location at the target structure, and automatically activate said UAV to traverse said target structure and activate said paint fluid dispensing system to apply a paint color to said particular location at said target structure according to said desired visual image.

8. The painting system as claimed in claim 7 , wherein based on comparison results between the visual image painted by the UAV on said target structure with a corresponding desired visual image overlay of the received digital image at said mobile device display, automatically determine one or more difference regions on the target structure, wherein the UAV is navigated along a path to traverse a difference region to modify said image within said difference region.

9. The painting system as claimed in claim 7 , wherein said mobile device and said UAV is configured to iteratively repeat said navigating to a target position, said traversing said target structure along the flight path, and said activating said paint fluid dispensing system to apply a paint color to said particular location at said target structure until the desired visual image is painted on said target structure.

10. A method for painting a target structure comprising:
receiving, at a processor of a mobile device, a digital image of a target structure, said received digital image of said target structure including an overlay of a visual image desired to be physically rendered on said target structure;
configuring a controller device of an unmanned aerial vehicle (UAV) to receive navigation instruction signals for real time navigating said UAV to a target location at the target structure, said UAV having a paint fluid dispensing system provided thereon;
obtaining, at said processor of said mobile device, real-time images of said UAV relative to the target structure for recognizing a location of said UAV relative to said target structure and for generating said navigation instructions
to configure said UAV to traverse said target structure along a flight path; and
generating, by said processor of said mobile device, further instruction signals
to configure said UAV controller device to automatically activate said paint fluid dispensing system during said traversal along the flight path to apply a paint fluid on the target structure corresponding to said visual image overlay of said received digital image while navigating said target structure along said flight path, said processor of said mobile device using said obtained real-time images of said UAV relative to the target location for real-time refining said navigation instru

Assignees

• IBM

Inventors

• Lai Jui-Hsin
• Ma Yu
• Shi Conglei
• Xia Yinglong

Classifications

• B64U2201/20

  Remote controls · CPC title

• B64D1/18 Primary

  by spraying, e.g. insecticides · CPC title

• B64C39/024

  of the remote controlled vehicle type, i.e. RPV · CPC title

• G05D1/0011

  associated with a remote control arrangement · CPC title

• H04N5/23229

  Electricity · mapped topic