Mobile user interface design testing tool

Patent metadata

Field	Value
Publication number	US-10599399-B2
Application number	US-201816003067-A
Country	US
Kind code	B2
Filing date	Jun 7, 2018
Priority date	Dec 20, 2016
Publication date	Mar 24, 2020
Grant date	Mar 24, 2020

Abstract

A tool for verifying a user interface (UI) design of a mobile application receives a screenshot of the application's UI operating at a target device and retrieves a reference UI design image that corresponds to the received screenshot. The tool generates a plurality of images based on discrepancies between the screenshot and the reference UI design image. The plurality of images include a set of differential images in which each pixel location has a value that is based on a difference between corresponding pixels at the same pixel location of the reference UI design image and of the screenshot. The plurality of imagers also include at least one blended image that is an overlay of the reference UI design image with the screenshot.

First claim

What is claimed is:

1. A method performed by a computing device for verifying a user interface (UI) design of an application, the method comprising:
receiving a screenshot of the application's UI operating at a target device by the computing device;
retrieving, by the computing device, a reference UI design image that corresponds to the received screenshot;
generating, by the computing device, a plurality of images based on one or more discrepancies between the screenshot and the reference UI design image, wherein the plurality of images comprises:
a set of differential images in which each pixel location has a value that is based on a difference between corresponding pixels at a same pixel location of the reference UI design image and of the screenshot; and
at least one blended image that is an overlay of the reference UI design image with the screenshot; and

presenting the generated plurality of images in one presentation.

2. The method of claim 1 , wherein the plurality of images further comprises an image having an identified UI element and a delineation for the identified UI element.

3. The method of claim 2 , further comprising presenting quantitative information regarding a position and a size of the identified UI element.

4. The method of claim 1 , wherein the set of differential images comprises a binary image, wherein each pixel location of the binary image either has a first color pixel to indicate that a difference between the reference UI design image and the screenshot at the pixel location is below or equal to a threshold or a second color pixel to indicate that the difference between the reference UI design image and the screenshot at the pixel location is above the threshold.

5. The method of claim 4 , wherein the set of differential images comprises an intensity variation image, wherein each pixel location of the intensity variation image shows an intensity of the difference between the reference UI design image and the screenshot at the pixel location.

6. The method of claim 1 , wherein the blended image is a transparent overlay of the reference UI design image and the screenshot, wherein the reference UI design image is made more prominent than the screenshot in the transparent overlay.

7. The method of claim 1 , further comprising presenting the reference UI design image and the screenshot in a same view as the generated plurality of images.

8. A method of verifying a user interface (UI) of an application by a computing device, the method comprising:
receiving, by the computing device, a screenshot of the application's UI operating at a target device;
retrieving, by the computing device, a reference UI design image that corresponds to the received screenshot;
identifying, by the computing device, a first UI element in the reference UI design image and a corresponding second UI element in the captured screenshot;
presenting, by the computing device, quantitative information regarding a position, a size, and a color of the identified UI element, wherein the quantitative information of the identified UI element from the screenshot is presented together with the corresponding identified UI element from the reference UI design image; and
presenting, by the computing device, an indication of whether the quantitative information of the first identified UI element differ with the corresponding second identified UI element by greater than a threshold amount.

9. The method of claim 8 , further comprising generating a plurality of images based on one or more discrepancies between the screenshot and the reference UI design image and presenting the plurality of images in one view with the quantitative information.

10. The method of claim 8 , further comprising presenting a blended image based on the reference UI design image and the screenshot, wherein the blended image comprises a delineation of the first identified UI element and a delineation of the second identified UI element.

11. The method of claim 10 , wherein the delineation of the first identified UI element comprises an indication of whether the first identified UI element fulfills a set of constraints.

12. The method of claim 11 , wherein the delineation of the first UI element has a different color when it fulfills its set of constraints than when it fails to fulfill its set of constraints.

13. A computing device comprising:
a display device;
a set of one or more processing units; and
a storage device storing a set of instructions, wherein an execution of the set of instructions by the set of processing units configures the computing device to perform acts comprising:
receiving a screenshot of the application's UI operating at a target device;
retrieving a reference UI design image that corresponds to the received screenshot;
generating a plurality of images based on discrepancies between the screenshot and the reference UI design image, wherein the plurality of images comprise:
a set of differential images in which each pixel location has a value that is based on a difference between corresponding pixels at a same pixel location of the reference UI design image and of the screenshot; and
at least one blended image that is an overlay of the reference UI design image with the screenshot; and

presenting the generated plurality of images in one presentation at the display device.

14. The computing device of claim 13 , wherein the plurality of images further comprises an image having an identified UI element and a delineation for the identified UI element.

15. The computing device of claim 14 , wherein the computing device is further configured to present quantitative information regarding a position, a size and a color of the identified UI element.

16. The computing device of claim 13 , wherein the set of differential images comprises an intensity variation image.

17. The computing device of claim 16 , wherein each pixel location of the intensity variation image shows an intensity of the difference between the reference UI design image and the screenshot at the pixel location.

18. The computing device of claim 13 , wherein the blended image is a transparent overlay of the reference UI design image and the screenshot, wherein the reference UI design image is made more prominent than the screenshot in the transparent overlay.

19. The computing device of claim 13 , wherein the computing device is further configured to present the reference UI design image and the screenshot in a same view as the generated plurality of images.

Assignees

• IBM

Inventors

• Chen Chun-Fu
• Girolami Paolo
• Ligman Joseph W
• Pistoia Marco

Classifications

• G06F9/451

  Execution arrangements for user interfaces · CPC title

• G06F8/20 Primary

  Software design · CPC title

• G06T2200/24

  involving graphical user interfaces [GUIs] · CPC title

• G06T2207/30168

  Image quality inspection · CPC title

• G06T11/60

  Creating or editing images; Combining images with text · CPC title