Computerized recognition of tabular data from an image

Patent metadata

Field	Value
Publication number	US-12536826-B2
Application number	US-202218082567-A
Country	US
Kind code	B2
Filing date	Dec 15, 2022
Priority date	Jun 23, 2022
Publication date	Jan 27, 2026
Grant date	Jan 27, 2026

Abstract

Embodiments disclosed herein concern improved techniques for recognition of tabularized data within an image and construction of a computer-readable electronic table representing the recognized tabularized data. The image can be a screen image produced by a software application program operating on a computing device. The tabular data, such as pertaining to a table, can be recognized by recognizing rows, columns, text, controls and lines.

First claim

What is claimed is:

1 . A computer-implemented method for constructing an electronic table representing a table contained within an image, the method comprising:
receiving an image including the table, the image being a screen image from a display associated with a computing device; processing the image to capture object data from the image, the object data pertaining to objects within the image, the objects within the image include at least control objects, text objects and line objects, wherein the object data for a control object includes control object data, the object data for a text object includes text object data, and the object data for a line object includes line object data; determining columns and rows for the table contained within the image based on at least a portion of the captured object data from the image; identifying content for cells within the table contained within the image based on at least a portion of the captured object data from the image; and storing electronic data to a table data structure such that the electronic data is stored to one or more of the cells of the table data structure, the electronic data being stored to the table data structure being determined at least in part by the captured object data from the image, wherein the determining of the columns for the table contained within the image comprises:
identifying a header region for the table within the image using an image mask; and
detecting the columns for the table based on the captured object data for a plurality of the objects that are positionally within the header region.

2 . A computer-implemented method as recited in claim 1 , wherein the screen image is presented on the display by an application program operating on the computing device.

3 . A computer-implemented method as recited in claim 1 , wherein the image includes at least an image of an electronic window that includes at least the table.

4 . A computer-implemented method as recited in claim 1 , wherein the line objects are determined from a masking process that evaluates the image for lines.

5 . A computer-implemented method as recited in claim 1 , wherein the determining of the columns for the table contained within the image comprises:
evaluating the line objects to locate at least one candidate table position or region within the image; and determining at least one column for the table based on the at least one candidate table position or region within the image.

6 . A computer-implemented method as recited in claim 1 ,
wherein the at least one position or region within the image pertains to a corner of the table within the image, and wherein the determining of the at least one column for the table operates to determine at least a first column for the table.

7 . A computer-implemented method as recited in claim 1 , wherein the determining of the columns for the table contained within the image comprises:
identifying, from the captured object data, text, controls and/or slider bars within the header region for the table; and detecting the columns for the table based on the text, controls, and/or slider bars within the header region for the table.

8 . A computer-implemented method as recited in claim 1 , wherein the detecting the columns evaluates the plurality of objects in a positional order left-to-right in order to determine column boundaries.

9 . A computer-implemented method as recited in claim 8 , wherein the evaluation of each of the objects is dependent on the type of the object.

10 . A computer-implemented method as recited in claim 8 , wherein the determining of the columns for the table contained within the image comprises:
processing a current object from the plurality of objects, looking ahead to a next object from the plurality of objects; and influencing the determining of the columns for the table based on the current object and the next object.

11 . A computer-implemented method as recited in claim 8 , wherein the determining of the columns for the table contained within the image comprises:
considering a series of the objects from the plurality of objects to classify a current object of the plurality of objects as part of a current column or not.

12 . A computer-implemented method as recited in claim 11 , wherein the classification for the current object is based at least in part on an object type for at least one of the objects in the series of objects.

13 . A computer-implemented method as recited in claim 1 , wherein the determining of the rows for the table contained within the image comprises:
identifying text blocks that are substantially positioned in rows within the image; and examining the identified text blocks to determine the row boundary positions for the rows.

14 . A computer-implemented method as recited in claim 13 , wherein the determining of the rows for the table contained within the image comprises:
identifying horizontal divider lines from a masking process; and adjusting the determined row boundary positions based on the corresponding identified horizontal divider lines.

15 . A computer-implemented method as recited in claim 1 , wherein the method comprises:
placing at least a portion of the captured object data into an ordered list ordered by position within the image.

16 . A non-transitory computer readable medium including at least computer program code tangibly stored therein for constructing an electronic table representing a table contained within an image, the computer readable medium comprising:
computer program code for receiving an image including the table, the image being a screen image from a display associated with a computing device; computer program code for processing the image to capture object data from the image, the object data pertaining to objects within the image, the objects within the image include at least control objects, text objects and line objects, wherein the object data for a control object includes control object data, the object data for a text object includes text object data, and the object data for a line object includes line object data; computer program code for determining columns and rows for the table contained within the image based on at least a portion of the captured object data from the image; computer program code for identifying content for cells within the table contained within the image based on at least a portion of the captured object data from the image; and computer program code for storing electronic data to the electronic table such that the electronic data is stored to one or more of the cells of the electronic table, the electronic data being stored to the electronic table being determined at least in part by the captured object data from the image, wherein the computer program code for determining of the columns for the table contained within the image comprises:
computer program code for identifying a header region for the table within the image; and
computer program code for detecting the columns for the table based on the captured object data for a plurality of the objects that are positionally within the header region.

17 . A computer readable medium as recited in claim 16 , wherein the image is a screen image is presented on the display by an application program operating on the computing device, and wherein the screen image includes at least an image of a screen window that includes at least the table.

18 . A computer readable medium as recited in claim 16 , wherein the line objects are determined from a masking

Assignees

• Automation Anywhere Inc

Inventors

• Truscott Jesse
• Lodhiya Harshil
• Anand Virinchipuram

Classifications

• G06V2201/02

  Recognising information on displays, dials, clocks · CPC title

• G06F3/04847

  Interaction techniques to control parameter settings, e.g. interaction with sliders or dials · CPC title

• G06V30/413

  Classification of content, e.g. text, photographs or tables · CPC title

• G06V10/26

  Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion · CPC title

• G06V30/412 Primary

  Layout analysis of documents structured with printed lines or input boxes, e.g. business forms or tables · CPC title