Method and system for overlapping sliding window segmentation of image based on FPGA

Patent metadata

Field	Value
Publication number	US-11972504-B2
Application number	US-202318324174-A
Country	US
Kind code	B2
Filing date	May 26, 2023
Priority date	Aug 10, 2022
Publication date	Apr 30, 2024
Grant date	Apr 30, 2024

Abstract

Disclosed a method and a system for overlapping sliding window segmentation of an image based on an FPGA. According to the method, on-chip BRAMs storage resource cost of FPGA is determined; each on-chip BRAM in FPGA is used to cache the pixel data of each segmented sub-image in parallel; when the pixel data received by the BRAMs reaches a preset threshold or the last pixel of the segmented sub-image is written into the on-chip BRAMs, the data is written from the on-chip BRAMs to an off-chip DDR memory in a burst continuous writing mode; the repeated data generated by segmentation of horizontally overlapping sliding windows are written into the on-chip BRAMs corresponding to the current segmented sub-image and adjacent segmented sub-images thereof respectively in a synchronous and parallel manner.

First claim

The invention claimed is:

1. A method for overlapping sliding window segmentation of an original input image based on a field programmable gate array (FPGA), wherein the original input image is m rows×n columns, and a size of a segmented sub-image is k rows×k columns; a step size of an overlapping sliding window process is s, and s<k, the overlapping sliding window process comprises a horizontal overlapping sliding window process and a vertical overlapping sliding window process, and a sequence of the horizontal overlapping sliding window process is horizontal from left to right, and the vertical overlapping sliding window process is vertical from top to bottom; if a boundary is not divisible, expansion is implemented by filling all-zero pixels outwards; the FPGA needs ┌n/s┐ block random access memories (BRAMs) in total, which are numbered as 0,1, 2, . . . , ┌n/s┐−1, where ┌ ┐ means rounding up, where m, n and k are positive integers, l<m, l<n, k<m, and k<n;
the method comprises:
step S 1 : starting from a first row of the original input image, writing pixel data from (1,1) to (1,s) into BRAM_ 0 in sequence, wherein BRAM_ 0 corresponds to a (0,0) th segmented sub-image, (1,s+1) to (1,k) are overlapping areas of the (0,0) th segmented sub-image and a (0,1) th segmented sub-image in the horizontal overlapping sliding window process in the first row; and writing data in the overlapping areas into BRAM_ 0 and BRAM_ 1 in synchronous and parallel manner, wherein BRAM_ 1 corresponds to the (0,1) th segmented sub-image;
step S 2 : executing the horizontal overlapping sliding window process, and writing pixel data of (1, k+1) to (1,2s) in the first row into BRAM_ 1 ; during the horizontal overlapping sliding window process, starting a burst continuous write operation of each BRAM of the BRAMs to an off-chip DDR memory when data received by each BRAM reaches a preset threshold or when a last pixel of each segmented sub-image is written into a BRAM of the BRAMs corresponding to the segmented sub-image;
step S 3 : (1, 2s+1) to (1,s+k) pixels being overlaps of the (0,1) th segmented sub-image and a (0,2) th segmented sub-images in the horizontal overlapping sliding window process of the first row, and writing overlapped data synchronously into BRAM_ 1 and BRAM_ 2 , wherein BRAM_ 2 corresponds to the (0,2) th segmented sub-image;
step S 4 : executing the horizontal overlapping sliding window process in the first row, until the horizontal overlapping sliding window process and segmentation are performed on all pixels in the first row;
step S 5 , referring to the first row, sequentially executing the horizontal overlapping sliding window process from a second row to a k th row so that a first complete horizontal overlapping sliding window process is executed and data corresponding to the first complete horizontal overlapping sliding window process is written into the off-chip DDR memory;
step S 6 : executing the vertical overlapping sliding window process, wherein last (k−s) rows and k columns of pixel data of respective segmented sub-images (0,0), (0,1), . . . , (0, ┌n/s┐−1) are used as starting (k−s) rows and k columns of pixel data in a next batch of segmented sub-images (1,0), (1,1), . . . , (1, ┌n/s┐−1), and are spliced with each row of pixel data written subsequently;
step S 7 : executing the horizontal overlapping sliding window process from a k+1 th row to a s+k th row of the original input image according to steps S 1 -S 5 , in such a manner that a second batch of segmented sub-images (1,0), (1,1), . . . , (1, ┌n/s┐−1) has been written to corresponding BRAM_ 0 , BRAM_ 1 , . . . , BRAM_┌n/s┐, respectively, and executing the burst continuous write operation from the BRAMs to the off-chip DDR memory; and
step S 8 : repeating the horizontal overlapping sliding window process and the vertical overlapping sliding window process until an overlapping sliding window of the entire original input image is segmented and written into the off-chip DDR memory, respectively.

2. The method according to claim 1 , wherein during the vertical overlapping sliding window process and segmentation in step S 6 , the next batch of segmented sub-images and a previous batch of segmented sub-images are respectively stored in a same corresponding BRAM of the BRAMs, and (k−s)*k pixel data in the overlapping area formed by the vertical overlapping sliding window process are spliced with subsequent pixel data in the off-chip DDR memory by starting address rollback offset, thereby realizing data reuse in a vertical overlapping area.

3. The method according to claim 1 , wherein the preset threshold in step S 2 is equal to a maximum capacity of burst continuous writing, in such a manner that a highest access efficiency to the off-chip DDR is achieved.

4. The method according to claim 1 , wherein internal pixel data of each segmented sub-image written in the off-chip DDR memory in step S 8 are continuously addressed in row-major order, and respective segmented sub-images are sorted in column-major order with continuous addresses, that is, the sub-images are stored in an order of (0,0), (1,0), (2,0), . . . , (┌m/s┐−1,0), (0,1), (1,1), . . . , (┌m/s┐−1, ┌n/s┐−1).

5. A system, comprising one or more processors for implementing the method according to claim 1 .

6. A non-transitory computer-readable storage medium, wherein a program is stored thereon, and when executed by a processor, the program implements the method according to claim 1 .

Assignees

• Zhejiang Lab

Inventors

• Hu Tang
• Yu Xiao
• Li Xiangdi
• Ren Songnan
• Yan Li

Classifications

• G06T1/60 Primary

  Memory management · CPC title

• G06T1/20 Primary

  Processor architectures; Processor configuration, e.g. pipelining · CPC title