Multi-stage segmentation using synthetic images

Patent metadata

Field	Value
Publication number	US-10991101-B2
Application number	US-201916299743-A
Country	US
Kind code	B2
Filing date	Mar 12, 2019
Priority date	Mar 12, 2019
Publication date	Apr 27, 2021
Grant date	Apr 27, 2021

Abstract

The example embodiments are directed to refinement process for generating an accurate image segmentation map. A refinement network may enhance an initially generated segmentation map using a model that is trained using synthetic images. In one example, the method may include storing an image of content which includes a plurality of categories of data, receiving an initial segmentation map of the image, the initial segmentation map comprising pixel probability values with respect to the plurality of categories, executing a refinement predictive model on the initial segmentation map and the image to generate a refined segmentation map, wherein the predictive model is trained using synthetic images of the plurality of categories of data, and generating a segmented image based on the refined segmentation map.

First claim

What is claimed is:

1. A computing system comprising:
a storage configured to storage an image of content which includes a plurality of categories of data; and
a processor configured to receive an initial segmentation map of the image, the initial segmentation map comprising pixel probability values with respect to the plurality of categories, and execute a refinement predictive model on the initial segmentation map and the image to generate a refined segmentation map, the refinement predictive model being trained based on synthetic images of the plurality of categories,
wherein the processor is configured to generate a segmented image based on the refined segmentation map.

2. The computing system of claim 1 , wherein the synthetic images comprise artificially-generated images having a known ground truth.

3. The computing system of claim 1 , wherein the synthetic images comprise at least one synthetic image dedicated to each category of data from among the plurality of categories of material.

4. The computing system of claim 1 , wherein the processor is further configured to generate the initial segmentation map of the image via execution of on an initial predictive model trained on real images of the plurality of categories of data.

5. The computing system of claim 1 , wherein the refinement predictive model comprises a cascaded neural network that separately processes a representation similar to the original representation of the image and the initial segmentation map, and combines the separately processed results to generate the refined segmentation map.

6. The computing system of claim 1 , wherein the processor is further configured to build the plurality of synthetic images based on image templates.

7. The computing system of claim 1 , wherein the processor is configured to assign at least one category of data from among the multiple categories to each pixel.

8. The computing system of claim 7 , wherein the executing of the refinement predictive model causes the processor to convert a pixel categorization included in the initial segmentation map into a sub-pixel categorization in the refined segmentation map.

9. A method comprising:
storing an image of content which includes a plurality of categories of data;
receiving an initial segmentation map of the image, the initial segmentation map comprising pixel probability values with respect to the plurality of categories;
executing a refinement predictive model on the initial segmentation map and the image to generate a refined segmentation map, wherein the predictive model is trained and optimized based on synthetic images of the plurality of categories of data; and
generating a segmented image based on the refined segmentation map.

10. The method of claim 9 , wherein the synthetic images comprise artificially-generated images having a known ground truth.

11. The method of claim 9 , wherein the synthetic images comprise at least one synthetic image dedicated to each category of data from among the plurality of categories of data.

12. The method of claim 9 , further comprising generating the initial segmentation map of the image via execution of on an initial predictive model trained on real images of the plurality of categories of data.

13. The method of claim 9 , wherein the refinement predictive model comprises a cascaded neural network that separately processes a representation similar to the original representation of the image and the initial segmentation map, and combines the separately processed results to generate the refined segmentation map.

14. The method of claim 9 , further comprising building the plurality of synthetic images based on image templates.

15. The method of claim 9 , wherein the generating comprises assigning at least one category from among the multiple categories to each pixel.

16. The method of claim 15 , wherein the executing of the refinement predictive model converts a pixel categorization included in the initial segmentation map into a sub-pixel categorization in the refined segmentation map.

17. A non-transitory computer-readable medium comprising:
storing an image of content which includes a plurality of categories of data;
receiving an initial segmentation map of the image, the initial segmentation map comprising pixel probability values with respect to the plurality of categories;
executing a refinement predictive model on the initial segmentation map and the image to generate a refined segmentation map, wherein the predictive model is trained and optimized based on synthetic images of the plurality of categories of data; and
generating a segmented image based on the refined segmentation map.

18. The non-transitory computer-readable medium of claim 17 , wherein the synthetic images comprise artificially-generated images having a known ground truth.

19. The non-transitory computer-readable medium of claim 17 , wherein the synthetic images comprise at least one synthetic image dedicated to each category of data from among the plurality of categories of data.

20. The non-transitory computer-readable medium of claim 17 , wherein the method further comprises generating the initial segmentation map of the image via execution of on an initial predictive model trained on real images of the plurality of categories of data.

Assignees

• Gen Electric

Inventors

• Brada Rafi Shmuel
• Wein Ron
• Wilson Gregory
• Santamaria-Pang Alberto
• Gugel Leonid

Classifications

• G06V10/774

  Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06V10/26

  Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion · CPC title

• G06V10/82

  using neural networks · CPC title

• G06T7/11 Primary

  Region-based segmentation · CPC title

• G06T7/10 Primary

  Segmentation; Edge detection (motion-based segmentation G06T7/215) · CPC title