Method and apparatus for image processing

Patent metadata

Field	Value
Publication number	US-9799098-B2
Application number	US-59740608-A
Country	US
Kind code	B2
Filing date	Apr 24, 2008
Priority date	Apr 24, 2007
Publication date	Oct 24, 2017
Grant date	Oct 24, 2017

Abstract

Identifying objects in images is a difficult problem, particularly in cases an original image is noisy or has areas narrow in color or grayscale gradient. A technique employing a convolutional network has been identified to identify objects in such images in an automated and rapid manner. One example embodiment trains a convolutional network including multiple layers of filters. The filters are trained by learning and are arranged in successive layers and produce images having at least a same resolution as an original image. The filters are trained as a function of the original image or a desired image labeling; the image labels of objects identified in the original image are reported and may be used for segmentation. The technique can be applied to images of neural circuitry or electron microscopy, for example. The same technique can also be applied to correction of photographs or videos.

First claim

What is claimed is:

1. A method for image processing comprising:
training a convolutional network defined by nodes and edges, between nearest neighbor nodes, that form a lattice of multiple layers of filters arranged in successive layers of filters to produce at least one image in the successive layers of filters with at least a same resolution as an original image, wherein images at nodes of a given layer of the successive layers compose a given image of the given layer as a function of images at nodes of a previous layer convolved with corresponding filters of the previous layer, the previous layer immediately preceding the given layer in the convolutional network, the training including partitioning an affinity graph, representing affinities between affinity nodes of an output of the convolutional network, by cutting affinity edges with weak affinity to create clusters of affinity nodes that correspond to different image segments, the training further including cutting edges between nodes on boundaries of the edges.

2. The method of claim 1 wherein training the convolutional network includes training with back propagation learning.

3. The method of claim 1 wherein training the convolutional network includes training with stochastic gradient learning.

4. The method of claim 1 further including training the convolutional network as a function of training data.

5. The method of claim 1 further including producing an image at a resolution higher than the original image.

6. The method of claim 1 further including configuring the convolutional network to alternate between applying linear filtering and non-linear transformations.

7. An image processing system comprising:
a memory; and
a processor, the processor being configured to execute:
a training module, stored in the memory, to train a convolutional network defined by nodes and edges, between nearest neighbor nodes, that form a lattice of multiple layers of filters arranged in successive layers of filters to produce at least one image in the successive layers of filters with at least a same resolution as an original image, wherein images at nodes of a given layer of the successive layers compose a given image of the given layer as a function of images at nodes of a previous layer convolved with corresponding filters of the previous layer, the previous layer immediately preceding the given layer in the convolutional network, the training module including a partitioning module configured to partition an affinity graph, representing affinities between affinity nodes of an output of the convolutional network, by cutting affinity edges with weak affinity to create clusters of affinity nodes that correspond to different image segments and by cutting edges between nodes on boundaries of the edges.

8. The image processing system of claim 7 wherein the training module is configured to train the convolutional network with back propagation learning.

9. The image processing system of claim 7 wherein the training module is configured to train the convolutional network with stochastic gradient learning.

10. The image processing system of claim 7 wherein the training module is configured to train the convolutional network as a function of training data.

11. The image processing system of claim 7 wherein the convolutional network is arranged to produce an image at a resolution higher than the original image.

12. The image processing system of claim 7 wherein the convolutional network is configured to alternate between applying linear filtering and non-linear transformations.

13. A method for image processing comprising:
training a convolutional network defined by nodes and edges, between nearest neighbor nodes, that form a lattice of multiple layers of filters trained by learning and arranged in successive layers of filters to produce images in the successive layers of filters with at least a same resolution as an original image, wherein images at nodes of a given layer of the successive layers compose a given image of the given layer as a function of images at nodes of a previous layer convolved with corresponding filters of the previous layer, the previous layer immediately preceding the given layer in the convolutional network, the training including partitioning an affinity graph, representing affinities between affinity nodes of an output of the convolutional network, by cutting affinity edges with weak affinity to create clusters of affinity nodes that correspond to different image segments, the training further including cutting edges between nodes on boundaries of the edges;
training the filters as a function of the original image or a desired image labeling; and
reporting image labels of objects identified in the original image.

14. The method of claim 13 wherein training the filters includes training with back propagation.

15. The method of claim 13 wherein training the filters includes training with stochastic gradient learning.

16. The method of claim 13 further including training the filters as a function of training data.

17. The method of claim 13 further including producing an image with a higher resolution than the original image.

18. The method of claim 13 further including configuring the convolutional network to alternate between applying linear filtering and non-linear transformations.

19. The method of claim 13 further including accepting imaging labels, grouping together neighboring pixels with similar labels to produce new image labels corresponding to objects, and reporting the new image labels corresponding to the objects.

20. The method of claim 19 further including displaying the objects on a screen with different illustrations representing differences between or among the objects.

21. The method of claim 20 further including applying different colors to the objects.

22. An image processing system comprising:
a processor, the processor being configured to execute:
a first training module to train a convolutional network defined by nodes and edges, between nearest neighbor nodes, that form a lattice of multiple layers of filters trained by learning and arranged in successive layers of filters to produce at least one image in the successive layers of filters with at least a same resolution as an original image, wherein images at nodes of a given layer of the successive layers compose a given image of the given layer as a function of images at nodes of a previous layer convolved with corresponding filters of the previous layer, the previous layer immediately preceding the given layer in the convolutional network, the training including partitioning an affinity graph, representing affinities between affinity nodes of an output of the convolutional network, by cutting affinity edges with weak affinity to create clusters of affinity nodes that correspond to different image segments, the training further including cutting edges between nodes on boundaries of the edges;
a second training module to train the filters as a function of the original image or a desired image labeling; and
a reporting module to report image labels of objects identified in the original image.

23. The image processing system of claim 22 wherein the first training module is configured to train the convolutional network by back propagation.

24. The image processing system of claim 22 wherein the first training module is configured to train the convolutional network by gradient learning.

Assignees

• Seung H Sebastian
• Murray Joseph F
• Jain Viren
• Turaga Srinivas C
• Helmstaedter Moritz
• Denk Winfried
• Massachusetts Inst Technology
• Max-Planck-Gesellschaft Zur Forderung Der Wss E V

Inventors

• Seung H Sebastian
• Murray Joseph F
• Jain Viren
• Turaga Srinivas C
• Helmstaedter Moritz
• Denk Winfried

Classifications

• G06V10/454

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title

• G06T5/60

  using machine learning, e.g. neural networks · CPC title

• G06V10/267 Primary

  by performing operations on regions, e.g. growing, shrinking or watersheds · CPC title

• G06T5/001 Primary

  Physics · mapped topic

• G06K9/4628

  Physics · mapped topic