Method and system of recurrent semantic segmentation for image processing

Patent metadata

Field	Value
Publication number	US-10685446-B2
Application number	US-201815870608-A
Country	US
Kind code	B2
Filing date	Jan 12, 2018
Priority date	Jan 12, 2018
Publication date	Jun 16, 2020
Grant date	Jun 16, 2020

Abstract

A system, article, and method of recurrent semantic segmentation for image processing by factoring historical semantic segmentation.

First claim

What is claimed is:

1. A computer-implemented method of semantic segmentation for image processing, comprising:
obtaining a video sequence of frames of image data and comprising a current frame;
recurrently generating a semantic segmentation map in a view of a current pose of the current frame and comprising obtaining data to form the semantic segmentation map from a 3D semantic segmentation model at least partly based on voxels, a 3D mesh, or three-dimensional coordinates of objects in the content of the frames, wherein individual semantic segmentation maps are each associated with a different current frame from the video sequence;
extracting historically-influenced semantic features of the semantic segmentation map;
extracting current semantic features of the current frame;
matching an image location of the current semantic features and the historically-influenced semantic features to form an input feature vector that represents a single area of the image;
generating a current and historical semantically segmented frame comprising using both the current semantic features and the historically-influenced semantic features in the input feature vector as input to a neural network that indicates semantic labels for areas of the current and historical semantically segmented frame; and
semantically updating the 3D semantic segmentation model comprising using the current and historical semantically segmented frame.

2. The method of claim 1 comprising geometrically updating the 3D semantic segmentation model with data of individual current frames as the video sequence is being analyzed.

3. The method of claim 1 comprising rendering an updated semantic segmentation map at a current pose of each current frame used to geometrically update the 3D semantic segmentation model.

4. The method of claim 1 wherein extracting historically-influenced semantic features of the semantic segmentation map comprises inputting semantic segmentation label data from the semantic segmentation map to an extraction neural network and outputting the semantic features.

5. The method of claim 4 comprising placing the semantic segmentation label data into the form of tensors where one of the dimensions of the tensor is multiple likely semantic labels for a single pixel location.

6. The method of claim 4 wherein the neural network uses a convolutive neural network (CNN) that is a residual network (ResNet).

7. The method of claim 1 wherein the current semantic features and the historical semantic features are both in the form of tensors wherein one of the dimensions of the tensor is multiple likely semantic labels for a single pixel location.

8. The method of claim 1 wherein generating the current and historical semantically segmented frame comprises concatenating the current semantic features and the historically-influenced semantic features to form the input feature vector with both the current semantic features and the historically-influenced semantic features and to be input to the neural network.

9. The method of claim 8 wherein the feature vectors are a part of tensors, and the concatenation comprises tensor concatenation forming the input feature vectors; and the method comprising inputting a matrix at a time into the neural network and from the concatenated tensors.

10. The method of claim 1 wherein the 3D semantic segmentation model is geometrically updated by using a red, green, blue, depth scheme with simultaneous localization and mapping (RGBD-SLAM).

11. The method of claim 10 comprising determining a new pose estimate by using both the current frame in a current pose and a rendered image from a previous pose, wherein the rendered image is obtained by raycast projection from a 3D geometric model separate from the 3D semantic segmentation model;
providing the current and historical semantically segmented frame at the new pose estimate; and
updating the 3D semantic segmentation model comprising registering semantic labels of the current and historical semantically segmented frame at the new pose estimate and to the 3D semantic segmentation model.

12. A computer-implemented system of semantic segmentation for image processing, comprising:
at least one display;
at least one memory
at least one processor communicatively coupled to the display and the memory; and
a semantic segmentation unit operated by the at least one processor and to operate by:
obtaining a video sequence of frames of image data and comprising a current frame;
recurrently generating a semantic segmentation map in a view of a current pose of the current frame and comprising obtaining data to form the semantic segmentation map from a 3D semantic segmentation at least partly based on voxels, a 3D mesh, or three-dimensional coordinates of objects in the content of the frames, wherein individual semantic segmentation maps are each associated with a different current frame from the video sequence;
extracting historically-influenced semantic features of the semantic segmentation map;
extracting current semantic features of the current frame;
matching an image location of the current semantic features and the historically-influenced semantic features to form an input feature vector that represents a single area of the image;

generating a current and historical semantically segmented frame comprising using both the current semantic features and the historically-influenced semantic features in the input feature vector as input to a neural network that indicates semantic labels for areas of the current and historical semantically segmented frame; and
semantically updating the 3D semantic segmentation model comprising using the current and historical semantically segmented frame.

13. The system of claim 12 wherein the semantic segmentation unit is to operate by geometrically updating the 3D semantic segmentation model with data of individual current frames as the video sequence is being analyzed.

14. The system of claim 12 wherein the semantic segmentation unit is to operate by rendering an updated semantic segmentation map at a current pose of individual current frames used to geometrically update the 3D segmentation model.

15. The system of claim 12 wherein extracting historically-influenced semantic features of the semantic segmentation map comprises inputting semantic segmentation label data from the semantic segmentation map to an extraction neural network and outputting the semantic features.

16. The system of claim 15 wherein the semantic segmentation unit is to operate by placing the semantic segmentation label data into the form of tensors where one of the dimensions of the tensor is multiple likely semantic labels for a single pixel location.

17. The system of claim 15 wherein the neural network uses a convolutive neural network (CNN) that is a residual network (ResNet).

18. The system of claim 12 wherein generating the current and historical semantically segmented frame comprises concatenating the current semantic features and the historically-influenced semantic features to form an input feature vector with both the current semantic features and the historically-influenced semantic features and to be input to the neural network.

19. At least one non-transitory computer-readable medium having stored thereon instructions that when executed cause a computing device to operate by:
obtaining a video sequence of frames of image data and comprising a current frame;
recurrently generating a semantic segmentation map in a view of a current pose of the current frame and comp

Assignees

• Intel Corp

Inventors

• Fleishman Shahar
• Ken Korem Naomi
• Kliger Mark

Classifications

• G06V10/82 Primary

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06T7/174 Primary

  involving the use of two or more images · CPC title

• G06F18/2413

  based on distances to training or reference patterns · CPC title

• G06F18/29

  Graphical models, e.g. Bayesian networks · CPC title