Methods and apparatus for discriminative semantic transfer and physics-inspired optimization of features in deep learning

Patent metadata

Field	Value
Publication number	US-11669718-B2
Application number	US-201816609732-A
Country	US
Kind code	B2
Filing date	May 22, 2018
Priority date	May 23, 2017
Publication date	Jun 6, 2023
Grant date	Jun 6, 2023

Abstract

Methods and apparatus for discrimitive semantic transfer and physics-inspired optimization in deep learning are disclosed. A computation training method for a convolutional neural network (CNN) includes receiving a sequence of training images in the CNN of a first stage to describe objects of a cluttered scene as a semantic segmentation mask. The semantic segmentation mask is received in a semantic segmentation network of a second stage to produce semantic features. Using weights from the first stage as feature extractors and weights from the second stage as classifiers, edges of the cluttered scene are identified using the semantic features.

First claim

The invention claimed is:

1. At least one machine-readable non-transitory medium comprising a plurality of instructions, that when executed on a computing device, facilitate the computing device to perform a method comprising:
receiving, with a first stage of a pipeline framework, a sequence of training images in a convolutional neural network (CNN) to describe objects of a cluttered scene as a semantic segmentation mask;
receiving, with a second stage of the pipeline framework, the semantic segmentation mask in a semantic segmentation network and to produce semantic features; and
using, with a third stage of the pipeline framework, weights from the first stage as feature extractors and weights from the second stage as classifiers to identify edges of the cluttered scene using the semantic features.

2. The at least one machine-readable non-transitory medium of claim 1 , wherein:
the third stage is further configured to fine-tune the CNN that receives the training images and identifies the edges of the cluttered scene.

3. The at least one machine-readable non-transitory medium of claim 1 , wherein:
the second stage is further configured to use a softmax operation in the semantic segmentation network.

4. The at least one machine-readable non-transitory medium of claim 1 , wherein:
objects of the cluttered scene are part of a room layout dataset.

5. The at least one machine-readable non-transitory medium of claim 4 , wherein:
the room layout dataset comprises pixels; and
wherein the second stage is configured to treat each pixel of the room layout dataset as a sample in a fully connected layer of the semantic segmentation network.

6. The at least one machine-readable non-transitory medium of claim 4 , wherein: the second stage is further configured to model relationships between the room layout dataset and objects in the CNN.

7. The at least one machine-readable non-transitory medium of claim 1 , wherein:
the third stage is further configured to label edges of the cluttered scene.

8. A data processing system comprising:
a processing core configured to implement a convolutional neural network (CNN);
an I/O hub controller coupled to the processing core and configured to provide network, data storage, and access to the processing core;
a graphics processor coupled to the I/O hub controller and configured to implement:
a first stage configured to receive a sequence of training images in a convolutional neural network (CNN) to describe objects of a cluttered scene as a semantic segmentation mask;
a second stage configured to receive the semantic segmentation mask in a semantic segmentation network and to produce semantic features; and
a third stage configured to use weights from the first stage as feature extractors and weights from the second stage as classifiers in order to identify edges of the cluttered scene using the semantic features.

9. The data processing system of claim 8 , wherein the third stage is further configured to fine-tune the CNN that receives the training images and identifies the edges of the cluttered scene.

10. The data processing system of claim 8 , wherein the second stage is further configured to use a softmax operation in the semantic segmentation network.

11. The data processing system of claim 8 , wherein objects of the cluttered scene are part of a room layout dataset comprising pixels.

12. The data processing system of claim 11 , wherein the second stage is further configured to treat each pixel of the room layout dataset as a sample in a fully connected layer of the semantic segmentation network.

13. The data processing system of claim 11 , wherein the second stage is further configured to model relationships between the room layout dataset and objects in the CNN.

14. The data processing system of claim 8 , wherein the third stage is further configured to label edges of the cluttered scene.

15. A computation training method for a convolutional neural network (CNN) comprising:
receiving a sequence of training images in the CNN of a first stage to describe objects of a cluttered scene as a semantic segmentation mask;
receiving the semantic segmentation mask in a semantic segmentation network of a second stage to produce semantic features; and
using weights from the first stage as feature extractors and weights from the second stage as classifiers in a third stage to identify edges of the cluttered scene using the semantic features.

16. The method of claim 15 , further comprising:
fine-tuning the CNN in the third stage.

17. The method of claim 15 , further comprising:
using a softmax operation in the semantic segmentation network of the second stage.

18. The method of claim 17 , wherein objects of the cluttered scene are part of a room layout dataset comprising pixels.

19. The method of claim 18 , further comprising:
treating each pixel in the room layout dataset as a sample in a fully connected layer of the semantic segmentation network of the second stage.

20. The method of claim 18 , further comprising:
modelling relationships in the second stage between the room layout dataset and objects in the CNN.

21. The method of claim 15 , further comprising:
labeling edges of the cluttered scene pixel-wise in the third stage.

Assignees

• Intel Corp

Inventors

• Yao Anbang
• Zhao Hao
• Lu Ming
• Guo Yiwen
• Chen Yurong

Classifications

• G06V20/41

  Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49) · CPC title

• G06V10/454

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title

• G06V20/10

  Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) · CPC title

• G06N3/063 Primary

  using electronic means · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title