Method of processing image data in a connectionist network

Patent metadata

Field	Value
Publication number	US-2019171939-A1
Application number	US-201816202688-A
Country	US
Kind code	A1
Filing date	Nov 28, 2018
Priority date	Dec 5, 2017
Publication date	Jun 6, 2019
Grant date	—

Abstract

A method of processing image data in a connectionist network includes: determining, a plurality of offsets, each offset representing an individual location shift of an underlying one of the plurality of output picture elements, determining, from the plurality of offsets, a grid for sampling from the plurality of input picture elements, wherein the grid comprises a plurality of sampling locations, each sampling location being defined by means of a respective pair of one of the plurality of offsets and the underlying one of the plurality of output picture elements, sampling from the plurality of input picture elements in accordance with the grid, and transmitting, as output data for at least a subsequent one of the plurality of units of the connectionist network, a plurality of sampled picture elements resulting from the sampling, wherein the plurality of sampled picture elements form the plurality of output picture elements.

First claim

We claim:

1 . A method of processing image data in a connectionist network comprising a plurality of units ( 10 , 38 , 72 , 74 , 76 , 79 ), each unit having a data input ( 12 ), a data output ( 14 ), and at least one processing parameter, wherein processing of input data by means of the unit is at least partially determined by the at least one processing parameter of the unit,
wherein the method implements at least a respective one ( 10 ) of the plurality of units, and wherein the method comprises:
receiving, as raw input data ( 58 , 60 , 62 , 64 ) or input data processed by a preceding one ( 72 ) of the plurality of units, a plurality of input picture elements ( 24 ) representing an image acquired by means of an image sensor,
determining, for a plurality of output picture elements, a plurality of offsets ( 21 ), each offset ( 21 ) representing an individual location shift of an underlying one of the plurality of output picture elements,
determining, from the plurality of offsets ( 21 ), a grid ( 30 ) for sampling from the plurality of input picture elements ( 24 ), wherein the grid ( 30 ) comprises a plurality of sampling locations ( 32 ), each sampling location ( 32 ) being defined by means of a respective pair of one of the plurality of offsets ( 21 ) and the underlying one of the plurality of output picture elements,
sampling from the plurality of input picture elements ( 24 ) in accordance with the grid ( 30 ), and transmitting, as output data ( 18 ) for at least a subsequent one ( 38 , 74 , 76 , 78 ) of the plurality of units of the connectionist network, a plurality of sampled picture elements ( 33 ) resulting from the sampling, wherein the plurality of sampled picture elements ( 33 ) form the plurality of output picture elements.

2 . The method according to claim 1 ,
wherein the individual location shift represented by a respective one of the plurality of offsets ( 21 ) defines an arbitrary location relative to the underlying one of the plurality of output picture elements.

3 . The method according to claim 1 ,
wherein, for a respective one of the plurality of sampling locations ( 32 ) of the grid ( 30 ), sampling comprises interpolating a respective one ( 34 , 36 ) of the plurality of output picture elements associated with the respective one of the plurality of sampling locations ( 32 ) when the respective one of the plurality of sampling locations ( 32 ) is offside any of the plurality of input picture elements ( 24 ).

4 . The method according to claim 1 ,
wherein at least some of the plurality of offsets ( 21 ) are spatially limited to a predefined threshold.

5 . The method according to claim 1 ,
wherein determining the plurality of offsets ( 21 ) is carried out by means of a localization connectionist network ( 20 ), in particular neural network, comprising one or more units, each unit having a data input, a data output, and at least one processing parameter, wherein processing of the input data by means of the one or more units is at least partially determined by the at least one processing parameter.

6 . The method according to claim 5 ,
wherein the one or more units of the localization connectionist network ( 20 ) are convolutional units, wherein a respective convolutional unit implements a convolution of at least some of the plurality of input picture elements ( 24 ) received by the respective convolutional unit with a kernel filter, and where-in the sampling does not comprise a convolution with a kernel filter.

7 . The method according to claim 5 ,
wherein the localization connectionist network ( 20 ) is trained together with the connectionist network by means of the feed-forward algorithm and the back-propagation algorithm, wherein training comprises modifying the at least one processing parameter of at least some of the plurality of units of the connectionist network and/or of the one or more units of the localization connectionist network ( 20 ).

8 . The method according to claim 7 ,
wherein during training of the localization connectionist network ( 20 ) and the connectionist network and if the unit ( 10 ) implemented by the method receives input data processed by a preceding one ( 72 ) of the plurality of units of the connectionist network, training data from the localization connectionist network ( 20 ) is selectively not used for modifying the at least one processing parameter of the preceding one ( 72 ) of the plurality of units of the connectionist network ( 20 ).

9 . The method according to claim 7 ,
the localization connectionist network ( 20 ) and the connectionist network are trained before using the connectionist network for a predefined purpose.

10 . The method according to claim 1 ,
wherein at least one of the plurality of units of the connectionist network is a convolutional unit, wherein a respective convolutional unit implements a convolution of at least some of the plurality of input picture elements ( 24 ) or output picture elements ( 33 ) received by the respective convolutional unit with a kernel filter.

11 . The method according to claim 1 ,
wherein the connectionist network implements a classifier for at least parts of the image represented by the plurality of input picture elements ( 24 ).

12 . The method according to claim 1 ,
wherein the connectionist network implements a classifier for traffic signs.

13 . The method according to claim 1 ,
wherein the output picture elements ( 33 ) are transmitted as output data to a plurality of subsequent units ( 74 , 76 , 78 ) of the connectionist network.

14 . The method according to claim 1 ,
wherein the sampling does not comprise a convolution of the plurality of in-put picture elements ( 24 ) with a kernel filter.

15 . A vehicle with a camera unit having an image sensor acquiring at least one image of an object in the vicinity of the vehicle,
the vehicle further comprising a processing unit configured to carry out the method according to claim 1 .

Assignees

• Aptiv Tech Ltd

Inventors

• Zhu Weimeng
• Siegemund Jan

Classifications

• G06V10/82

  using neural networks · CPC title

• G06V20/582 Primary

  of traffic signs · CPC title

• G06N3/084 Primary

  Backpropagation, e.g. using gradient descent · CPC title

• G06N20/00

  Machine learning · CPC title

• G06V10/454

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title