Deep convolutional image up-sampling

Patent metadata

Field	Value
Publication number	US-10410322-B2
Application number	US-201715479675-A
Country	US
Kind code	B2
Filing date	Apr 5, 2017
Priority date	Apr 5, 2017
Publication date	Sep 10, 2019
Grant date	Sep 10, 2019

Abstract

An image and/or temporal sequence of images is received. The image and/or sequence of images was captured by an image capturing device of a vehicle apparatus onboard a vehicle and was down-sampled thereby. A scale of the image(s) is determined. An up-sampling network receives the image(s) and the scale. The up-sampling network determines appropriate network weights based on the scale. Based on the appropriate network weights, the up-sampling network generates a higher resolution image having a pre-defined scale.

First claim

That which is claimed:

1. An apparatus in communication with a Cloud-based computing environment, the apparatus comprising:
a communications interface for communicating with the Cloud-based computing environment;
a graphical processing unit configured to operate an up-sampling network; and
a processing unit configured to:
receive a temporal sequence of images comprising a plurality of down-sampled images (a) captured by an image capturing device of a vehicle apparatus onboard a vehicle and (b) down-sampled by the vehicle apparatus;
determine a scale of the plurality of down-sampled images;
cause the up-sampling network to receive the plurality of down-sampled images and the scale of the plurality of down-sampled images;
cause the up-sampling network to determine appropriate network weights based on the scale of the plurality of down-sampled images; and
cause the up-sampling network to generate a higher resolution image having a pre-defined scale based on the plurality of down-sampled images and the appropriate network weights, wherein (a) the higher resolution image is a composite higher resolution image and (b) generating the higher resolution image comprises up-sampling each of the plurality of down-sampled images to generate a temporal sequence of higher resolution images and performing a convolution of the temporal sequence of higher resolution images to generate the composite higher resolution image.

2. A method comprising:
receiving a temporal sequence of images comprising a plurality of down-sampled images (a) captured by an image capturing device of a vehicle apparatus onboard a vehicle and (b) down-sampled by the vehicle apparatus;
determining a scale of the plurality of down-sampled images;
receiving, by an up-sampling network, the plurality of down-sampled images and the scale of the plurality of down-sampled images;
determining, by the up-sampling network, appropriate network weights based on the scale of the plurality of down-sampled images; and
generating, by the up-sampling network, a higher resolution image having a pre-defined scale based on the plurality of down-sampled images and the appropriate network weights, wherein (a) the higher resolution image is a composite higher resolution image and (b) generating the higher resolution image comprises up-sampling each of the plurality of down-sampled images to generate a temporal sequence of higher resolution images and performing a convolution of the temporal sequence of higher resolution images to generate the composite higher resolution image.

3. A method according to claim 2 , wherein training the up-sampling network comprises:
receiving or accessing a full resolution image;
generating an instance of training data, wherein the instance of training data comprises (a) a plurality of down-sampled training images, each of the down-sampled training images being a down-sampled representation of the full resolution image at a particular scale and (b) the full resolution image;
receiving by a neural network the instance of training data;
for a particular down-sampled training image of the instance of training data, generating an up-sampled training image;
determining a loss function based on the up-sampled training image and the full resolution image; and
modifying one or more network weights based on the loss function.

4. A method according to claim 3 , wherein the higher resolution image is of the same resolution as the full resolution image.

5. A method according to claim 2 , wherein training the up-sampling network comprises:
receiving or accessing a temporal sequence of full resolution images;
generating an instance of training data, wherein the training data comprises (a) a plurality of temporal sequences of down-sampled images, each of the temporal sequences of down-sampled images being a down-sampled representation of the temporal sequence of full resolution images at a particular scale and (b) the temporal sequence of full resolution images;
receiving by a neural network the instance of training data;
for a particular temporal sequence of the down-sampled images, generating a temporal sequence of higher resolution images;
performing a convolution of the temporal sequence of higher resolution images to generate a training composite higher resolution image;
determining a loss function based on the training composite higher resolution image and at least one full resolution image of the temporal sequence of full resolution images; and
modifying one or more network weights based on the loss function.

6. A method according to claim 2 , wherein the up-sampling network is defined by a plurality of sets of network weights, wherein each set of network weights corresponds to a scale of a series of scales.

7. A method according to claim 6 , wherein:
the series of scales comprises a first scale and a second scale,
the first scale corresponds to a first set of network weights and the second scale corresponds to a second set of network weights,
the scale of the image is between the first scale and the second scale, and
the appropriate network weights are determined based on the first set of network weights and the second set of network weights.

8. A method according to claim 2 , wherein determining the scale of the image comprises at least one of analyzing the image, analyzing meta data corresponding to the image.

9. A method according to claim 2 , further comprising:
performing an image-based localization technique based on the higher resolution image to determine corrected pose information; and
providing the corrected pose information, wherein the corrected pose information is received by the vehicle apparatus and the vehicle apparatus determines at least one routing decision based on the corrected pose information.

10. A method according to claim 2 , further comprising:
receiving pose information corresponding to the image;
extracting feature information from the higher resolution image, the feature information corresponding to at least one feature detected within the higher resolution image; and
updating a feature map based on the pose information and the extracted feature information, the feature map being a layer of a digital map.

11. An apparatus comprising at least one processor, at least one memory storing computer program code, and an up-sampling network, the at least one memory and the computer program code configured to, with the processor, cause the apparatus to at least:
receive a temporal sequence of images comprising a plurality of down-sampled images (a) captured by an image capturing device of a vehicle apparatus onboard a vehicle and (b) down-sampled by the vehicle apparatus;
determine a scale of the plurality of down-sampled images;
receive, by the up-sampling network, the plurality of down-sampled images and the scale of the plurality of down-sampled images;
determine, by the up-sampling network, appropriate network weights based on the scale of the plurality of down-sampled images; and
generate, by the up-sampling network, a higher resolution image having a pre-defined scale based on the plurality of down-sampled images and the appropriate network weights, wherein (a) the higher resolution image is a composite higher resolution image and (b) generating the higher resolution image comprises up-sampling each of the plurality of down-sampled images to generate a temporal sequence of higher resolution images and performing a convolution of the temporal sequence of higher resolution images to generate the composite higher resolution image.

12. An apparatus according to claim 11 , wherein training the up-sampling network comprises:
receiving or accessin

Assignees

• Here Global Bv

Inventors

• Viswanathan Anirudh
• Christian Peter

Classifications

• G06T3/4046 Primary

  using neural networks · CPC title

• G06T3/4092

  Image resolution transcoding, e.g. by using client-server architectures · CPC title

• G06T3/4053

  based on super-resolution, i.e. the output image resolution being higher than the sensor resolution · CPC title