Systems and methods for generating object detection labels using foveated image magnification for autonomous driving

Patent metadata

Field	Value
Publication number	US-12340482-B2
Application number	US-202217652348-A
Country	US
Kind code	B2
Filing date	Feb 24, 2022
Priority date	Mar 16, 2021
Publication date	Jun 24, 2025
Grant date	Jun 24, 2025

Abstract

Systems and methods for processing high resolution images are disclosed. The methods include generating a saliency map of a received high-resolution image using a saliency model. The saliency map includes a saliency value associated with each of a plurality of pixels of the high-resolution image. The method then includes using the saliency map for generating an inverse transformation function that is representative of an inverse mapping of one or more first pixel coordinates in a warped image to one or more second pixel coordinates in the high-resolution image, and implementing an image warp for converting the high-resolution image to the warped image using the inverse transformation function. The warped image is a foveated image that includes at least one region having a higher resolution than one or more other regions of the warped image.

First claim

The invention claimed is:

1. A method of processing a high-resolution image, the method comprising, by a processor:
receiving a high-resolution image;
generating, using a saliency model, a saliency map of the high-resolution image, the saliency map comprising a saliency value associated with each of a plurality of pixels of the high-resolution image;
generating, using the saliency map, an inverse transformation function that is representative of an inverse mapping of one or more first pixel coordinates in a warped image to one or more second pixel coordinates in the high-resolution image;
implementing, using the inverse transformation function, an image warp for converting the high-resolution image to the warped image, the warped image being a foveated image that includes at least one region having a higher resolution than one or more other regions of the warped image; and
saving the warped image to a data store.

2. The method of claim 1 , further comprising:
generating, using an object detection model, one or more bounding box predictions in a frame of reference of the warped image; and
transforming, using the inverse transformation function, first coordinates of the one or more bounding box predictions in the warped image to second coordinates of the one or more bounding box predictions in a frame of reference of the high-resolution image.

3. The method of claim 2 , further comprising using the second coordinates of the one or more bounding box predictions for controlling navigation of an autonomous vehicle.

4. The method of claim 1 , further comprising generating the saliency model based on one or more bounding box predictions in at least one prior frame of a video stream, the high-resolution image being a part of the video stream and is captured after the at least one prior frame.

5. The method of claim 1 , further comprising generating the saliency model based on one or more bounding box predictions in a dataset-wide prior comprising a training dataset.

6. The method of claim 1 , wherein the at least one region having the higher resolution in the warped image has a high likelihood of including an object of interest.

7. The method of claim 1 , further comprising reducing a resolution of the one or more other regions of the warped image.

8. The method of claim 1 , wherein implementing, using the inverse transformation function, the image warp for converting the high-resolution image to the warped image comprises for each of a plurality of pixels of the warped image:
finding an input pixel in the high-resolution image; and
bi-linearly interpolating that pixel's intensity or color from one or more pixels in the high-resolution image adjacent the input pixel.

9. The method of claim 1 , wherein the inverse transformation function is a differentiable function that is trained using backpropagation.

10. The method of claim 1 , further comprising introducing symmetries about each of a plurality of edges of the saliency map for cropping regularization of the warped image.

11. A system for processing a high-resolution image, the system comprising:
a processor; and
a non-transitory computer readable medium comprising programming instructions that when executed by the processor, will cause the processor to:
receive a high-resolution image;
generate, using a saliency model, a saliency map of the high-resolution image, the saliency map comprising a saliency value associated with each of a plurality of pixels of the high-resolution image;
generate, using the saliency map, an inverse transformation function that is representative of an inverse mapping of one or more first pixel coordinates in a warped image to one or more second pixel coordinates in the high-resolution image;
implement, using the inverse transformation function, an image warp for converting the high-resolution image to the warped image, the warped image being a foveated image that includes at least one region having a higher resolution than one or more other regions of the warped image; and
save the warped image to a data store.

12. The system of claim 11 , further comprising programming instructions that when executed by the processor, will cause the processor to:
generate, using an object detection model, one or more bounding box predictions in a frame of reference of the warped image; and
transform, using the inverse transformation function, first coordinates of the one or more bounding box predictions in the warped image to second coordinates of the one or more bounding box predictions in a frame of reference of the high-resolution image.

13. The system of claim 12 , further comprising programming instructions that when executed by the processor, will cause the processor to use the second coordinates of the one or more bounding box predictions for controlling navigation of an autonomous vehicle.

14. The system of claim 11 , further comprising programming instructions that when executed by the processor, will cause the processor to generate the saliency model based on one or more bounding box predictions in at least one prior frame of a video stream, the high-resolution image being a part of the video stream and is captured after the at least one prior frame.

15. The system of claim 11 , further comprising programming instructions that when executed by the processor, will cause the processor to generate the saliency model based on one or more bounding box predictions in a dataset-wide prior comprising a training dataset.

16. The system of claim 11 , wherein the at least one region having the higher resolution in the warped image has a high likelihood of including an object of interest.

17. The system of claim 11 , further comprising programming instructions that when executed by the processor, will cause the processor to reduce a resolution of the one or more other regions of the warped image.

18. The system of claim 11 , wherein the programming instructions that when executed by the processor, will cause the processor to implement, using the inverse transformation function, the image warp for converting the high-resolution image to the warped image further comprise programming instructions to cause the processor to, for each of a plurality of pixels of the warped image:
find an input pixel in the high-resolution image; and
bi-linearly interpolate that pixel's intensity or color from one or more pixels in the high-resolution image adjacent the input pixel.

19. The system of claim 11 , wherein the inverse transformation function is a differentiable function that is trained using backpropagation.

20. A non-transitory computer program product for processing a high-resolution image, the computer program product comprising a memory that stores programming instructions that are configured to cause a processor to:
receive a high-resolution image;
generate, using a saliency model, a saliency map of the high-resolution image, the saliency map comprising a saliency value associated with each of a plurality of pixels of the high-resolution image;
generate, using the saliency map, an inverse transformation function that is representative of an inverse mapping of one or more first pixel coordinates in a warped image to one or more second pixel coordinates in the high-resolution image;
implement, using the inverse transformation function, an image warp for converting the high-resolution image to the warped image, the warped image being a foveated image that includes at least one region having a higher resolution than one or more other regions of the

Assignees

• Ford Global Tech Llc

Inventors

• Cebron Nicolas
• Ramanan Deva K
• Li Mengtian
• Thavamani Chitteshwaran

Classifications

• G06T3/4046

  using neural networks · CPC title

• G06T3/4007

  based on interpolation, e.g. bilinear interpolation (image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403) · CPC title

• G06T3/18 Primary

  Image warping, e.g. rearranging pixels individually · CPC title

• G06T3/053 Primary

  Detail-in-context presentations (fisheye or wide-angle transformations G06T3/047) · CPC title