System and method for eye-gaze direction-based pre-training of neural networks

Patent metadata

Field	Value
Publication number	US-12468388-B2
Application number	US-202318489338-A
Country	US
Kind code	B2
Filing date	Oct 18, 2023
Priority date	Oct 26, 2022
Publication date	Nov 11, 2025
Grant date	Nov 11, 2025

Abstract

A method of training a disparity estimation network. The method includes obtaining an eye-gaze dataset having first images with at least one gaze direction associated with each of the first images. A gaze prediction neural network is trained based on the eye-gaze dataset to develop a model trained to provide a gaze prediction for an external image. A depth database is obtained that includes second images having depth information associated with each of the second images. A disparity estimation neural network for object detection is trained based on an output from the gaze prediction neural network and an output from the depth database.

First claim

What is claimed is:

1 . A method of training a disparity estimation network, the method comprising:
obtaining an eye-gaze dataset including a first plurality of images with at least one gaze direction associated with each of the first plurality of images; training a gaze prediction neural network based on the eye-gaze dataset to develop a model trained to provide a gaze prediction for an external image; obtaining a depth database including a second plurality of images having depth information associated with each of the second plurality of images, wherein the first plurality of images matches the second plurality of images.; and training a disparity estimation neural network for object detection based on an output from the gaze prediction neural network and an output from the depth database.

2 . The method of claim 1 , wherein the first plurality of images is captured by at least one optical sensor and the at least one gaze direction associated with each of the first plurality of images is captured by a gaze direction system configured to determine an eye gaze direction for at least one eye.

3 . The method of claim 2 , wherein the second plurality of images is captured by the at least one optical sensor and the depth information associated with each of the second plurality of image is captured by a distance sensor configured to determine a distance between an object and the distance sensor.

4 . The method of claim 1 , wherein training the gaze prediction neural network on the eye-gaze dataset includes performing a dilation on the at least one gaze direction.

5 . The method of claim 4 , wherein the dilation corresponds to an area of focus of an eye.

6 . The method of claim 1 , wherein training the gaze prediction neural network includes associating at least one eye gaze direction with a corresponding one of the first plurality of images.

7 . The method of claim 1 , wherein the output from the depth database includes transforming the depth information into normalized disparity maps according to normalized disparity label.

8 . The method of claim 7 , wherein the output from the depth database includes the second plurality of images.

9 . The method of claim 8 , wherein a resolution of the normalized disparity maps matches a scaled version of a corresponding one of the second plurality of images.

10 . The method of claim 1 , wherein training the disparity estimation neural network includes minimizing a least absolute deviation between a ground truth measurement and a prediction by the disparity estimation neural network.

11 . The method of claim 10 , including performing a back propagation when training the disparity estimation neural network to minimize the least absolute deviation.

12 . A non-transitory computer-readable storage medium embodying programmed instructions which, when executed by a processor, are operable for performing a method comprising:
obtaining an eye-gaze dataset including a first plurality of images with at least one gaze direction associated with each of the first plurality of images; training a gaze prediction neural network based on the eye-gaze dataset to develop a model trained to provide a gaze prediction for an external image, wherein training the gaze prediction neural network on the eye-gaze dataset includes performing a dilation on the at least one gaze direction.; obtaining a depth database including a second plurality of images having depth information associated with each of the second plurality of images; and training a disparity estimation neural network for object detection based on an output from the gaze prediction neural network and an output from the depth database.

13 . The non-transitory computer-readable storage medium of claim 12 , wherein the first plurality of images is captured by at least one optical sensor and the at least one gaze direction associated with each of the first plurality of images is captured by a gaze direction system configured to determine an eye gaze direction for at least one eye.

14 . The non-transitory computer-readable storage medium of claim 13 , wherein the second plurality of images is captured by the at least one optical sensor and the depth information associated with each of the second plurality of image is captured by a distance sensor configured to determine a distance between an object and the distance sensor.

15 . The non-transitory computer-readable storage medium of claim 12 , wherein training the gaze prediction neural network includes associating at least one eye gaze direction with a corresponding one of the first plurality of images.

16 . The non-transitory computer-readable storage medium of claim 12 , wherein the output from the depth database includes transforming the depth information into normalized disparity maps according to normalized disparity label.

17 . The non-transitory computer-readable storage medium of claim 16 , wherein the output from the depth database includes the second plurality of images.

18 . A vehicle system, the system comprising:
at least one optical sensor configured to capture a plurality of images; at least one distance sensor configured to measure a plurality of distances from the at least one distance sensor; an eye gaze measurement system configured to determine an eye position of a driver; and a controller in communication with the at least one optical sensor, the at least one distance sensor, and the eye gaze measurement system, wherein the controller is configured to:
obtain an eye-gaze dataset including a first plurality of images with at least one gaze direction associated with each of the first plurality of images;
train a gaze prediction neural network based on the eye-gaze dataset to develop a model trained to provide a gaze prediction for an external image;
obtain a depth database including a second plurality of images having depth information associated with each of the second plurality of images; and
train a disparity estimation neural network for object detection based on an output from the gaze prediction neural network and an output from the depth database, wherein training the disparity estimation neural network includes minimizing a least absolute deviation between a ground truth measurement and a prediction by the disparity estimation neural network.

19 . The vehicle system of claim 18 , wherein the first plurality of images matches the second plurality of images.

20 . The non-transitory computer-readable storage medium of claim 12 , wherein the first plurality of images matches the second plurality of images.

Assignees

• Gm Global Tech Operations Llc

Inventors

• Hecht Ron M
• Tsimhoni Omer
• Levi Dan
• Oron Shaul
• Forgacs Andrea
• Rahamim Ohad
• Celniker Gershon

Classifications

• G06V10/761

  Proximity, similarity or dissimilarity measures · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06V2201/07

  Target detection · CPC title

• G06T2207/20228

  Disparity calculation for image-based rendering · CPC title

• G06T7/50

  Depth or shape recovery · CPC title