Limited rate context based eye gaze encoding

Patent metadata

Field	Value
Publication number	US-12200233-B2
Application number	US-202318165476-A
Country	US
Kind code	B2
Filing date	Feb 7, 2023
Priority date	Feb 7, 2023
Publication date	Jan 14, 2025
Grant date	Jan 14, 2025

Abstract

A system and method for limited rate context-based eye gaze encoding includes generating, based on an outward looking camera situated in a vehicle, a first video stream of a surrounding environment to a controller. The controller generates a scene description based on the first video stream and selects a corresponding scene from a predefined list of known scenes. Further, the controller, based on the selected corresponding scene, selects a codebook of encoding and decoding parameters from a plurality of predefined codebooks. Based on an inward looking camera situated in the vehicle, a second video stream of a face of a driver to an eye tracker controller is generated, where the eye tracker controller estimates a gaze direction of the driver. Encoded data comprising the estimated gaze direction of the driver is transmitted over a bandwidth limited channel to a decoder, based on the selected codebook, the encoded data.

First claim

What is claimed is:

1. A system for limited rate context-based eye gaze encoding comprising:
an outward looking camera, situated in a vehicle, configured to capture and send a first video stream of a surrounding environment to a controller, wherein the controller, based on the first video stream, is configured to generate and encode a scene description;
the controller further configured to select, based on an encoded scene description, a corresponding scene from a predefined list of known scenes;
the controller further configured to select, based on the selected corresponding scene, a codebook of encoding and decoding parameters from a plurality of predefined codebooks;
an inward looking camera, situated in the vehicle, configured to capture and send a second video stream of a face of a driver to an eye tracker controller, wherein the eye tracker controller, based on the second video stream, is configured to estimate a gaze direction of the driver;
an encoder configured to output, based on a selected codebook, an encoded data comprising the estimated gaze direction of the driver; and
a transmitter, within the vehicle, configured to send the encoded data over a bandwidth limited channel to a decoder, wherein the decoder is configured to receive the selected codebook from the controller and to decode, based on the selected codebook, the encoded data.

2. The system of claim 1 , further comprising an efficiency estimation component configured to receive the encoded data and determine an efficiency of the encoding.

3. The system of claim 2 , wherein if the efficiency of the encoding is below a predetermined threshold the selected codebook is replaced with a different codebook.

4. The system of claim 1 , wherein the decoder is located within the vehicle.

5. The system of claim 1 , wherein the inward looking camera situated in the vehicle is further configured to capture an image of a hand of the driver and wherein a hand position controller is configured to estimate an orientation of the hand of the driver, and wherein the encoder is further configured to output the encoded data comprising the orientation of the hand of the driver.

6. The system of claim 1 , wherein the inward looking camera situated in the vehicle is further configured to capture an image of a head of the driver and wherein a head orientation controller is configured to estimate an orientation of the head of the driver, and wherein the encoder is further configured to output the encoded data comprising the orientation of the head of the driver.

7. The system of claim 1 , wherein the inward looking camera situated in the vehicle is further configured to capture an image of a body of the driver and wherein a body pose controller is configured to estimate a pose of the body of the driver, and wherein the encoder is further configured to output the encoded data comprising the pose of the body of the driver.

8. The system of claim 1 , wherein the controller is further configured to select the corresponding scene from the predefined list of known scenes also based on a location, or a speed, or an acceleration, or a pedal position, or a turn signal status of the vehicle.

9. The system of claim 1 , wherein the controller is configured to generate a new codebook based on the corresponding scene.

10. A method for limited rate context-based eye gaze encoding comprising:
capturing and sending, using an outward looking camera situated in a vehicle, a first video stream of a surrounding environment to a controller;
generating, by the controller, a scene description based on the first video stream;
selecting, by the controller, based on the scene description, a corresponding scene from a predefined list of known scenes;
selecting, by the controller, based on the selected corresponding scene, a codebook of encoding and decoding parameters from a plurality of predefined codebooks;
capturing and sending, using an inward looking camera situated in the vehicle, a second video stream of a face of a driver to an eye tracker controller;
estimating, by the eye tracker controller, based on the second video stream, a gaze direction of the driver;
outputting, by an encoder, based on the selected codebook, an encoded data comprising the estimated gaze direction of the driver;
sending, by a transmitter within the vehicle, the encoded data over a bandwidth limited channel to a decoder; and
decoding, by the decoder, based on the selected codebook, the encoded data.

11. The method of claim 10 , further comprising receiving the encoded data and determining an efficiency of the encoding.

12. The method of claim 11 , further comprising replacing the selected codebook if the efficiency of the encoding is below a predetermined threshold with a different codebook.

13. The method of claim 10 , wherein the decoder is located within the vehicle.

14. The method of claim 10 , further comprising capturing, using the inward looking camera situated in the vehicle, an image of a hand of the driver, and estimating, using a hand position controller, an orientation of the hand of the driver, and outputting, by the encoder, the encoded data comprising the orientation of the hand of the driver.

15. The method of claim 10 , further comprising capturing, using the inward looking camera situated in the vehicle, an image of a head of the driver, and estimating, using a head orientation controller, an orientation of the head of the driver, and outputting, by the encoder, the encoded data comprising the orientation of the head of the driver.

16. The method of claim 10 , further comprising capturing, using the inward looking camera situated in the vehicle, an image of a body of the driver, and estimating, using a body pose controller, a pose of the body of the driver, and outputting, by the encoder, the encoded data comprising the pose of the body of the driver.

17. The method of claim 10 , further comprising selecting the corresponding scene from the predefined list of known scenes based also on a location, or a speed, or an acceleration, or a pedal position, or a turn signal status of the vehicle.

18. The method of claim 10 , further comprising selecting the corresponding scene from the predefined list of known scenes based also on a detected target and an amount of free space around the vehicle.

19. The method of claim 10 , further comprising generating a new codebook based on the corresponding scene.

20. A method for on-line limited rate context based eye gaze encoding comprising:
capturing and sending, using an outward looking camera situated in a vehicle, a first video stream of a surrounding environment to a controller;
generating, by the controller, a scene description based on the first video stream;
generating, by the controller, based on the scene description, a codebook of encoding and decoding parameters;
capturing and sending, using an inward looking camera situated in the vehicle, a second video stream of a face of a driver to an eye tracker controller;
estimating, by the eye tracker controller, based on the second video stream, a gaze direction of the driver;
outputting, by an encoder, based on the codebook, an encoded data comprising the estimated gaze direction of the driver;
sending, by a transmitter within the vehicle, the encoded data at a first rate over a bandwidth limited channel to a decoder;
sending, by the transmitter within the vehicle, the scene description at a second rate over the bandwidth limited channel to the decoder, wherein the second rate is lower than the f

Assignees

• Gm Global Tech Operations Llc

Inventors

• Bustin Ronit
• Tsimhoni Omer
• Forgacs Andrea
• Glaser Yi Guo
• Glaser Daniel S
• Hecht Ron M

Classifications

• H04N19/179

  the unit being a scene or a shot · CPC title

• G06T2207/30201

  Face · CPC title

• G06F3/013 Primary

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G06T2207/30252

  Vehicle exterior; Vicinity of vehicle · CPC title

• G06T7/70

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title