System and method of object-based navigation

Patent metadata

Field	Value
Publication number	US-11282385-B2
Application number	US-201815961829-A
Country	US
Kind code	B2
Filing date	Apr 24, 2018
Priority date	Apr 24, 2018
Publication date	Mar 22, 2022
Grant date	Mar 22, 2022

Abstract

In an aspect of the disclosure, a method, a computer-readable medium, and apparatus for operating a computational network are provided. The apparatus may obtain a set of navigational instructions describing a route to a destination. The apparatus may obtain first image data through a first camera that is oriented toward the route, the first image data depicting a first scene associated with the route. The apparatus may determine a first field of view associated with a user that is navigating the route to the destination based on a first sensor that is oriented toward the user. The apparatus may identify at least one salient object represented in the first scene based on the first field of view. The apparatus may output instructional information describing a first navigational instruction of the set of navigational instructions with reference to the at least one salient object.

First claim

What is claimed is:

1. A method for providing navigational instructions using at least one artificial neural network, the method comprising:
obtaining, at one or more processors, a set of navigational instructions describing a route to a destination;
obtaining, at the one or more processors, first image data through a first camera that is oriented toward the route, the first image data depicting a first scene associated with the route;
obtaining, at the one or more processors, second image data captured through a second camera oriented toward a user that is navigating the route to the destination;
storing, at a memory, field-of-view information including at least one angle associated with an orientation of the head of the user, wherein the at least one angle comprises at least one of an azimuthal angle or an elevational angle;
determining, at the one or more processors, a first field of view associated with the user based on a gaze of the user in the second image data determined using the at least one artificial neural network, wherein the first field of view is based on the field-of-view information;
identifying, at the one or more processors, at least one salient object represented in the first scene based on the first field of view;
outputting, at a display or a speaker, instructional information identifying the at least one salient object and describing a first navigational instruction of the set of navigational instructions with reference to the at least one salient object, wherein the first navigational instruction indicates at least one of a distance or a direction of travel with reference to the at least one salient object;
reconfiguring, at the one or more processors, based on the first scene, at least one or more settings of a microphone, the first camera, the second camera, the display, or the speaker.

2. The method of claim 1 , further comprising: determining, at the one or more processors, the gaze of the user by processing the second image data using a deep neural network (DNN) implemented by the at least one artificial neural network.

3. The method of claim 1 , wherein the storing the field-of-view information comprises: storing, at the memory, the field-of-view information before the obtaining the set of navigational instructions describing the route to the destination.

4. The method of claim 1 , further comprising: processing, at the one or more processors, the first image data using a first deep neural network (DNN) implemented by the at least one artificial neural network for identification of a set of objects represented in the first scene, wherein the identifying the at least one salient object represented in the first scene based on the first field of view is further based on the set of objects represented in the first scene.

5. The method of claim 4 , wherein the identifying the at least one salient object represented in the first scene based on the first field of view comprises: processing, at the one or more processors, the set of objects using a second DNN implemented by the at least one artificial neural network for identification of the at least one salient object that is estimated to be at least partially within the first field of view.

6. The method of claim 5 , further comprising: adjusting, at the one or more processors, one or more weights associated with the second DNN based on the first field of view to influence a saliency of the at least one salient object.

7. The method of claim 6 , wherein the one or more weights are adjusted based at least in part on a profile associated with the user, the profile being based on at least one of an age associated with the user, a sex associated with the user, a driving experience level associated with the user, a preference associated with the user, historical information associated with the user, or a location associated with the user.

8. The method of claim 1 , wherein the first image data depicts the first scene over a plurality of time steps, and the at least one salient object is in motion over the plurality of time steps.

9. The method of claim 8 , further comprising: determining, at the one or more processors, the at least one salient object is in motion, wherein the instructional information describing the first navigational instruction of the set of navigational instructions with reference to the at least one salient object is based on the motion of the at least one salient object.

10. A vehicle for providing navigational instructions using at least one artificial neural network, the vehicle comprising:
a first camera;
a second camera;
at least one of a display or a speaker;
a navigation planner configured to obtain a set of navigational instructions describing a route to a destination;
an object detector configured to obtain first image data through the first camera of that is oriented toward the route, the first image data depicting a first scene associated with the route;
an attention detector configured to:
obtain second image data captured through the second camera oriented toward a user in the vehicle that is navigating the route to the destination;
store field-of-view information including at least one angle associated with an orientation of the head of the user, wherein the at least one angle comprises an azimuthal angle or an elevational angle; and
determine a first field of view associated with the user based on a gaze of the user in the second image data determined using the at least one artificial neural network, wherein the field of view is based on the field-of-view information;

one or more processors configured to identify at least one salient object represented in the first scene based on the first field of view; and
an object instruction component configured to output instructional information identifying the at least one salient object, at the at least one of the display or the speaker, describing a first navigational instruction of the set of navigational instructions with reference to the at least one salient object, wherein the first navigational instruction indicates at least one of a distance and/or a direction of travel with reference to the at least one salient object;
wherein the one or more processors are further configured to reconfigure, based on the first scene, at least one or more settings of a microphone, the first camera, the second camera, the display, or the speaker.

11. The vehicle of claim 10 , wherein the attention detector is further configured to determine the gaze of the user by processing the second image data using a deep neural network (DNN) implemented by the at least one artificial neural network.

12. The vehicle of claim 10 , wherein the attention detector is further configured to store the field-of-view information before the navigation planner obtains the set of navigational instructions describing the route to the destination.

13. The vehicle of claim 10 , wherein the object detector is further configured to process the first image data using a first deep neural network (DNN) implemented by the at least one artificial neural network for identification of a set of objects represented in the first scene, and wherein the identifying the at least one salient object represented in the first scene based on the first field of view is further based on the set of objects represented in the first scene.

14. The vehicle of claim 13 , wherein the object detector configured to identify the at least one salient object represented in the first scene based on the first field of view is further configured to process the set of objects using a second DNN implemented by the at least one artificial neural network f

Assignees

• Qualcomm Inc
• Qualcomm Incorproated

Inventors

• Lewis M Anthony
• CALLE Richard Anthony
• Bergan Charles

Classifications

• G01C21/3644 Primary

  Landmark guidance, e.g. using POIs or conspicuous other objects · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/454

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G08G1/096816 Primary

  where the complete route is transmitted to the vehicle at once · CPC title