System and method for providing content in autonomous vehicles based on perception dynamically determined at real-time

Patent metadata

Field	Value
Publication number	US-10015537-B2
Application number	US-201615199378-A
Country	US
Kind code	B2
Filing date	Jun 30, 2016
Priority date	Jun 30, 2016
Publication date	Jul 3, 2018
Grant date	Jul 3, 2018

Abstract

In one embodiment, an image analysis is performed on an image captured using a camera mounted on an autonomous vehicle, the image representing an exterior environment of an autonomous vehicle. Localization information surrounding the autonomous vehicle is obtained at a point in time. A perception of an audience external to the autonomous vehicle is determined based on the image analysis and the localization information. One or more content items are received from one or more content servers over a network in response to the perception of the audience. A first content item selected from the one or more content items is displayed on a display device mounted on an exterior surface of the autonomous vehicle.

First claim

What is claimed is:

1. A computer-implemented method for content delivery to an autonomous vehicle, the method comprising:
performing an image analysis on an image captured using a camera mounted on an autonomous vehicle, the image representing an exterior environment of an autonomous vehicle;
obtaining localization information surrounding the autonomous vehicle at a point in time;
determining a perception associated with an audience external to the autonomous vehicle based on the image analysis and the localization information;
receiving a plurality of content items from one or more content servers over a network in response to the perception associated with the audience;
ranking the plurality of content items using a predetermined ranking algorithm to select a first content item and a second content item from the plurality of content items;
displaying the first content item selected from the one or more content items on a first display device mounted on an exterior surface of the autonomous vehicle;
superimposing the second content item onto the image captured by the camera, generating an augmented image augmented with the second content item; and
displaying the augmented image on a second display device mounted on an interior surface of the autonomous vehicle, wherein the second display device includes a touch screen to allow a passenger to interact with one or more choices presented by the second content item, wherein the second display device is configured to appear as an ordinary window of the autonomous vehicle, and wherein when the passenger looks at the second display device, an object of the augmented image is displayed as a virtual object in a virtual reality manner as if the passenger looked at a physical object through the ordinary window of the autonomous vehicle.

2. The method of claim 1 , further comprising transmitting perception information describing the perception to the one or more of content servers to exchange the content items provided by the content servers.

3. The method of claim 2 , wherein the perception information comprises one or more keywords describing the perception, and wherein the content servers are cloud servers communicatively coupled to the autonomous vehicle over a cloud network.

4. The method of claim 1 , wherein the content items are ranked based on a user profile of the passenger.

5. The method of claim 1 , further comprising recognizing and determining the audience represented by the image based on the image analysis, wherein the first content item is selected based on the determined audience.

6. The method of claim 2 , further comprising:
capturing a second image representing a user interaction of a first user as part of the audience regarding the first content item;
compiling a user profile concerning the first user based on the second image captured by the camera, the second image including a user image of the first user;
transmitting the user interaction and the user profile to the one or more content servers to exchange a third content item; and
displaying the third content item on the first display device.

7. The method of claim 6 , further comprising:
transmitting the user interaction and the user profile to a plurality of content servers in addition to the one or more content servers;
receiving a second set of content items from at least some of the content servers; and
ranking the second set of content items using the predetermined ranking algorithm, wherein the second content item is selected based on the rankings.

8. The method of claim 1 , wherein the first display device is mounted on the same exterior surface of the autonomous vehicle on which the cameras is mounted.

9. A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations of providing content to autonomous vehicle, the operations including
performing an image analysis on an image captured using a camera mounted on an autonomous vehicle, the image representing an exterior environment of an autonomous vehicle;
obtaining localization information surrounding the autonomous vehicle at a point in time;
determining a perception associated with an audience external to the autonomous vehicle based on the image analysis and the localization information;
receiving a plurality of content items from one or more content servers over a network in response to the perception associated with the audience;
ranking the plurality of content items using a predetermined ranking algorithm to select a first content item and a second content item from the plurality of content items;
displaying the first content item selected from the one or more content items on a first display device mounted on an exterior surface of the autonomous vehicle;
superimposing the second content item onto the image captured by the camera, generating an augmented image augmented with the second content item; and
displaying the augmented image on a second display device mounted on an interior surface of the autonomous vehicle, wherein the second display device includes a touch screen to allow a passenger to interact with one or more choices presented by the second content item, wherein the second display device is configured to appear as an ordinary window of the autonomous vehicle, and wherein when the passenger looks at the second display device, an object of the augmented image is displayed as a virtual object in a virtual reality manner as if the passenger looked at a physical object through the ordinary window of the autonomous vehicle.

10. The machine-readable medium of claim 9 , wherein the operations further comprise transmitting perception information describing the perception to the one or more content servers to exchange the content items provided by the content servers.

11. The machine-readable medium of claim 10 , wherein the perception information comprises one or more keywords describing the perception, and wherein the content servers are cloud servers communicatively coupled to the autonomous vehicle over a cloud network.

12. The machine-readable medium of claim 9 , wherein the content items are ranked based on a user profile of the passenger.

13. The machine-readable medium of claim 9 , wherein the operations further comprise recognizing and determining the audience represented by the image based on the image analysis, wherein the first content item is selected based on the determined audience.

14. The machine-readable medium of claim 10 , wherein the operations further comprise:
capturing a second image representing a user interaction of a first user as part of the audience regarding the first content item;
compiling a user profile concerning the first user based on the second image captured by the camera, the second image including a user image of the first user;
transmitting the user interaction and the user profile to the one or more content servers to exchange a third content item; and
displaying the third content item on the first display device.

15. The machine-readable medium of claim 14 , wherein the operations further comprise:
transmitting the user interaction and the user profile to a plurality of content servers in addition to the one or more content servers;
receiving a second set of content items from at least some of the content servers; and
ranking the second set of content items using the predetermined ranking algorithm, wherein the second content item is selected based on the rankings.

16. The machine-readable medium of claim 9 , wherein the first display device is mo

Assignees

• Baidu Usa Llc

Inventors

• Li Liyun
• Liu Shaoshan
• Wu Shuang
• Peng James
• Wang Jingao

Classifications

• H04N21/2668

  Creating a channel for a dedicated end-user group, e.g. insertion of targeted commercials based on end-user profiles {(information retrieval from the Internet by querying with filtering and personalisation G06F16/9535; arrangements for replacing or switching information during the broadcast H04H20/10; push services over packet-switching network H04L12/1859; adaptation of message content in packet-switching networks H04L51/063)} · CPC title

• H04N21/251

  Learning process for intelligent management, e.g. learning user preferences for recommending movies (details of learning user preferences for the retrieval of video data in a video database G06F16/739; computer systems using learning methods G06N3/08) · CPC title

• H04N21/41422 Primary

  located in transportation means, e.g. personal vehicle (arrangements specially adapted for transportation systems in broadcast systems H04H20/62) · CPC title

• H04N21/4223

  Cameras (H04N23/00 takes precedence) · CPC title

• G06T7/70

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title