Surface selection for projection-based augmented reality system

Patent metadata

Field	Value
Publication number	US-11070792-B1
Application number	US-201816236322-A
Country	US
Kind code	B1
Filing date	Dec 28, 2018
Priority date	Dec 28, 2018
Publication date	Jul 20, 2021
Grant date	Jul 20, 2021

Abstract

A method includes retrieving a media content item for display to a user in an environment. The method includes receiving, from one or more interaction devices each including a projector and a camera, one or more media objects captured by one or more cameras, where one or more of the media objects include images of the user, and determining a location of the user in the environment based on an analysis of the one or more of the media objects. The method also includes identifying one of the interaction devices based on the determined location of the user and sending, to the identified interaction device, the media content item and instructions causing the projector associated with the identified interaction device to project the media content item.

First claim

What is claimed is:

1. A method comprising:
by a computing device, retrieving a media content item for display to a user in an environment;
by the computing device, receiving, from one or more interaction devices each comprising a projector and a camera, one or more media objects captured by one or more cameras, wherein one or more of the media objects comprise images of the user;
by the computing device, determining a location of the user in the environment based on an analysis of the one or more of the media objects;
by the computing device, identifying a plurality of projectable surfaces in the environment based on the analysis of the one or more of the media objects, wherein the media content item is to be projected to one of the plurality of projectable surfaces;
by the computing device, selecting a projectable surface among the plurality of projectable surfaces based on one or more characteristics associated with the media content item and the determined location of the user;
by the computing device, identifying one of the one or more interaction devices based on the selected projectable surface; and
by the computing device, sending, to the identified interaction device, the media content item and instructions causing the projector associated with the identified interaction device to project the media content item on the selected projectable surface.

2. The method of claim 1 , wherein identifying the plurality of projectable surfaces comprises identifying the plurality of projectable surfaces based at least in part on the determined location of the user.

3. The method of claim 1 , wherein identifying the plurality of projectable surfaces comprises identifying the plurality of projectable surfaces based at least in part on one or more characteristics associated with the media content item.

4. The method of claim 1 , wherein identifying the plurality of projectable surfaces comprises identifying the plurality of projectable surfaces that the user is facing.

5. The method of claim 1 , wherein each of the one or more of interaction devices comprises an audio sensor, further comprising:
receiving, from each of the one or more interaction devices, audio signals collected by the audio sensor associated with the interaction device;
extracting one or more features associated with a scene-understanding model based on an analysis of the received audio signals and media objects;
assigning a scene category to the environment based at least in part on the one or more features; and
wherein the interaction device is identified further based on the scene category.

6. The method of claim 1 , further comprising:
sending, to the identified interaction device, instructions causing the projector associated with the identified interaction device to project the media content item on a tangible object at a first location;
receiving, from the one or more interaction devices, one or more additional media objects captured by the cameras, wherein the one or more additional media objects comprise images of the tangible object;
determining, based on the one or more additional media objects, that the tangible object has moved from the first location to a second location; and
sending, to a different interaction device, instructions causing the projector associated with the different interaction device to project the media content item on the tangible object at the second location.

7. The method of claim 1 , further comprising:
receiving, from the one or more interaction devices, one or more additional media objects captured by the cameras, wherein the one or more additional media objects comprise images of the user;
determining, based on the one or more additional media objects, that the user has moved to a new location; and
sending, to a different interaction device in proximity to the new location, instructions causing the projector associated with the different interaction device to project the media content item.

8. The method of claim 1 , further comprising:
identifying a gesture of the user based on the one or more media objects, wherein the interaction device is identified based at least in part on the gesture of the user.

9. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
retrieve a media content item for display to a user in an environment;
receive, from one or more interaction devices each comprising a projector and a camera, one or more media objects captured by one or more cameras, wherein one or more of the media objects comprise images of the user;
determine a location of the user in the environment based on an analysis of the one or more of the media objects;
identify a plurality of projectable surfaces in the environment based on the analysis of the one or more of the media objects, wherein the media content item is to be projected to one of the plurality of projectable surfaces;
select a projectable surface among the plurality of projectable surfaces based on one or more characteristics associated with the media content item and the determined location of the user;
identify one of the one or more interaction devices based on the selected projectable surface; and
send, to the identified interaction device, the media content item and instructions causing the projector associated with the identified interaction device to project the media content item on the selected projectable surface.

10. The media of claim 9 , wherein identifying the plurality of projectable surfaces comprises identifying the plurality of projectable surfaces based at least in part on the determined location of the user.

11. The media of claim 9 , wherein identifying the plurality of projectable surfaces comprises identifying the plurality of projectable surfaces based at least in part on one or more characteristics associated with the media content item.

12. The media of claim 9 , wherein identifying the plurality of projectable surfaces comprises identifying the plurality of projectable surfaces that the user is facing.

13. The media of claim 9 , wherein each of the one or more of interaction devices comprises an audio sensor, and wherein the software is further operable when executed to:
receive, from each of the one or more interaction devices, audio signals collected by the audio sensor associated with the interaction device;
extract one or more features associated with a scene-understanding model based on an analysis of the received audio signals and media objects;
assign a scene category to the environment based at least in part on the one or more features; and
wherein the interaction device is identified further based on the scene category.

14. The media of claim 9 , wherein the software is further operable when executed to:
send, to the identified interaction device, instructions causing the projector associated with the identified interaction device to project the media content item on a tangible object at a first location;
receive, from the one or more interaction devices, one or more additional media objects captured by the cameras, wherein the one or more additional media objects comprise images of the tangible object;
determine, based on the one or more additional media objects, that the tangible object has moved from the first location to a second location; and
send, to a different interaction device, instructions causing the projector associated with the different interaction device to project the media content item on the tangible object at the second location.

15. The media of claim 9 , wherein the software is further operable when e

Assignees

• Facebook Inc

Inventors

• Elmieh Baback
• Hsu Joyce
• Snibbe Scott
• Mesguich Havilio Amir
• Chang Angela
• Jais Alexandre
• Crossen Rex

Classifications

• G06Q10/40 Primary

  Business processes related to social networking or social networking services · CPC title

• G06T2219/2004

  Aligning objects, relative positioning of parts · CPC title

• G06T2219/2021

  Shape modification · CPC title

• G06T19/20

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title