Compositing images for augmented reality

Patent metadata

Field	Value
Publication number	US-10692288-B1
Application number	US-201715634843-A
Country	US
Kind code	B1
Filing date	Jun 27, 2017
Priority date	Jun 27, 2016
Publication date	Jun 23, 2020
Grant date	Jun 23, 2020

Abstract

A method may include capturing a first image of a physical environment using a mobile device. The mobile device may include a physical camera and a display. The method may also include receiving a second image from a content provider system. The second image may be generated by the content provider system by rendering a view from a virtual camera in a virtual environment. The virtual environment may represent at least a portion of the physical environment. A location of the virtual camera in the virtual environment may correspond to a location of the physical camera in the physical environment. The second image may include a view of a computer-generated object. The method may additionally include generating a third image by compositing the first image and the second image, and causing the third image to be displayed on the display of the mobile device.

First claim

What is claimed is:

1. A method comprising:
capturing a first image of a physical environment using a mobile device, wherein the mobile device comprises a physical camera and a display;
receiving a second image from a content provider system, wherein:
the second image is generated by the content provider system by rendering a view from a virtual camera in a virtual environment, wherein the virtual environment represents at least a portion of the physical environment, and a location of the virtual camera in the virtual environment corresponds to a location of the physical camera in the physical environment; and
the second image comprises a view of a computer-generated object comprising a virtual character, wherein:
the virtual character is moved according to a live motion-capture performance;
a time that the second image is generated in the virtual environment corresponds to a time when the first image is captured in the physical environment, such that when a third image is generated by compositing the first image and the second image, the first image and the second image are synchronized in time;
the second image comprises a four-channel image with an alpha channel that indicates portions of the second image that should be at least semi-transparent such that corresponding portions of the first image are at least partially visible as background through the second image; and
the portions of the second image that should be at least semi-transparent include a shadow of the computer-generated object;

generating the third image by compositing the first image and the second image; and
causing the third image to be displayed on the display of the mobile device.

2. The method of claim 1 , wherein the computer-generated object is not present in the first image.

3. The method of claim 1 , wherein the portions of the second image that should be at least semi-transparent include a mist or fog effect.

4. The method of claim 1 , wherein the second image comprises a cutout that at least partially obscures the view of the computer-generated object, wherein the cutout corresponds to an object in the physical environment.

5. The method of claim 1 , wherein the live motion-capture performance comprises:
a motion-capture environment comprising at least one physical item that corresponds to a physical item in the physical environment; and
a live motion-capture performer performing in the motion-capture environment.

6. The method of claim 1 , further comprising removing a chroma-key background from the second image to composite the first image and the second image.

7. A mobile device comprising:
a display;
a physical camera;
one or more processors; and
one or more memory devices comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
causing the physical camera to capture a first image of a physical environment;
receiving a second image from a content provider system, wherein:
the second image is generated by the content provider system by rendering a view from a virtual camera in a virtual environment, wherein the virtual environment represents at least a portion of the physical environment, and a location of the virtual camera in the virtual environment corresponds to a location of the physical camera in the physical environment; and
the second image comprises a view of a computer-generated object comprising a virtual character, wherein:
the virtual character is moved according to a live motion-capture performance;
a time that the second image is generated in the virtual environment corresponds to a time when the first image is captured in the physical environment, such that when a third image is generated by compositing the first image and the second image, the first image and the second image are synchronized in time;
the second image comprises a four-channel image with an alpha channel that indicates portions of the second image that should be at least semi-transparent such that corresponding portions of the first image are at least partially visible as background through the second image; and
the portions of the second image that should be at least semi-transparent include a shadow of the computer-generated object;

generating the third image by compositing the first image and the second image; and
causing the third image to be displayed on the display.

8. The mobile device of claim 7 , further comprising a light sensor that captures lighting information about the physical environment from a perspective of the physical camera; and
wherein the operations further comprise transmitting the lighting information to the content provider system, wherein the virtual environment is lighted based on the lighting information about the physical environment.

9. The mobile device of claim 7 , wherein the operations further comprise:
transmitting a focal length of the physical camera to the content provider system, wherein a virtual focal length of the virtual camera is adjusted to match the focal length of the physical camera.

10. The mobile device of claim 7 , further comprising a physical scanner that receives physical information about the physical environment; and
wherein the operations further comprise transmitting the physical information to the content provider system, wherein objects in the virtual environment are adjusted based on the physical information.

11. The mobile device of claim 7 , further comprising visual fiducials, wherein a position of the visual fiducials is determined by a plurality of cameras, wherein the location of the physical camera in the physical environment is based on the determined position of the visible fiducials.

12. The mobile device of claim 7 , further comprising an accelerometer that captures motion information for the mobile device; and
wherein the operations further comprise transmitting the motion information to the content provider system, wherein the location of the virtual camera in the virtual environment is adjusted based on the motion information.

13. A non-transitory, computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
capturing a first image of a physical environment using a mobile device, wherein the mobile device comprises a physical camera and a display;
receiving a second image from a content provider system, wherein:
the second image is generated by the content provider system by rendering a view from a virtual camera in a virtual environment, wherein the virtual environment represents at least a portion of the physical environment, and a location of the virtual camera in the virtual environment corresponds to a location of the physical camera in the physical environment; and
the second image comprises a view of a computer-generated object comprising a virtual character, wherein:
the virtual character is moved according to a live motion-capture performance;
a time that the second image is generated in the virtual environment corresponds to a time when the first image is captured in the physical environment, such that when a third image is generated by compositing the first image and the second image, the first image and the second image are synchronized in time;
the second image comprises a four-channel image with an alpha channel that indicates portions of the second image that should be at least semi-transparent such that corresponding portions of the first image are at least partially visible as background through the second image; and
the portions of the second image that should be at

Assignees

• Lucasfilm Entertainment Co Ltd

Inventors

• Rasmussen Nicholas
• Koperwas Michael
• Alexander Iv Earle M

Classifications

• A63F13/42

  by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle · CPC title

• A63F13/213

  comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence) · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06T19/20

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title