Augmented reality surface painting

Patent metadata

Field	Value
Publication number	US-10068547-B2
Application number	US-201213538644-A
Country	US
Kind code	B2
Filing date	Jun 29, 2012
Priority date	Jun 29, 2012
Publication date	Sep 4, 2018
Grant date	Sep 4, 2018

Abstract

Techniques for simulating interactions using an augmented reality device are described. Embodiments receive a request to paint over portions of a visual scene. Here, the visual scene is captured using one or more camera devices of the augmented reality device and is presented on a display of the augmented reality device. A first object in the visual scene corresponding to the first location is identified. Additionally, embodiments paint at least a portion of the first object as specified by the received request, and render a series of frames depicting the first object, such that the painted at least a portion of the first object is shown as painted when viewed from different perspectives using the augmented reality device.

First claim

What is claimed is:

1. A computer-implemented method to maintain conformity of a virtual painting onto a physical object captured and portrayed via an augmented reality device, even as the augmented reality device physically moves relative to the physical object in a physical environment over time, the computer-implemented method comprising:
presenting, via a display of the augmented reality device, the physical object within a visual scene captured in the physical environment using one or more camera devices of the augmented reality device, the physical object having an area of a first color;
receiving a request to at least partially fill the area using virtual paint of a second color different from the first color;
responsive to the request, identifying the physical object depicted by the visual scene, based on at least one of: (i) one or more edges determined for the physical object from analyzing the visual scene; and (ii) predefined geometrical data specifying geometrical characteristics of a specified type of the physical object;
determining one or more illumination characteristics of the physical environment;
generating, based on an initial image of the identified physical object in the visual scene and based further on the one or more illumination characteristics of the physical environment, an initial virtual painting projected onto the area of the physical object in the initial image and as viewed from a first physical position of the augmented reality device relative to the physical object in the physical environment; and
responsive to the augmented reality device being moved from the first physical position to a second physical position relative to the physical object in the physical environment, the second physical position being distinct from the first physical position, generating, based on an updated image of the identified physical object in the visual scene and based further on the one or more illumination characteristics of the physical environment, an updated virtual painting projected onto the area of the physical object in the updated image and as viewed from the second physical position of the augmented reality device, wherein the updated virtual painting is visually distinct from the initial virtual painting in terms of at least one visual aspect selected from size, shape, orientation, pattern, and color;
wherein the updated virtual painting is generated in order to maintain conformity of the virtual painting onto the physical object even as the augmented reality device physically moves relative to the physical object in the physical environment over time, wherein the initial virtual painting and the updated virtual painting are output.

2. The computer-implemented method of claim 1 , wherein identifying the physical object within the visual scene comprises:
analyzing the visual scene to determine the one or more edges of the physical object within the visual scene, wherein the physical object is detected based on the determined one or more edges.

3. The computer-implemented method of claim 1 , wherein identifying the physical object within the visual scene comprises:
retrieving the predefined geometrical data, which specifies geometrical characteristics of the specified type of the physical object; and
analyzing the visual scene using the predefined geometrical data in order to determine that the physical object is an instance of one of the specified type of physical object within the visual scene.

4. The computer-implemented method of claim 1 , wherein the area is of a specific shape.

5. The computer-implemented method of claim 1 ,
wherein the one or more illumination characteristics include at least one of a position of a light source within the physical environment, an angle of the light source, an indication of whether the light source is omnidirectional, a color of the light source, an intensity of the light source, and a reflectivity value of the physical object.

6. The computer-implemented method of claim 1 , wherein the request specifies to remove content at a first location within the visual scene, wherein adjusting the visual scene by painting at least a portion of the identified first object comprises:
removing the portion of the identified physical object from the visual scene as portrayed by the augmented reality device; and
replacing the removed portion with replacement content that is displayed by the augmented reality device, wherein the replacement content is determined based at least in part on content within the visual scene surrounding the removed portion of the identified physical object in the physical environment.

7. The computer-implemented method of claim 1 , further comprising:
upon detecting that at least a predefined threshold of the physical object has been painted, rendering a series of frames depicting the physical object as an animated virtual object within the visual scene.

8. The computer-implemented method of claim 1 , wherein the area is at least partially filled using the virtual paint of the second color as viewed from the augmented reality device, wherein the area is of a specific shape on one or more surfaces of the physical object, wherein the computer-implemented method further comprises:
generating a plurality of virtual objects corresponding to the physical object and portrayed at distinct locations separate from a physical location of the physical object in the physical environment;
wherein each of the plurality of virtual objects is visually distinct from the physical object in terms of at least one visual aspect selected from size, shape, orientation, pattern, and color, wherein each of the plurality of virtual objects is visually distinct from one another in terms of at least one visual aspect selected from size, shape, orientation, pattern, and color;
wherein each of the plurality of virtual objects is updated in order to maintain conformity of the respective virtual object to the physical environment as portrayed by the augmented reality device, in order to simulate actual presence of the respective virtual object in the physical environment;
wherein the presence of the respective virtual object is simulated at a respective physical location, in the physical environment, that is not occupied by any physical object;
wherein the conformity of the respective virtual object to the physical environment is maintained even as the augmented reality device physically moves relative to the respective physical location over time, wherein the respective updated virtual object is output via the augmented reality device.

9. The computer-implemented method of claim 8 , wherein the area is at least substantially filled using the virtual paint of the second color as viewed from the augmented reality device, wherein the computer-implemented method further comprises:
rendering a series of frames depicting the physical object, wherein the area remains painted in the rendered series of frames when the physical object is viewed from different perspectives as a physical position of the augmented reality device changes relative to the physical object, wherein rendering the series of frames comprises:
inserting the initial virtual painting into a first frame of the series of frames, when the physical object is viewed from the first physical position of the augmented reality device relative to the physical object; and
inserting the updated virtual painting into a second frame of the series of frames, when the physical object is viewed from the second physical position of the augmented reality device relative to the physical object.

10. The computer-implemented method of claim 9 , wherein the area is completely filled using the virtual paint of the se

Assignees

• Sobeski David
• Leake Brian
• Mitchell Kenny
• Disney Entpr Inc

Inventors

• Sobeski David
• Leake Brian
• Mitchell Kenny

Classifications

• G09G5/026 Primary

  Control of mixing and/or overlay of colours in general (G09G5/022 and G09G5/024 take precedence) · CPC title